summaryrefslogtreecommitdiffstats
path: root/llvm/lib/Target/NVPTX/NVPTXInstrInfo.td
Commit message (Collapse)AuthorAgeFilesLines
* [NVPTX] Restructure shfl instrinsics and add variants that return a predicate.Artem Belevich2019-10-141-0/+5
| | | | | | | | | Also, amend constraints for non-sync variants that are no longer available on sm_70+ with PTX6.4+. Differential Revision: https://reviews.llvm.org/D68892 llvm-svn: 374790
* DAG/GlobalISel: Correct type profile of bitcount opsMatt Arsenault2019-09-131-4/+4
| | | | | | | | The result integer does not need to be the same width as the input. AMDGPU, NVPTX, and Hexagon all have patterns working around the types matching. GlobalISel defines these as being different type indexes. llvm-svn: 371797
* PTX 6.3 extends `wmma` instruction to support s8/u8/s4/u4/b1 -> s32.Artem Belevich2019-04-251-0/+3
| | | | | | | | | | | | All of the new instructions are still handled mostly by tablegen. I've slightly refactored the code to drive intrinsic/instruction generation from a master list of supported variants, so all irregularities have to be implemented in one place only. The test generation script wmma.py has been refactored in a similar way. Differential Revision: https://reviews.llvm.org/D60015 llvm-svn: 359247
* [NVPTX] generate correct MMA instruction mnemonics with PTX63+.Artem Belevich2019-04-251-0/+4
| | | | | | | | | | | PTX 6.3 requires using ".aligned" in the MMA instruction names. In order to generate correct name, now we pass current PTX version to each instruction as an extra constant operand and InstPrinter adjusts its output accordingly. Differential Revision: https://reviews.llvm.org/D59393 llvm-svn: 359246
* [NVPTX] Fix the codegen for llvm.round.Bixia Zheng2019-04-011-9/+0
| | | | | | | | | | | | | | | | | | | | | | | | Summary: Previously, we translate llvm.round to PTX cvt.rni, which rounds to the even interger when the source is equidistant between two integers. This is not correct as llvm.round should round away from zero. This change replaces llvm.round with a round away from zero implementation through target specific custom lowering. Modify a few affected tests to not check for cvt.rni. Instead, we check for the use of a few constants used in implementing round. We are also adding CUDA runnable tests to check for the values produced by llvm.round to test-suites/External/CUDA. Reviewers: tra Subscribers: jholewinski, sanjoy, jlebar, hiraditya, llvm-commits Tags: #llvm Differential Revision: https://reviews.llvm.org/D59947 llvm-svn: 357407
* Update the file headers across all of the LLVM projects in the monorepoChandler Carruth2019-01-191-4/+3
| | | | | | | | | | | | | | | | | to reflect the new license. We understand that people may be surprised that we're moving the header entirely to discuss the new license. We checked this carefully with the Foundation's lawyer and we believe this is the correct approach. Essentially, all code in the project is now made available by the LLVM project under our new license, so you will see that the license headers include that license only. Some of our contributors have contributed code under our old license, and accordingly, we have retained a copy of our old license notice in the top-level files in each project and repository. llvm-svn: 351636
* [NVPTX] Allow libcalls that are defined in the current module.Justin Lebar2018-12-261-1/+20
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The patch adds a possibility to make library calls on NVPTX. An important thing about library functions - they must be defined within the current module. This basically should guarantee that we produce a valid PTX assembly (without calls to not defined functions). The one who wants to use the libcalls is probably will have to link against compiler-rt or any other implementation. Currently, it's completely impossible to make library calls because of error LLVM ERROR: Cannot select: i32 = ExternalSymbol '...'. But we can lower ExternalSymbol to TargetExternalSymbol and verify if the function definition is available. Also, there was an issue with a DAG during legalisation. When we expand instruction into libcall, the inner call-chain isn't being "integrated" into outer chain. Since the last "data-flow" (call retval load) node is located in call-chain earlier than CALLSEQ_END node, the latter becomes a leaf and therefore a dead node (and is being removed quite fast). Proposed here solution relies on another data-flow pseudo nodes (ProxyReg) which purpose is only to keep CALLSEQ_END at legalisation and instruction selection phases - we remove the pseudo instructions before register scheduling phase. Patch by Denys Zariaiev! Differential Revision: https://reviews.llvm.org/D34708 llvm-svn: 350069
* [SelectionDAG] Move (repeated) SDTIntShiftDOp double shift node def to ↵Simon Pilgrim2018-11-161-3/+0
| | | | | | | | common code. NFCI. Prep work for PR39467. llvm-svn: 347067
* [NVPTX] Remove ftz variants of cvt with rounding modeBenjamin Kramer2018-08-211-36/+6
| | | | | | | | These do not exist in ptxas, it refuses to compile them. Differential Revision: https://reviews.llvm.org/D51042 llvm-svn: 340317
* [NVPTX] Added a feature to use short pointers for const/local/shared AS.Artem Belevich2018-05-091-0/+1
| | | | | | | | | | | | Const/local/shared address spaces are all < 4GB and we can always use 32-bit pointers to access them. This has substantial performance impact on kernels that uses shared memory for intermediary results. The feature is disabled by default. Differential Revision: https://reviews.llvm.org/D46147 llvm-svn: 331941
* [NVPTX, CUDA] Added support for m8n32k16 and m32n8k16 variants of wmma ↵Artem Belevich2018-04-181-0/+1
| | | | | | | | | | instructions. The new instructions were added added for sm_70+ GPUs in CUDA-9.1. Differential Revision: https://reviews.llvm.org/D45068 llvm-svn: 330296
* [NVPTX] TblGen-ized lowering of WMMA intrinsics.Artem Belevich2018-03-151-0/+1
| | | | | | | | NFC. Differential Revision: https://reviews.llvm.org/D43151 llvm-svn: 327672
* [NVPTX] Removed always-true predicates in NVPTX.Artem Belevich2018-02-281-23/+6
| | | | | | | | | | | NVPTX stopped supporting GPUs older than sm_20 (Fermi) quite a while back. Removal of support of pre-Fermi GPUs made a lot of predicates in the NVPTX backend pointless as they can't ever be false any more. It's time to retire them. NFC intended. Differential Revision: https://reviews.llvm.org/D43843 llvm-svn: 326349
* [NVPTX] added match.{any,all}.sync instructions, intrinsics & builtins.Artem Belevich2017-09-261-0/+1
| | | | | | Differential Revision: https://reviews.llvm.org/D38191 llvm-svn: 314223
* Revert "[NVPTX] added match.{any,all}.sync instructions, intrinsics & ↵Justin Lebar2017-09-251-1/+0
| | | | | | | | | | | | | | | builtins.", rL314135. Causing assertion failures on macos: > Assertion failed: (Num < NumOperands && "Invalid child # of SDNode!"), > function getOperand, file > /Users/buildslave/jenkins/workspace/clang-stage1-cmake-RA-incremental/llvm/include/llvm/CodeGen/SelectionDAGNodes.h, > line 835. http://green.lab.llvm.org/green/job/clang-stage1-cmake-RA-incremental/42739/testReport/LLVM/CodeGen_NVPTX/surf_read_cuda_ll/ llvm-svn: 314142
* [NVPTX] added match.{any,all}.sync instructions, intrinsics & builtins.Artem Belevich2017-09-251-0/+1
| | | | | | Differential Revision: https://reviews.llvm.org/D38191 llvm-svn: 314135
* [NVPTX] Implemented bar.warp.sync, barrier.sync, and vote{.sync} ↵Artem Belevich2017-09-211-0/+3
| | | | | | | | instructions/intrinsics/builtins. Differential Revision: https://reviews.llvm.org/D38148 llvm-svn: 313898
* Changed EOL back to LF. NFC.Artem Belevich2017-07-201-3165/+3165
| | | | llvm-svn: 308671
* Add extra operand to CALLSEQ_START to keep frame part set up previouslySerge Pavlov2017-05-091-4/+5
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Using arguments with attribute inalloca creates problems for verification of machine representation. This attribute instructs the backend that the argument is prepared in stack prior to CALLSEQ_START..CALLSEQ_END sequence (see http://llvm.org/docs/InAlloca.htm for details). Frame size stored in CALLSEQ_START in this case does not count the size of this argument. However CALLSEQ_END still keeps total frame size, as caller can be responsible for cleanup of entire frame. So CALLSEQ_START and CALLSEQ_END keep different frame size and the difference is treated by MachineVerifier as stack error. Currently there is no way to distinguish this case from actual errors. This patch adds additional argument to CALLSEQ_START and its target-specific counterparts to keep size of stack that is set up prior to the call frame sequence. This argument allows MachineVerifier to calculate actual frame size associated with frame setup instruction and correctly process the case of inalloca arguments. The changes made by the patch are: - Frame setup instructions get the second mandatory argument. It affects all targets that use frame pseudo instructions and touched many files although the changes are uniform. - Access to frame properties are implemented using special instructions rather than calls getOperand(N).getImm(). For X86 and ARM such replacement was made previously. - Changes that reflect appearance of additional argument of frame setup instruction. These involve proper instruction initialization and methods that access instruction arguments. - MachineVerifier retrieves frame size using method, which reports sum of frame parts initialized inside frame instruction pair and outside it. The patch implements approach proposed by Quentin Colombet in https://bugs.llvm.org/show_bug.cgi?id=27481#c1. It fixes 9 tests failed with machine verifier enabled and listed in PR27481. Differential Revision: https://reviews.llvm.org/D32394 llvm-svn: 302527
* [NVPTX] Add support for ISD::ABS loweringSimon Pilgrim2017-05-061-3165/+3164
| | | | | | | | Use the ISD::ABS opcode directly Differential Revision: https://reviews.llvm.org/D32944 llvm-svn: 302356
* Fix spelling compliment->complement. Mostly refering to 2s complement. NFCCraig Topper2017-04-111-1/+1
| | | | llvm-svn: 299970
* [NVPTX] Added missing LDU/LDG intrinsics for f16.Artem Belevich2017-03-021-0/+1
| | | | | | Differential Revision: https://reviews.llvm.org/D30512 llvm-svn: 296784
* [NVPTX] Added support for .f16x2 instructions.Artem Belevich2017-02-231-6/+118
| | | | | | | | | | | | | This patch enables support for .f16x2 operations. Added new register type Float16x2. Added support for .f16x2 instructions. Added handling of vectorized loads/stores of v2f16 values. Differential Revision: https://reviews.llvm.org/D30057 Differential Revision: https://reviews.llvm.org/D30310 llvm-svn: 296032
* [NVPTX] Implement NVPTXTargetLowering::getSqrtEstimate.Justin Lebar2017-01-311-10/+1
| | | | | | | | | | | | | | | | | | | | | | | | Summary: This lets us lower to sqrt.approx and rsqrt.approx under more circumstances. * Now we emit sqrt.approx and rsqrt.approx for calls to @llvm.sqrt.f32, when fast-math is enabled. Previously, we only would emit it for calls to @llvm.nvvm.sqrt.f. (With this patch we no longer emit sqrt.approx for calls to @llvm.nvvm.sqrt.f; we rely on intcombine to simplify llvm.nvvm.sqrt.f into llvm.sqrt.f32.) * Now we emit the ftz version of rsqrt.approx when ftz is enabled. Previously, we only emitted rsqrt.approx when ftz was disabled. Reviewers: hfinkel Subscribers: llvm-commits, tra, jholewinski Differential Revision: https://reviews.llvm.org/D28508 llvm-svn: 293605
* [NVPTX] Standardize asm printer on "foo \tbar".Justin Lebar2017-01-181-48/+48
| | | | | | | Some instructions were printed as "foo\tbar", but most are printed as "foo \bar". Standardize on the latter form. llvm-svn: 292306
* [NVPTX] Clean up nested !strconcat calls.Justin Lebar2017-01-181-2/+1
| | | | | | | !strconcat is a variadic function; it will concatenate an arbitrary number of strings. There's no need to nest it. llvm-svn: 292305
* [NVPTX] Implement min/max in tablegen, rather than with custom DAGComine logic.Justin Lebar2017-01-181-0/+6
| | | | | | | | | | | | | | | | | | | | | | | Summary: This change also lets us use max.{s,u}16. There's a vague warning in a test about this maybe being less efficient, but I could not come up with a case where the resulting SASS (sm_35 or sm_60) was different with or without max.{s,u}16. It's true that nvcc seems to emit only max.{s,u}32, but even ptxas 7.0 seems to have no problem generating efficient SASS from max.{s,u}16 (the casts up to i32 and back down to i16 seem to be implicit and nops, happening via register aliasing). In the absence of evidence, better to have fewer special cases, emit more straightforward code, etc. In particular, if a new GPU has 16-bit min/max instructions, we want to be able to use them. Reviewers: tra Subscribers: jholewinski, llvm-commits Differential Revision: https://reviews.llvm.org/D28732 llvm-svn: 292304
* [NVPTX] Lower integer absolute value idiom to abs instruction.Justin Lebar2017-01-181-0/+12
| | | | | | | | | | | | Summary: Previously we lowered it literally, to shifts and xors. Reviewers: tra Subscribers: jholewinski, llvm-commits Differential Revision: https://reviews.llvm.org/D28722 llvm-svn: 292303
* [NVPTX] Improve lowering of llvm.ctpop.Justin Lebar2017-01-181-5/+9
| | | | | | | | | | | | | | | | | | Summary: Avoid an unnecessary conversion operation when using the result of ctpop.i32 or ctpop.i16 as an i32, as in both cases the ptx instruction we run returns an i32. (Previously if we used the value as an i32, we'd do an unnecessary zext+trunc.) Reviewers: tra Subscribers: jholewinski, llvm-commits Differential Revision: https://reviews.llvm.org/D28721 llvm-svn: 292302
* [NVPTX] Add lowering for llvm.bitreverse.Justin Lebar2017-01-181-0/+10
| | | | | | | | | | Reviewers: tra Subscribers: llvm-commits, jholewinski Differential Revision: https://reviews.llvm.org/D28720 llvm-svn: 292301
* [NVPTX] Improve lowering of llvm.ctlz.Justin Lebar2017-01-181-9/+23
| | | | | | | | | | | | | | | | | Summary: * Disable "ctlz speculation", which inserts a branch on every ctlz(x) which has defined behavior on x == 0 to check whether x is, in fact zero. * Add DAG patterns that avoid re-truncating or re-expanding the result of the 16- and 64-bit ctz instructions. Reviewers: tra Subscribers: llvm-commits, jholewinski Differential Revision: https://reviews.llvm.org/D28719 llvm-svn: 292299
* [NVPTX] Added support for half-precision floating point.Artem Belevich2017-01-131-8/+214
| | | | | | | | | | | | | | | | Only scalar half-precision operations are supported at the moment. - Adds general support for 'half' type in NVPTX. - fp16 math operations are supported on sm_53+ GPUs only (can be disabled with --nvptx-no-f16-math). - Type conversions to/from fp16 are supported on all GPU variants. - On GPU variants that do not have full fp16 support (or if it's disabled), fp16 operations are promoted to fp32 and results are converted back to fp16 for storage. Differential Revision: https://reviews.llvm.org/D28540 llvm-svn: 291956
* [NVPTX] Only lower sin/cos to approximate instructions if unsafe math is ↵Artem Belevich2017-01-131-2/+5
| | | | | | | | | | | | | | allowed. Previously we'd always lower @llvm.{sin,cos}.f32 to {sin.cos}.approx.f32 instruction even when unsafe FP math was not allowed. Clang-generated IR is not affected by this as it uses precise sin/cos from CUDA's libdevice when unsafe math is disabled. Differential Revision: https://reviews.llvm.org/D28619 llvm-svn: 291936
* [NVPTX] Fix some Clang-tidy modernize and Include What You Use warnings; ↵Eugene Zelenko2017-01-091-1/+1
| | | | | | other minor fixes (NFC). llvm-svn: 291490
* Replace APFloatBase static fltSemantics data members with getter functionsStephan Bergmann2016-12-141-2/+2
| | | | | | | | | | | | | At least the plugin used by the LibreOffice build (<https://wiki.documentfoundation.org/Development/Clang_plugins>) indirectly uses those members (through inline functions in LLVM/Clang include files in turn using them), but they are not exported by utils/extract_symbols.py on Windows, and accessing data across DLL/EXE boundaries on Windows is generally problematic. Differential Revision: https://reviews.llvm.org/D26671 llvm-svn: 289647
* Target: Remove unused patterns and transforms. NFC.Peter Collingbourne2016-10-071-15/+0
| | | | llvm-svn: 283515
* [NVPTX] Added intrinsics for atom.gen.{sys|cta}.* instructions.Artem Belevich2016-09-281-0/+4
| | | | | | | | These are only available on sm_60+ GPUs. Differential Revision: https://reviews.llvm.org/D24943 llvm-svn: 282607
* [NVPTX] Implement llvm.fabs.f32, llvm.max.f32, etc.Justin Lebar2016-09-091-16/+110
| | | | | | | | | | | | | | | | | | | | Summary: Previously these only worked via NVPTX-specific intrinsics. This change will allow us to convert these target-specific intrinsics into the general LLVM versions, allowing existing LLVM passes to reason about their behavior. It also gets us some minor codegen improvements as-is, from situations where we canonicalize code into one of these llvm intrinsics. Reviewers: majnemer Subscribers: llvm-commits, jholewinski, tra Differential Revision: https://reviews.llvm.org/D24300 llvm-svn: 281092
* [SelectionDAG] Rename fextend -> fpextend, fround -> fpround, frnd -> froundMichael Kuperstein2016-08-181-6/+6
| | | | | | | | | | The names of the tablegen defs now match the names of the ISD nodes. This makes the world a slightly saner place, as previously "fround" matched ISD::FP_ROUND and not ISD::FROUND. Differential Revision: https://reviews.llvm.org/D23597 llvm-svn: 279129
* [NVPTX] Fix sign/zero-extending ldg/ldu instruction selectionJustin Holewinski2016-05-021-0/+2
| | | | | | | | | | | | | | | | | Summary: We don't have sign-/zero-extending ldg/ldu instructions defined, so we need to emulate them with explicit CVTs. We were originally handling the i8 case, but not any other cases. Fixes PR26185 Reviewers: jingyue, jlebar Subscribers: jholewinski Differential Revision: http://reviews.llvm.org/D19615 llvm-svn: 268272
* [NVPTX] Set ctlz_zero_undef to Expand so LegalizeDAG will convert it to ↵Craig Topper2016-04-231-7/+0
| | | | | | ctlz. Remove the now unneccessary isel patterns. NFC llvm-svn: 267265
* [NVPTX] Handle ldg created from sign-/zero-extended loadJustin Holewinski2016-04-051-0/+10
| | | | | | | | | | Reviewers: jingyue Subscribers: jholewinski Differential Revision: http://reviews.llvm.org/D18053 llvm-svn: 265389
* [NVPTX] Annotate some instructions as hasSideEffects = 0.Justin Lebar2016-04-011-106/+124
| | | | | | | | | | | | | | | | | | | | Summary: Tablegen tries to infer this from the selection DAG patterns defined for the instructions, but it can't always. An instructive example is CLZr64. CLZr32 is correctly inferred to have no side-effects, but the selection DAG pattern for CLZr64 is slightly more complicated, and in particular the ctlz DAG node is not at the root of the pattern. Thus tablegen can't infer that CLZr64 has no side-effects. Reviewers: jholewinski Subscribers: jholewinski, tra, llvm-commits Differential Revision: http://reviews.llvm.org/D17472 llvm-svn: 265089
* [NVPTX] Annotate param loads/stores as mayLoad/mayStore.Justin Lebar2016-03-011-56/+60
| | | | | | | | | | | | | | | | | | Summary: Tablegen was unable to determine that param loads/stores were actually reading or writing from memory. I think this isn't a problem in practice for param stores, because those occur in a block right before we make our call. But param loads don't have to at the very beginning of a function, so should be annotated as mayLoad so we don't incorrectly optimize them. Reviewers: jholewinski Subscribers: jholewinski, llvm-commits Differential Revision: http://reviews.llvm.org/D17471 llvm-svn: 262381
* [NVPTX] Remove workaround for tablegen crash in NVPTXInstrInfo.td.Justin Lebar2016-03-011-28/+7
| | | | | | | | | | | | Summary: Looks like this was caused by a typo. Reviewers: jholewinski Subscribers: jholewinski, llvm-commits, tra Differential Revision: http://reviews.llvm.org/D17357 llvm-svn: 262380
* [NVPTX] Use different, convergent MIs for convergent calls.Justin Lebar2016-03-011-47/+44
| | | | | | | | | | | | | | | | | | | | | | | Summary: Calls sometimes need to be convergent. This is already handled at the LLVM IR level, but it also needs to be handled at the MI level. Ideally we'd propagate convergence from instructions, down through the selection DAG, and into MIs. But this is Hard, and would affect optimizations in the SDNs -- right now only SDNs with two operands have any flags at all. Instead, here's a much simpler hack: Add new opcodes for NVPTX for convergent calls, and generate these when lowering convergent LLVM calls. Reviewers: jholewinski Subscribers: jholewinski, chandlerc, joker.eph, jhen, tra, llvm-commits Differential Revision: http://reviews.llvm.org/D17423 llvm-svn: 262373
* [NVPTX] Nix hack used to emit '{' and '}' for NVPTX calls.Justin Lebar2016-03-011-9/+3
| | | | | | | | | | | | Summary: Tablegen understands backslash as an escape char; that's sufficient. Reviewers: jholewinski Subscribers: llvm-commits, tra, jholewinski Differential Revision: http://reviews.llvm.org/D17432 llvm-svn: 262372
* [NVPTX] Reformat NVPTXInstrInfo.td, and add additional comments.Justin Lebar2016-03-011-1418/+1400
| | | | | | | | | | | | | | | Summary: Also simplify some of the embedded C++ logic. No functional changes. Reviewers: jholewinski Subscribers: llvm-commits, tra, jholewinski Differential Revision: http://reviews.llvm.org/D17354 llvm-svn: 262371
* [NVPTX] Annotate call machine instructions as calls.Justin Lebar2016-02-171-0/+2
| | | | | | | | | | | | | | | | | Summary: Otherwise we'll try to do unsafe optimizations on these MIs, such as sinking loads below calls. (I suspect that this is not the only bug in the NVPTX instruction tablegen files; I need to comb through them.) Reviewers: jholewinski, tra Subscribers: jingyue, jhen, llvm-commits Differential Revision: http://reviews.llvm.org/D17315 llvm-svn: 261113
* Reapply r235977 "[DebugInfo] Add debug locations to constant SD nodes"Sergey Dmitrouk2015-04-281-4/+4
| | | | | | | | | | | | | | | | | | | | | | | | | [DebugInfo] Add debug locations to constant SD nodes This adds debug location to constant nodes of Selection DAG and updates all places that create constants to pass debug locations (see PR13269). Can't guarantee that all locations are correct, but in a lot of cases choice is obvious, so most of them should be. At least all tests pass. Tests for these changes do not cover everything, instead just check it for SDNodes, ARM and AArch64 where it's easy to get incorrect locations on constants. This is not complete fix as FastISel contains workaround for wrong debug locations, which drops locations from instructions on processing constants, but there isn't currently a way to use debug locations from constants there as llvm::Constant doesn't cache it (yet). Although this is a bit different issue, not directly related to these changes. Differential Revision: http://reviews.llvm.org/D9084 llvm-svn: 235989
OpenPOWER on IntegriCloud