summaryrefslogtreecommitdiffstats
path: root/llvm/lib/CodeGen/SelectionDAG/LegalizeDAG.cpp
Commit message (Collapse)AuthorAgeFilesLines
...
* [TargetLowering] Add code size information on isFPImmLegal. NFCAdhemerval Zanella2019-03-181-1/+2
| | | | | | | | | | | This allows better code size for aarch64 floating point materialization in a future patch. Reviewers: evandro Differential Revision: https://reviews.llvm.org/D58690 llvm-svn: 356389
* [SDAG][AArch64] Legalize VECREDUCENikita Popov2019-03-111-0/+31
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Fixes https://bugs.llvm.org/show_bug.cgi?id=36796. Implement basic legalizations (PromoteIntRes, PromoteIntOp, ExpandIntRes, ScalarizeVecOp, WidenVecOp) for VECREDUCE opcodes. There are more legalizations missing (esp float legalizations), but there's no way to test them right now, so I'm not adding them. This also includes a few more changes to make this work somewhat reasonably: * Add support for expanding VECREDUCE in SDAG. Usually experimental.vector.reduce is expanded prior to codegen, but if the target does have native vector reduce, it may of course still be necessary to expand due to legalization issues. This uses a shuffle reduction if possible, followed by a naive scalar reduction. * Allow the result type of integer VECREDUCE to be larger than the vector element type. For example we need to be able to reduce a v8i8 into an (nominally) i32 result type on AArch64. * Use the vector operand type rather than the scalar result type to determine the action, so we can control exactly which vector types are supported. Also change the legalize vector op code to handle operations that only have vector operands, but no vector results, as is the case for VECREDUCE. * Default VECREDUCE to Expand. On AArch64 (only target using VECREDUCE), explicitly specify for which vector types the reductions are supported. This does not handle anything related to VECREDUCE_STRICT_*. Differential Revision: https://reviews.llvm.org/D58015 llvm-svn: 355860
* [LegalizeDAG] Use APInt::getSplat helper to create bitreverse masks. NFCI.Simon Pilgrim2019-02-261-10/+6
| | | | llvm-svn: 354867
* [LegalizeDAG] Expand SADDO/SSUBO using SADDSAT/SSUBSAT (PR37763)Simon Pilgrim2019-02-261-5/+17
| | | | | | | | | | If SADDSAT/SSUBSAT are legal, then we can expand SADDO/SSUBO by performing a ADD/SUB and a SADDO/SSUBO and then compare the results. I looked at doing this for UADDO/USUBO as well but as we don't have to do as many range comparisons I didn't see any/much benefit. Differential Revision: https://reviews.llvm.org/D58637 llvm-svn: 354866
* [SDAG] Support vector UMULO/SMULONikita Popov2019-02-201-3/+5
| | | | | | | | | | | | | | | Second part of https://bugs.llvm.org/show_bug.cgi?id=40442. This adds an extra UnrollVectorOverflowOp() method to SDAG, because the general UnrollOverflowOp() method can't deal with multiple results. Additionally we need to expand UMULO/SMULO during vector op legalization, as it may result in unrolling, which may need additional type legalization. Differential Revision: https://reviews.llvm.org/D57997 llvm-svn: 354513
* [SelectionDAG] Extract [US]MULO expansion into TL method; NFCNikita Popov2019-02-171-147/+3
| | | | | | | | | | | | In preparation for supporting vector expansion. Add an isPostTypeLegalization flag to makeLibCall(), because this expansion relies on the legalized form using MERGE_VALUES. Drop the corresponding variant of ExpandLibCall, which is no longer used. Differential Revision: https://reviews.llvm.org/D58006 llvm-svn: 354226
* [SelectionDAG] Fix return calling convention in expansion of ?MULOwhitequark2019-02-121-3/+9
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Summary: The SMULO/UMULO DAG nodes, when not directly supported by the target, expand to a multiplication twice as wide. In case that the resulting type is not legal, the legalizer cannot directly call the intrinsic with the wide arguments; instead, it "pre-lowers" them by splitting them in halves. rL283203 made sure that on big endian targets, the legalizer passes the argument halves in the correct order. It did not do the same for the return value halves because the existing code used a hack; it put an illegal type into DAG and hoped that nothing would break and it would be correctly lowered elsewhere. rL307207 fixed this, handling return value halves similar to how argument handles are handled, but did not take big-endian targets into account. This commit fixes the expansion on big-endian targets, such as the out-of-tree OR1K target. Reviewers: eli.friedman, vadimcn Subscribers: george-hopkins, efriedma, llvm-commits Differential Revision: https://reviews.llvm.org/D45355 llvm-svn: 353854
* Revert "[SelectionDAG] Extract [US]MULO expansion into TL method; NFC"Nikita Popov2019-02-091-3/+141
| | | | | | | | This reverts commit r353611. Triggers an assertion during the libcall expansion on ARM. llvm-svn: 353612
* [SelectionDAG] Extract [US]MULO expansion into TL method; NFCNikita Popov2019-02-091-141/+3
| | | | | | | | | In preparation for supporting vector expansion. Also drop a variant of ExpandLibCall, of which the MULO expansions were the only user. llvm-svn: 353611
* [Intrinsic] Unsigned Fixed Point Multiplication IntrinsicLeonard Chan2019-02-041-1/+3
| | | | | | | | | | | | | Add an intrinsic that takes 2 unsigned integers with the scale of them provided as the third argument and performs fixed point multiplication on them. This is a part of implementing fixed point arithmetic in clang where some of the more complex operations will be implemented as intrinsics. Differential Revision: https://reviews.llvm.org/D55625 llvm-svn: 353059
* [TargetLowering] Rename getExpandedFixedPointMultiplication to ↵Simon Pilgrim2019-01-241-1/+1
| | | | | | | | expandFixedPointMul. NFCI. Match the (much shorter) name used in various legalization methods. llvm-svn: 352056
* Update the file headers across all of the LLVM projects in the monorepoChandler Carruth2019-01-191-4/+3
| | | | | | | | | | | | | | | | | to reflect the new license. We understand that people may be surprised that we're moving the header entirely to discuss the new license. We checked this carefully with the Foundation's lawyer and we believe this is the correct approach. Essentially, all code in the project is now made available by the LLVM project under our new license, so you will see that the license headers include that license only. Some of our contributors have contributed code under our old license, and accordingly, we have retained a copy of our old license notice in the top-level files in each project and repository. llvm-svn: 351636
* Allow FP types for atomicrmw xchgMatt Arsenault2019-01-171-0/+18
| | | | llvm-svn: 351427
* [LegalizeDAG] Remove 'NeedInvert' code from expansion of BR_CC. Replace with ↵Craig Topper2019-01-131-4/+1
| | | | | | | | | | | | an assert. I accidentally triggered this code while doing some experiments and it doesn't look lke it could possibly work. It calls 'getNOT' on a node that should be a CondCode. I think to do this right we would need to swap the branch target and the fallthrough target. But that's not easy to do. Or we could create an explicit SetCC and feed that into a new BR_CC? llvm-svn: 351022
* [X86] Rename overly verbose method; NFCNikita Popov2019-01-131-5/+3
| | | | | | As suggested on D56636. llvm-svn: 351021
* [X86][AARCH64] Improve ISD::ABS supportSimon Pilgrim2019-01-121-0/+4
| | | | | | | | This patch takes some of the code from D49837 to allow us to enable ISD::ABS support for all SSE vector types. Differential Revision: https://reviews.llvm.org/D56544 llvm-svn: 350998
* [TargetLowering] Add ISD::ROTL/ROTR vector expansionSimon Pilgrim2018-12-131-40/+5
| | | | | | | | | | Move existing rotation expansion code into TargetLowering and set it up for vectors as well. Ideally this would share more of the funnel shift expansion, but we handle the shift amount modulo quite differently at the moment. Begun removing x86 vector rotate custom lowering to use the expansion. llvm-svn: 349025
* [Intrinsic] Signed Fixed Point Multiplication IntrinsicLeonard Chan2018-12-121-0/+10
| | | | | | | | | | | | Add an intrinsic that takes 2 signed integers with the scale of them provided as the third argument and performs fixed point multiplication on them. This is a part of implementing fixed point arithmetic in clang where some of the more complex operations will be implemented as intrinsics. Differential Revision: https://reviews.llvm.org/D54719 llvm-svn: 348912
* [SelectionDAG] Initial support for FSHL/FSHR funnel shift opcodes (PR39467)Simon Pilgrim2018-12-051-0/+7
| | | | | | | | | | This is an initial patch to add a minimum level of support for funnel shifts to the SelectionDAG and to begin wiring it up to the X86 SHLD/SHRD instructions. Some partial legalization code has been added to handle the case for 'SlowSHLD' where we want to expand instead and I've added a few DAG combines so we don't get regressions from the existing DAG builder expansion code. Differential Revision: https://reviews.llvm.org/D54698 llvm-svn: 348353
* [FPEnv] Add constrained CEIL/FLOOR/ROUND/TRUNC intrinsicsCameron McInally2018-11-051-0/+8
| | | | | | Differential Revision: https://reviews.llvm.org/D53411 llvm-svn: 346141
* [COFF, ARM64] Implement Intrinsic.sponentry for AArch64Mandeep Singh Grang2018-11-011-0/+1
| | | | | | | | | | | | | | | | Summary: This patch adds Intrinsic.sponentry. This intrinsic is required to correctly support setjmp for AArch64 Windows platform. Patch by: Yin Ma (yinma@codeaurora.org) Reviewers: mgrang, ssijaric, eli.friedman, TomTan, mstorsjo, rnk, compnerd, efriedma Reviewed By: efriedma Subscribers: efriedma, javed.absar, kristof.beyls, chrib, llvm-commits Differential Revision: https://reviews.llvm.org/D53996 llvm-svn: 345909
* Revert "[COFF, ARM64] Implement Intrinsic.sponentry for AArch64"Mandeep Singh Grang2018-11-011-1/+0
| | | | | | This reverts commit 585b6667b4712e3c7f32401e929855b3313b4ff2. llvm-svn: 345863
* [COFF, ARM64] Implement Intrinsic.sponentry for AArch64Mandeep Singh Grang2018-10-311-0/+1
| | | | | | | | | | | | | | Summary: This patch adds Intrinsic.sponentry. This intrinsic is required to correctly support setjmp for AArch64 Windows platform. Reviewers: mgrang, TomTan, rnk, compnerd, mstorsjo, efriedma Reviewed By: efriedma Subscribers: majnemer, chrib, javed.absar, kristof.beyls, llvm-commits Differential Revision: https://reviews.llvm.org/D53673 llvm-svn: 345791
* Fix comment typo. NFCI.Simon Pilgrim2018-10-311-1/+1
| | | | llvm-svn: 345758
* [SelectionDAG] SelectionDAGLegalize::ExpandBITREVERSE - ensure we use ShiftTySimon Pilgrim2018-10-311-6/+6
| | | | | | We should be using the getShiftAmountTy value type for shift amounts. llvm-svn: 345756
* [FPEnv] [FPEnv] Add constrained intrinsics for MAXNUM and MINNUMCameron McInally2018-10-301-0/+4
| | | | | | Differential Revision: https://reviews.llvm.org/D53216 llvm-svn: 345650
* [Intrinsic] Signed and Unsigned Saturation Subtraction IntirnsicsLeonard Chan2018-10-291-3/+7
| | | | | | | | | | | | Add an intrinsic that takes 2 integers and perform saturation subtraction on them. This is a part of implementing fixed point arithmetic in clang where some of the more complex operations will be implemented as intrinsics. Differential Revision: https://reviews.llvm.org/D53783 llvm-svn: 345512
* [TargetLowering] Move i64/vXi64 to f32/vXf32 UINT_TO_FP handling to ↵Simon Pilgrim2018-10-281-27/+0
| | | | | | TargetLowering::expandUINT_TO_FP. llvm-svn: 345478
* [TargetLowering] Move LegalizeDAG FP_TO_UINT handling to ↵Simon Pilgrim2018-10-271-22/+3
| | | | | | | | TargetLowering::expandFP_TO_UINT. NFCI. First step towards fixing PR17686 and adding vector support. llvm-svn: 345452
* [LegalizeDAG] Remove dead SINT_TO_FP legalization codeSimon Pilgrim2018-10-251-54/+19
| | | | | | | | As noticed on D52965, the SINT_TO_FP i64 to f32 legalization code has been dead for years - protected by an assert. Differential Revision: https://reviews.llvm.org/D53703 llvm-svn: 345290
* Missing semicolon.Simon Pilgrim2018-10-251-1/+1
| | | | llvm-svn: 345257
* [TargetLowering] Improve vXi64 UINT_TO_FP vXf64 support (P38226)Simon Pilgrim2018-10-251-25/+6
| | | | | | | | | | | | As suggested on D52965, this patch moves the i64 to f64 UINT_TO_FP expansion code from LegalizeDAG into TargetLowering and makes it available to LegalizeVectorOps as well. Not only does this help perform X86 lowering as a true vectorization instead of (partially vectorized) scalar conversions, it avoids the HADDPD op from the scalar code which can be slow on most targets. The AVX512F does have the vcvtusi2sdq scalar operation but we don't unroll to use it as it seems to only help for the v2f64 case - otherwise the unrolling cost will certainly be too high. My feeling is that we should leave it to the vectorizers - and if it generates the vector UINT_TO_FP we should use it. Differential Revision: https://reviews.llvm.org/D53649 llvm-svn: 345256
* [LegalizeDAG] ExpandLegalINT_TO_FP - cleanup UINT_TO_FP i64 -> f32 expansion.Simon Pilgrim2018-10-241-11/+12
| | | | | | | | Use SrcVT/DestVT types and correct shift type. Part of prep work for D52965 llvm-svn: 345158
* [LegalizeDAG] Share Vector/Scalar CTPOP ExpansionSimon Pilgrim2018-10-231-58/+2
| | | | | | | | As suggested on D53258, this patch move the CTPOP expansion code from SelectionDAGLegalize to TargetLowering to allow it to be reused by the VectorLegalizer. Proper vector support will be added by D53258. llvm-svn: 345066
* [LegalizeDAG] Share Vector/Scalar CTLZ ExpansionSimon Pilgrim2018-10-231-32/+5
| | | | | | | | As suggested on D53258, this patch shares common CTLZ expansion code between VectorLegalizer and SelectionDAGLegalize by putting it in TargetLowering. Extension to D53474 llvm-svn: 345060
* [LegalizeDAG] Share Vector/Scalar CTTZ ExpansionSimon Pilgrim2018-10-231-31/+5
| | | | | | | | | | As suggested on D53258, this patch demonstrates sharing common CTTZ expansion code between VectorLegalizer and SelectionDAGLegalize by putting it in TargetLowering. I intend to move CTLZ and (scalar) CTPOP over as well and then update D53258 accordingly. Differential Revision: https://reviews.llvm.org/D53474 llvm-svn: 345039
* [Intrinsic] Unigned Saturation Addition IntrinsicLeonard Chan2018-10-221-3/+5
| | | | | | | | | | | | Add an intrinsic that takes 2 integers and perform unsigned saturation addition on them. This is a part of implementing fixed point arithmetic in clang where some of the more complex operations will be implemented as intrinsics. Differential Revision: https://reviews.llvm.org/D53340 llvm-svn: 344971
* DAG: Change behavior of fminnum/fmaxnum nodesMatt Arsenault2018-10-221-1/+6
| | | | | | | | | | | Introduce new versions that follow the IEEE semantics to help with legalization that may need quieted inputs. There are some regressions from inserting unnecessary canonicalizes when these are matched from fast math fcmp + select which should be fixed in a future commit. llvm-svn: 344914
* [Intrinsic] Signed Saturation Addition IntrinsicLeonard Chan2018-10-161-0/+8
| | | | | | | | | | | Add an intrinsic that takes 2 integers and perform saturation addition on them. This is a part of implementing fixed point arithmetic in clang where some of the more complex operations will be implemented as intrinsics. Differential Revision: https://reviews.llvm.org/D53053 llvm-svn: 344629
* [LegalizeDAG] ExpandLegalINT_TO_FP - cleanup UINT_TO_FP i64 -> f64 expansion.Simon Pilgrim2018-10-161-20/+17
| | | | | | | | Use SrcVT/DestVT types, correct shift type and AND instead of ZERO_EXTEND_IN_REG. Part of prep work for D52965 llvm-svn: 344602
* [LegalizeDAG] Don't bother with final MUL+SRL stage for byte CTPOP. Simon Pilgrim2018-10-141-3/+4
| | | | | | | | The final stage of CTPOP expansion (v = (v * 0x01010101...) >> (Len - 8)) is completely pointless for the byte (Len = 8) case as it reduces to (v = (v * 0x01...) >> 0), but annoyingly this doesn't always get optimized away. Found while investigating generic vector CTPOP expansion (PR32655). llvm-svn: 344477
* Pull out repeated variables from SelectionDAGLegalize::ExpandBitCount.Simon Pilgrim2018-10-131-8/+2
| | | | | | The CTPOP case has been changed from VT.getSizeInBits to VT.getScalarSizeInBits - but this fits in with future work for vector support (PR32655) and doesn't affect any current (scalar) uses. llvm-svn: 344461
* [X86][SSE] Remove most of vector CTTZ custom lowering and use LegalizeDAG ↵Simon Pilgrim2018-10-131-2/+2
| | | | | | | | | | instead. There is one remnant - AVX1 custom splitting of 256-bit vectors - which is due to a regression where the X86ISD::ANDNP is still performed as a YMM. I've also tightened the CTLZ or CTPOP lowering in SelectionDAGLegalize::ExpandBitCount to require a legal CTLZ - it doesn't affect existing users and fixes an issue with AVX512 codegen. llvm-svn: 344457
* [X86][SSE] Begin removing vector CTTZ custom lowering and use LegalizeDAG ↵Simon Pilgrim2018-10-131-1/+1
| | | | | | | | instead. Adds CTTZ vector legalization support and begins the removal of the X86/SSE custom lowering. llvm-svn: 344453
* Pull out repeated value types. NFCI.Simon Pilgrim2018-10-121-6/+5
| | | | llvm-svn: 344355
* [SelectionDAG] Move VectorLegalizer::ExpandCTLZ codegen into ↵Simon Pilgrim2018-10-121-1/+1
| | | | | | | | SelectionDAGLegalize Generalize SelectionDAGLegalize's CTLZ expansion to handle vectors - lets VectorLegalizer::ExpandCTLZ to just pass the expansion on instead of repeating the same codegen. llvm-svn: 344349
* Revert r343948 "[LegalizeDAG] Make one of the ReplaceNode signatures take an ↵Craig Topper2018-10-081-8/+6
| | | | | | | | ArrayRef instead a pointer to an array. Add assert on size of array. NFC" The assert is failing some asan tests on the bots. llvm-svn: 343950
* [LegalizeDAG] Make one of the ReplaceNode signatures take an ArrayRef ↵Craig Topper2018-10-081-6/+8
| | | | | | instead a pointer to an array. Add assert on size of array. NFC llvm-svn: 343948
* [LegalizeDAG] Move legalization of scatter and masked store from ↵Craig Topper2018-10-081-0/+8
| | | | | | | | | | LegalizeVectorOps to LegalizeDAG. This is where we legalize gather and masked load so this is consistent. Since these ops are always on vectors I've chosen to go with LegalizeDAG since that's what we do for other vector only ops like BUILD_VECTOR, VECTOR_SHUFFLE, etc. The ScalarizeMaskedMemIntrinsic pass should take care of scalarizing these before SelectionDAG so hopefully we don't need to worry about illegally typed scalar ops being emitted in the legalizing. If we did we would need to do this in LegalizeVectorOps so we could get the second type legalization that runs between LegalizeVectorOps and LegalizeDAG. llvm-svn: 343947
* [DAG] SelectionDAGLegalize::ExpandLegalINT_TO_FP - use getFPExtendOrRound ↵Simon Pilgrim2018-09-261-11/+1
| | | | | | | | helper. NFCI. Handles SrcVT == DstVT as well. llvm-svn: 343121
OpenPOWER on IntegriCloud