summaryrefslogtreecommitdiffstats
path: root/llvm/lib/CodeGen/SelectionDAG/LegalizeVectorOps.cpp
Commit message (Collapse)AuthorAgeFilesLines
...
* [FPEnv] [FPEnv] Add constrained intrinsics for MAXNUM and MINNUMCameron McInally2018-10-301-0/+4
| | | | | | Differential Revision: https://reviews.llvm.org/D53216 llvm-svn: 345650
* [Intrinsic] Signed and Unsigned Saturation Subtraction IntirnsicsLeonard Chan2018-10-291-0/+2
| | | | | | | | | | | | Add an intrinsic that takes 2 integers and perform saturation subtraction on them. This is a part of implementing fixed point arithmetic in clang where some of the more complex operations will be implemented as intrinsics. Differential Revision: https://reviews.llvm.org/D53783 llvm-svn: 345512
* [VectorLegalizer] Enable TargetLowering::expandFP_TO_UINT support.Simon Pilgrim2018-10-281-3/+16
| | | | | | | | Add vector support to TargetLowering::expandFP_TO_UINT. This exposes an issue in X86TargetLowering::LowerVSELECT which was assuming that the select mask was the same width as the LHS/RHS ops - as long as the result is a sign splat we can easily sext/trunk this. llvm-svn: 345473
* [TargetLowering] Improve vXi64 UINT_TO_FP vXf64 support (P38226)Simon Pilgrim2018-10-251-0/+5
| | | | | | | | | | | | As suggested on D52965, this patch moves the i64 to f64 UINT_TO_FP expansion code from LegalizeDAG into TargetLowering and makes it available to LegalizeVectorOps as well. Not only does this help perform X86 lowering as a true vectorization instead of (partially vectorized) scalar conversions, it avoids the HADDPD op from the scalar code which can be slow on most targets. The AVX512F does have the vcvtusi2sdq scalar operation but we don't unroll to use it as it seems to only help for the v2f64 case - otherwise the unrolling cost will certainly be too high. My feeling is that we should leave it to the vectorizers - and if it generates the vector UINT_TO_FP we should use it. Differential Revision: https://reviews.llvm.org/D53649 llvm-svn: 345256
* [NFC] Rename minnan and maxnan to minimum and maximumThomas Lively2018-10-241-2/+2
| | | | | | | | | | | | | | | Summary: Changes all uses of minnan/maxnan to minimum/maximum globally. These names emphasize that the semantic difference between these operations is more than just NaN-propagation. Reviewers: arsenm, aheejin, dschuff, javed.absar Subscribers: jholewinski, sdardis, wdng, sbc100, jgravelle-google, jrtc27, atanasyan, llvm-commits Differential Revision: https://reviews.llvm.org/D53112 llvm-svn: 345218
* [LegalizeDAG] Share Vector/Scalar CTLZ ExpansionSimon Pilgrim2018-10-231-17/+4
| | | | | | | | As suggested on D53258, this patch shares common CTLZ expansion code between VectorLegalizer and SelectionDAGLegalize by putting it in TargetLowering. Extension to D53474 llvm-svn: 345060
* [LegalizeDAG] Remove unused variableBenjamin Kramer2018-10-231-2/+0
| | | | llvm-svn: 345040
* [LegalizeDAG] Share Vector/Scalar CTTZ ExpansionSimon Pilgrim2018-10-231-16/+4
| | | | | | | | | | As suggested on D53258, this patch demonstrates sharing common CTTZ expansion code between VectorLegalizer and SelectionDAGLegalize by putting it in TargetLowering. I intend to move CTLZ and (scalar) CTPOP over as well and then update D53258 accordingly. Differential Revision: https://reviews.llvm.org/D53474 llvm-svn: 345039
* [Intrinsic] Unigned Saturation Addition IntrinsicLeonard Chan2018-10-221-0/+1
| | | | | | | | | | | | Add an intrinsic that takes 2 integers and perform unsigned saturation addition on them. This is a part of implementing fixed point arithmetic in clang where some of the more complex operations will be implemented as intrinsics. Differential Revision: https://reviews.llvm.org/D53340 llvm-svn: 344971
* DAG: Change behavior of fminnum/fmaxnum nodesMatt Arsenault2018-10-221-0/+12
| | | | | | | | | | | Introduce new versions that follow the IEEE semantics to help with legalization that may need quieted inputs. There are some regressions from inserting unnecessary canonicalizes when these are matched from fast math fcmp + select which should be fixed in a future commit. llvm-svn: 344914
* [Intrinsic] Signed Saturation Addition IntrinsicLeonard Chan2018-10-161-0/+1
| | | | | | | | | | | Add an intrinsic that takes 2 integers and perform saturation addition on them. This is a part of implementing fixed point arithmetic in clang where some of the more complex operations will be implemented as intrinsics. Differential Revision: https://reviews.llvm.org/D53053 llvm-svn: 344629
* [X86][SSE] Begin removing vector CTTZ custom lowering and use LegalizeDAG ↵Simon Pilgrim2018-10-131-3/+15
| | | | | | | | instead. Adds CTTZ vector legalization support and begins the removal of the X86/SSE custom lowering. llvm-svn: 344453
* Pull out repeated value types. NFCI.Simon Pilgrim2018-10-121-3/+5
| | | | llvm-svn: 344354
* [SelectionDAG] Move VectorLegalizer::ExpandCTLZ codegen into ↵Simon Pilgrim2018-10-121-23/+4
| | | | | | | | SelectionDAGLegalize Generalize SelectionDAGLegalize's CTLZ expansion to handle vectors - lets VectorLegalizer::ExpandCTLZ to just pass the expansion on instead of repeating the same codegen. llvm-svn: 344349
* [LegalizeDAG] Move legalization of scatter and masked store from ↵Craig Topper2018-10-081-11/+2
| | | | | | | | | | LegalizeVectorOps to LegalizeDAG. This is where we legalize gather and masked load so this is consistent. Since these ops are always on vectors I've chosen to go with LegalizeDAG since that's what we do for other vector only ops like BUILD_VECTOR, VECTOR_SHUFFLE, etc. The ScalarizeMaskedMemIntrinsic pass should take care of scalarizing these before SelectionDAG so hopefully we don't need to worry about illegally typed scalar ops being emitted in the legalizing. If we did we would need to do this in LegalizeVectorOps so we could get the second type legalization that runs between LegalizeVectorOps and LegalizeDAG. llvm-svn: 343947
* [LegalizeVectorOps] Make ExpandStrictFPOp return the result corresponding to ↵Craig Topper2018-10-071-1/+1
| | | | | | | | the result number of the SDValue passed in. It was always returning the chain which seems to be the result number of the SDValue in the lit tests we have. But I don't know if that's guaranteed. llvm-svn: 343933
* [X86][LegalizeVectorOps] Use MERGE_VALUES to return two results from ↵Craig Topper2018-10-041-11/+3
| | | | | | | | | | LowerLoad. Remove special case code in LegalizeVectorOps that allowed us to only return one result. Previously we replaced the chain use ourself and return the data result. LegalizeVectorOps then detected that we'd done this and assumed the chain had already been handled. This commit instead returns a MERGE_VALUES node with two results joined from nodes. This allows LegalizeVectorOps to do all the replacements for us without any special casing. The MERGE_VALUES will be removed by DAG combine. llvm-svn: 343817
* [FPEnv] Support constrained FREM intrinsicCameron McInally2018-08-201-0/+2
| | | | | | Differential Revision: https://reviews.llvm.org/D50975 llvm-svn: 340201
* Remove trailing spaceFangrui Song2018-07-301-9/+9
| | | | | | sed -Ei 's/[[:space:]]+$//' include/**/*.{def,h,td} lib/**/*.{cpp,h} llvm-svn: 338293
* Fix corruption of result number in LegalizeVectorOps.cppUlrich Weigand2018-07-251-1/+2
| | | | | | | | | | | | When VectorLegalizer::LegalizeOp creates a new SDValue after iterating over its arguments, we need to refer to the same result number of the new node that the original value used. Reviewed by: cameron.mcinally Differential Revision: https://reviews.llvm.org/D49805 llvm-svn: 337939
* [Legalize] Elide MERGE_VALUES created by scalarizeVectorLoad.Nirav Dave2018-07-231-3/+8
| | | | | | | scalarizeVectorLoad creates MERGE_VALUES nodes which are immediately decomposed in expandLoad. Elide the node in these cases. llvm-svn: 337708
* [FPEnv] Expand constrained FP POWICameron McInally2018-06-151-4/+18
| | | | | | | | | | | | Modify ExpandStrictFPOp(...) to handle nodes that have scalar operands. Also, add a Strict FMA test and do some other light cleanup in the Strict FP code. Differential Revision: https://reviews.llvm.org/D48149 llvm-svn: 334863
* [FPEnv] Expand constrained FP operationsCameron McInally2018-06-131-8/+88
| | | | | | | | | | Add a helper function to expand constrained FP operations as needed. Note that the Strict POWI operation is not handled in this patch since the format is slightly different from the others. Differential Revision: https://reviews.llvm.org/D47491 llvm-svn: 334603
* Rename DEBUG macro to LLVM_DEBUG.Nicola Zaghen2018-05-141-8/+9
| | | | | | | | | | | | | | | | The DEBUG() macro is very generic so it might clash with other projects. The renaming was done as follows: - git grep -l 'DEBUG' | xargs sed -i 's/\bDEBUG\s\?(/LLVM_DEBUG(/g' - git diff -U0 master | ../clang/tools/clang-format/clang-format-diff.py -i -p1 -style LLVM - Manual change to APInt - Manually chage DOCS as regex doesn't match it. In the transition period the DEBUG() macro is still present and aliased to the LLVM_DEBUG() one. Differential Revision: https://reviews.llvm.org/D43624 llvm-svn: 332240
* Remove \brief commands from doxygen comments.Adrian Prantl2018-05-011-16/+16
| | | | | | | | | | | | | | | | We've been running doxygen with the autobrief option for a couple of years now. This makes the \brief markers into our comments redundant. Since they are a visual distraction and we don't want to encourage more \brief markers in new code either, this patch removes them all. Patch produced by for i in $(git grep -l '\\brief'); do perl -pi -e 's/\\brief //g' $i & done Differential Revision: https://reviews.llvm.org/D46290 llvm-svn: 331272
* DAG: Fix not legalizing vector fcanonicalizesMatt Arsenault2018-04-261-0/+1
| | | | | | If an fcanoncialize was done on a vector type that was legal, llvm-svn: 330981
* [IR][CodeGen] Remove dependency on EVT from IR/Function.cpp. Move EVT to ↵Craig Topper2018-03-291-1/+1
| | | | | | | | | | | | CodeGen layer. Currently EVT is in the IR layer only because of Function.cpp needing a very small piece of the functionality of EVT::getEVTString(). The rest of EVT is used in codegen making CodeGen a better place for it. The previous code converted a Type* to EVT and then called getEVTString. This was only expected to handle the primitive types from Type*. Since there only a few primitive types, we can just print them as strings directly. Differential Revision: https://reviews.llvm.org/D45017 llvm-svn: 328806
* Fix layering by moving ValueTypes.h from CodeGen to IRDavid Blaikie2018-03-231-1/+1
| | | | | | ValueTypes.h is implemented in IR already. llvm-svn: 328397
* Fix layering of MachineValueType.h by moving it from CodeGen to SupportDavid Blaikie2018-03-231-1/+1
| | | | | | | | | This is used by llvm tblgen as well as by LLVM Targets, so the only common place is Support for now. (maybe we need another target for these sorts of things - but for now I'm at least making them correct & we can make them better if/when people have strong feelings) llvm-svn: 328395
* [VectorLegalizer] Fix uint64_t typo in ExpandUINT_TO_FLOAT (PR36391)Simon Pilgrim2018-02-201-1/+1
| | | | | | | | ExpandUINT_TO_FLOAT can accept vXi32 or vXi64 inputs, so we need to use a uint64_t shift to generate the 2^(BW/2) constant. No test case unfortunately as no upstream target uses this, but its affecting a downstream target. llvm-svn: 325578
* [SelectionDAG] Fix codegen of vector stores with non byte-sized elements.Jonas Paulsson2018-01-201-1/+0
| | | | | | | | | | | | | | This was completely broken, but hopefully fixed by this patch. In cases where it is needed, a vector with non byte-sized elements is stored by extracting, zero-extending, shift:ing and or:ing the elements into an integer of the same width as the vector, which is then stored. Review: Eli Friedman, Ulrich Weigand https://reviews.llvm.org/D42100#inline-369520 https://bugs.llvm.org/show_bug.cgi?id=35520 llvm-svn: 323042
* [VectorLegalizer] Remove broken code in ExpandStore.Jonas Paulsson2018-01-111-28/+0
| | | | | | | | | | | | | | The code that is supposed to "Round odd types to the next pow of two" seems broken and as well completely unused (untested). It also seems that ExpandStore really shouldn't ever change the memory VT, which this in fact does. As a first step in fixing the broken handling of vector stores (of irregular types, e.g. an i1 vector), this code is removed. For discussion, see https://bugs.llvm.org/show_bug.cgi?id=35520. Review: Eli Friedman llvm-svn: 322275
* [SelectionDAG][X86][AArch64] Require targets to specify the promotion type ↵Craig Topper2018-01-011-37/+21
| | | | | | | | | | | | | | | | when using setOperationAction Promote for INT_TO_FP and FP_TO_INT Currently the promotion for these ignores the normal getTypeToPromoteTo and instead just tries to double the element width. This is because the default behavior of getTypeToPromote to just adds 1 to the SimpleVT, which has the affect of increasing the element count while keeping the scalar size the same. If multiple steps are required to get to a legal operation type, int_to_fp will be promoted multiple times. And fp_to_int will keep trying wider types in a loop until it finds one that works. getTypeToPromoteTo does have the ability to query a promotion map to get the type and not do the increasing behavior. It seems better to just let the target specify the promotion type in the map explicitly instead of letting the legalizer iterate via widening. FWIW, it's worth I think for any other vector operations that need to be promoted, we have to specify the type explicitly because the default behavior of getTypeToPromote isn't useful for vectors. The other types of promotion already require either the element count is constant or the total vector width is constant, but neither happens by incrementing the SimpleVT enum. Differential Revision: https://reviews.llvm.org/D40664 llvm-svn: 321629
* [SelectionDAG] Add some debug print messages to LegalizeVectorOps.Craig Topper2017-12-281-2/+15
| | | | llvm-svn: 321535
* [SelectionDAG][X86] Teach promotion legalization for fp_to_sint/fp_to_uint ↵Craig Topper2017-11-291-3/+11
| | | | | | | | | | | | to insert an assertsext/assertzext based on the original type If we put in an assertsext/zext here, we're able to generate better truncate code using pack on pre-avx512 targets. Similar is already done during type legalization. This is the equivalent for op legalization Differential Revision: https://reviews.llvm.org/D40591 llvm-svn: 319368
* DAG: Add nuw when splitting loads and storesMatt Arsenault2017-11-291-4/+2
| | | | | | | | | | | The object can't straddle the address space wrap around, so I think it's OK to assume any offsets added to the base object pointer can't overflow. Similar logic already appears to be applied in SelectionDAGBuilder when lowering aggregate returns. llvm-svn: 319272
* [X86] Mark ISD::FP_TO_UINT v16i8/v16i16 as Promote under AVX512 instead of ↵Craig Topper2017-11-281-2/+2
| | | | | | | | | | legal. Fix infinite loop in op legalization when promotion requires 2 steps. Previously we had an isel pattern to add the truncate. Instead use Promote to add the truncate to the DAG before isel. The Promote legalization code had to be updated to prevent an infinite loop if promotion took multiple steps because it wasn't remembering the previously tried value. llvm-svn: 319259
* Fix a bunch more layering of CodeGen headers that are in TargetDavid Blaikie2017-11-171-1/+1
| | | | | | | | All these headers already depend on CodeGen headers so moving them into CodeGen fixes the layering (since CodeGen depends on Target, not the other way around). llvm-svn: 318490
* [CodeGen] Fix some Clang-tidy modernize-use-bool-literals and Include What ↵Eugene Zelenko2017-09-211-9/+27
| | | | | | You Use warnings; other minor fixes (NFC). llvm-svn: 313941
* Added LLVM_FALLTHROUGH to address warning: this statement may fall through. NFC.Galina Kistanova2017-06-031-0/+1
| | | | llvm-svn: 304635
* DAG: Do not scalarize fsub if fneg is legalMatt Arsenault2017-02-151-0/+15
| | | | | | Tests will be included with future commit. llvm-svn: 295242
* Use SelectionDAG::getBuildVector/getSplatBuildVector helper functions where ↵Simon Pilgrim2017-01-301-7/+3
| | | | | | possible. NFCI. llvm-svn: 293520
* [SelectionDAG] Add expansion and promotion of [US]MUL_LOHINicolai Haehnle2016-12-081-0/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | Summary: Most targets set the action for these nodes to Expand even though there isn't actually any code for them in ExpandNode. Instead, targets simply relied on the fact that no code generates these nodes as long as the nodes aren't legal or custom. However, generating these nodes can be useful e.g. for divide-by-constant in wider integer types. Expand of [US]MUL_LOHI will use MULH[US] when legal or custom, and a sequence of half-width multiplications otherwise. Promote uses a wider multiply. This patch intends to not change the generated code, but indirect effects are possible since expansions/promotions that were previously done in DAGCombine may now be done in LegalizeDAG. See D24822 for a change that actually uses the new expansion. Reviewers: spatel, bkramer, venkatra, efriedma, hfinkel, ast, nadav, tstellarAMD Subscribers: arsenm, jyknight, nemanjai, wdng, nhaehnle, llvm-commits Differential Revision: https://reviews.llvm.org/D24956 llvm-svn: 289050
* [VectorLegalizer] Remove EVT::getSizeInBits code duplications. NFCI.Simon Pilgrim2016-11-211-8/+6
| | | | | | We were calling SVT.getSizeInBits() several times in a row - just call it once and reuse the result. llvm-svn: 287556
* [VectorLegalizer] Expansion of CTLZ using CTPOP when possibleSimon Pilgrim2016-11-081-6/+50
| | | | | | | | | | This patch avoids scalarization of CTLZ by instead expanding to use CTPOP (ref: "Hacker's Delight") when the necessary operations are available. This also adds the necessary cost models for X86 SSE2 targets (the main beneficiary) to ensure vectorization only happens when its useful. Differential Revision: https://reviews.llvm.org/D25910 llvm-svn: 286233
* getVectorElementType().getSizeInBits() -> getScalarSizeInBits() ; NFCISanjay Patel2016-09-141-2/+2
| | | | llvm-svn: 281495
* getValueType().getSizeInBits() -> getValueSizeInBits() ; NFCISanjay Patel2016-09-141-1/+1
| | | | llvm-svn: 281493
* getScalarType().getSizeInBits() -> getScalarSizeInBits() ; NFCISanjay Patel2016-09-141-3/+3
| | | | llvm-svn: 281489
* Replace a few more "fall through" comments with LLVM_FALLTHROUGHJustin Bogner2016-08-171-1/+1
| | | | | | Follow up to r278902. I had missed "fall through", with a space. llvm-svn: 278970
* [SelectionDAG] Get rid of bool parameters in SelectionDAG::getLoad, ↵Justin Lebar2016-07-151-19/+14
| | | | | | | | | | | | | | | | | | | | | | | getStore, and friends. Summary: Instead, we take a single flags arg (a bitset). Also add a default 0 alignment, and change the order of arguments so the alignment comes before the flags. This greatly simplifies many callsites, and fixes a bug in AMDGPUISelLowering, wherein the order of the args to getLoad was inverted. It also greatly simplifies the process of adding another flag to getLoad. Reviewers: chandlerc, tstellarAMD Subscribers: jholewinski, arsenm, jyknight, dsanders, nemanjai, llvm-commits Differential Revision: http://reviews.llvm.org/D22249 llvm-svn: 275592
OpenPOWER on IntegriCloud