summaryrefslogtreecommitdiffstats
path: root/llvm/lib/Transforms/Vectorize
Commit message (Collapse)AuthorAgeFilesLines
...
* [SLPVectorizer] Fix alternate opcode + shuffle cost function to correct ↵Simon Pilgrim2018-07-021-4/+3
| | | | | | | | | | handle SK_Select patterns. We were always using the opcodes of the first 2 scalars for the costs of the alternate opcode + shuffle. This made sense when we used SK_Alternate and opcodes were guaranteed to be alternating, but this fails for the more general SK_Select case. This fix exposes an issue demonstrated by the fmul_fdiv_v4f32_const test - the SLM model has v4f32 fdiv costs which are more than twice those of the f32 scalar cost, meaning that the cost model determines that the vectorization is not performant. Unfortunately it completely ignores the fact that the fdiv by a constant will be changed into a fmul by InstCombine for a much lower cost vectorization. But at least we're seeing this now... llvm-svn: 336095
* [SLPVectorizer] Only Alternate opcodes use ShuffleVector cases for ↵Simon Pilgrim2018-07-021-1/+5
| | | | | | | | getEntryCost/vectorizeTree. NFCI. Add assertions - we're already assuming this in how we use the AltOpcode and treat everything as BinaryOperators. llvm-svn: 336092
* [SLPVectorizer] Call InstructionsState.isOpcodeOrAlt with Instruction ↵Simon Pilgrim2018-07-011-11/+9
| | | | | | instead of an opcode. NFCI. llvm-svn: 336069
* [SLPVectorizer] Replace sameOpcodeOrAlt with InstructionsState.isOpcodeOrAlt ↵Simon Pilgrim2018-07-011-12/+10
| | | | | | | | helper. NFCI. This is a basic step towards matching more general instructions types than just opcodes. llvm-svn: 336068
* [SLPVectorizer] Use InstructionsState Op/Alt opcodes directly. NFCI.Simon Pilgrim2018-07-011-4/+2
| | | | llvm-svn: 336063
* [SLPVectorizer] Recognise non uniform power of 2 constantsSimon Pilgrim2018-06-261-12/+11
| | | | | | | | | | Since D46637 we are better at handling uniform/non-uniform constant Pow2 detection; this patch tweaks the SLP argument handling to support them. As SLP works with arrays of values I don't think we can easily use the pattern match helpers here. Differential Revision: https://reviews.llvm.org/D48214 llvm-svn: 335621
* [SLPVectorizer] Support alternate opcodes in tryToVectorizeListSimon Pilgrim2018-06-221-28/+13
| | | | | | | | | | Enable tryToVectorizeList to support InstructionsState alternate opcode patterns at a root (build vector etc.) as well as further down the vectorization tree. NOTE: This patch reduces some of the debug reporting if there are opcode mismatches - I can try to add it back if it proves a problem. But it could get rather messy trying to provide equivalent verbose debug strings via getSameOpcode etc. Differential Revision: https://reviews.llvm.org/D48488 llvm-svn: 335364
* [SLPVectorizer] reorderAltShuffleOperands should just take ↵Simon Pilgrim2018-06-221-7/+5
| | | | | | | | InstructionsState. NFCI. All calls were extracting the InstructionsState Opcode/AltOpcode values so we might as well pass it directly llvm-svn: 335359
* [SLPVectorizer] Relax alternate opcodes to accept any BinaryOperator pairSimon Pilgrim2018-06-221-27/+11
| | | | | | | | | | SLP currently only accepts (F)Add/(F)Sub alternate counterpart ops to be merged into an alternate shuffle. This patch relaxes this to accept any pair of BinaryOperator opcodes instead, assuming the target's cost model accepts the vectorization+shuffle. Differential Revision: https://reviews.llvm.org/D48477 llvm-svn: 335349
* [SLPVectorizer] Provide InstructionsState down the BoUpSLP vectorization ↵Simon Pilgrim2018-06-201-53/+55
| | | | | | | | | | call tree As described in D48359, this patch pushes InstructionsState down the BoUpSLP call hierarchy instead of the corresponding raw OpValue. This makes it easier to track the alternate opcode etc. and avoids us having to call getAltOpcode which makes it difficult to support more than one alternate opcode. Differential Revision: https://reviews.llvm.org/D48382 llvm-svn: 335170
* [SLPVectorizer] Move isOneOf after InstructionsState type. NFCI.Simon Pilgrim2018-06-201-71/+71
| | | | | | A future patch will have isOneOf use InstructionsState. llvm-svn: 335142
* [SLPVectorizer] Use InstructionsState to record AltOpcodeSimon Pilgrim2018-06-201-72/+73
| | | | | | | | | | | | This is part of a move towards generalizing the alternate opcode mechanism and not just supporting (F)Add/(F)Sub counterparts. The patch embeds the AltOpcode in the InstructionsState instead of calling getAltOpcode so often. I'm hoping to eventually remove all uses of getAltOpcode and handle alternate opcode selection entirely within getSameOpcode, that will require us to use InstructionsState throughout the BoUpSLP call hierarchy (similar to some of the changes in D28907), which I will begin in future patches. Differential Revision: https://reviews.llvm.org/D48359 llvm-svn: 335134
* [SLPVectorizer] Relax "alternate" opcode vectorisation to work with any ↵Simon Pilgrim2018-06-201-11/+11
| | | | | | | | | | | | | | SK_Select shuffle pattern D47985 saw the old SK_Alternate 'alternating' shuffle mask replaced with the SK_Select mask which accepts either input operand for each lane, equivalent to a vector select with a constant condition operand. This patch updates SLPVectorizer to make full use of this SK_Select shuffle pattern by removing the 'isOdd()' limitation. The AArch64 regression will be fixed by D48172. Differential Revision: https://reviews.llvm.org/D48174 llvm-svn: 335130
* [SLPVectorizer] Split Tree/Reduction cost calls to simplify debugging. NFCI.Simon Pilgrim2018-06-201-2/+3
| | | | llvm-svn: 335110
* [SLPVectorizer] Remove default OperandValueKind arguments from ↵Simon Pilgrim2018-06-191-14/+7
| | | | | | | | | | | | getArithmeticInstrCost calls (NFC) The getArithmeticInstrCost calls for shuffle vectors entry costs specify TargetTransformInfo::OperandValueKind arguments, but are just using the method's default values. This seems to be a copy + paste issue and doesn't affect the costs in anyway. The TargetTransformInfo::OperandValueProperties default arguments are already not being used. Noticed while working on D47985. Differential Revision: https://reviews.llvm.org/D48008 llvm-svn: 335045
* [SLPVectorizer] Pull out AltOpcode determination from reorderAltShuffleOperands.Simon Pilgrim2018-06-191-6/+8
| | | | | | Minor step towards making the alternate opcode system work with a wider range of opcode pairs. llvm-svn: 335032
* [VPlan] Add VPInstruction to VPRecipe transformation.Florian Hahn2018-06-184-0/+112
| | | | | | | | | | | | | | This patch introduces a VPInstructionToVPRecipe transformation, which allows us to generate code for a VPInstruction based VPlan re-using the existing infrastructure. Reviewers: dcaballe, hsaito, mssimpso, hfinkel, rengolin, mkuper, javed.absar, sguggill Reviewed By: dcaballe Differential Revision: https://reviews.llvm.org/D46827 llvm-svn: 334969
* [SLPVectorizer] Tidyup isShuffle helperSimon Pilgrim2018-06-181-31/+10
| | | | | | | | | | Ensure we keep track of the input vectors in all cases instead of just for SK_Select. Ideally we'd reuse the shuffle mask pattern matching in TargetTransformInfo::getInstructionThroughput here to easily add support for all TargetTransformInfo::ShuffleKind without mass code duplication, I've added a TODO for now but D48236 should help us here. Differential Revision: https://reviews.llvm.org/D48023 llvm-svn: 334958
* [VPlanRecipeBase] Add eraseFromParent().Florian Hahn2018-06-182-0/+9
| | | | | | | | | | Reviewers: dcaballe, hsaito, mkuper, hfinkel Reviewed By: dcaballe Differential Revision: https://reviews.llvm.org/D48081 llvm-svn: 334951
* [VPlan] Fix sanitizer problem with insertBefore.Florian Hahn2018-06-181-2/+2
| | | | llvm-svn: 334943
* [SLPVectorizer] Avoid calling const VL.size() repeatedly in for-loop. NFCI.Simon Pilgrim2018-06-181-1/+1
| | | | llvm-svn: 334934
* [VPlanRecipeBase] Add insertBefore helper.Florian Hahn2018-06-182-0/+12
| | | | | | | | | | Reviewers: dcaballe, mkuper, hfinkel, hsaito, mssimpso Reviewed By: dcaballe Differential Revision: https://reviews.llvm.org/D48080 llvm-svn: 334933
* [LV] Prevent LV to run cost model twice for VF=2Diego Caballero2018-06-151-3/+4
| | | | | | | | | | | | | | This is a minor fix for LV cost model, where the cost for VF=2 was computed twice when the vectorization of the loop was forced without specifying a VF. Reviewers: xusx595, hsaito, fhahn, mkuper Reviewed By: hsaito, xusx595 Differential Revision: https://reviews.llvm.org/D48048 llvm-svn: 334840
* [SLPVectorizer] Remove RawInstructionsData/getMainOpcode and merge into ↵Simon Pilgrim2018-06-141-49/+20
| | | | | | | | | | | | getSameOpcode This is part of the work to cleanup use of 'alternate' ops so we can use the more general SK_Select shuffle type. Only getSameOpcode calls getMainOpcode and much of the logic is repeated in both functions. This will require some reworking of D28907 but that patch has hit trouble and is unlikely to be completed anytime soon. Differential Revision: https://reviews.llvm.org/D48120 llvm-svn: 334701
* [SLPVectorizer] getSameOpcode - remove useless cast [NFC]Simon Pilgrim2018-06-131-3/+2
| | | | | | There's no need to cast the base Value to an Instruction llvm-svn: 334588
* [SLPVectorizer] getSameOpcode - remove unusued alternate code [NFC]Simon Pilgrim2018-06-131-4/+1
| | | | | | We early-out for the case where we don't use alternate opcodes, so no need to check for it later. llvm-svn: 334587
* [CostModel] Replace ShuffleKind::SK_Alternate with ShuffleKind::SK_Select ↵Simon Pilgrim2018-06-121-3/+2
| | | | | | | | | | | | | | | | | | (PR33744) As discussed on PR33744, this patch relaxes ShuffleKind::SK_Alternate which requires shuffle masks to only match an alternating pattern from its 2 sources: e.g. v4f32: <0,5,2,7> or <4,1,6,3> This seems far too restrictive as most SIMD hardware which will implement it using a general blend/bit-select instruction, so replaces it with SK_Select, permitting elements from either source as long as they are inline: e.g. v4f32: <0,5,2,7>, <4,1,6,3>, <0,1,6,7>, <4,1,2,3> etc. This initial patch just updates the name and cost model shuffle mask analysis, later patch reviews will update SLP to better utilise this - it still limits itself to SK_Alternate style patterns. Differential Revision: https://reviews.llvm.org/D47985 llvm-svn: 334513
* Use SmallPtrSet explicitly for SmallSets with pointer types (NFC).Florian Hahn2018-06-123-6/+6
| | | | | | | | | | | | | | Currently SmallSet<PointerTy> inherits from SmallPtrSet<PointerTy>. This patch replaces such types with SmallPtrSet, because IMO it is slightly clearer and allows us to get rid of unnecessarily including SmallSet.h Reviewers: dblaikie, craig.topper Reviewed By: dblaikie Differential Revision: https://reviews.llvm.org/D47836 llvm-svn: 334492
* Use SmallPtrSet instead of SmallSet in places where we iterate over the set.Craig Topper2018-06-092-3/+3
| | | | | | | | SmallSet forwards to SmallPtrSet for pointer types. SmallPtrSet supports iteration, but a normal SmallSet doesn't. So if it wasn't for the forwarding, this wouldn't work. These places were found by hiding the begin/end methods in the SmallSet forwarding llvm-svn: 334343
* [VPlan] Move recipe construction to VPRecipeBuilder.Florian Hahn2018-06-084-153/+218
| | | | | | | | | | | | | | This patch moves the recipe-creation functions out of LoopVectorizationPlanner, which should do the high-level orchestration of the transformations. Reviewers: dcaballe, rengolin, hsaito, Ayal Reviewed By: dcaballe Differential Revision: https://reviews.llvm.org/D47595 llvm-svn: 334305
* [VPlan] Move recipe based VPlan generation to separate function.Florian Hahn2018-06-082-41/+64
| | | | | | | | | | | | | | | | This first step separates VPInstruction-based and VPRecipe-based VPlan creation, which should make it easier to migrate to VPInstruction based code-gen step by step. Reviewers: Ayal, rengolin, dcaballe, hsaito, mkuper, mzolotukhin Reviewed By: dcaballe Subscribers: bollu, tschuett, rkruppe, llvm-commits Differential Revision: https://reviews.llvm.org/D47477 llvm-svn: 334284
* [LV] Fix PR36983. For a given recurrence, fix all phis in exit blockRoman Shirokiy2018-06-081-2/+1
| | | | | | | | | There could be more than one PHIs in exit block using same loop recurrence. Don't assume there is only one and fix each user. Differential Revision: https://reviews.llvm.org/D47788 llvm-svn: 334271
* Move Analysis/Utils/Local.h back to TransformsDavid Blaikie2018-06-041-1/+1
| | | | | | | | | | Review feedback from r328165. Split out just the one function from the file that's used by Analysis. (As chandlerc pointed out, the original change only moved the header and not the implementation anyway - which was fine for the one function that was used (since it's a template/inlined in the header) but not in general) llvm-svn: 333954
* [VPlan] Replace LLVM_ATTRIBUTE_USED with ifndef NDEBUGDiego Caballero2018-05-291-2/+3
| | | | | | | | | | | | | Minor replacement. LLVM_ATTRIBUTE_USED was introduced to silence a warning but using #ifndef NDEBUG makes more sense in this case. Reviewers: dblaikie, fhahn, hsaito Reviewed By: dblaikie Differential Revision: https://reviews.llvm.org/D47498 llvm-svn: 333476
* [LLVM-C] [OCaml] Remove LLVMAddBBVectorizePassFangrui Song2018-05-281-4/+0
| | | | | | | | | | | | Summary: It was fully replaced back in 2014, and the implementation was removed 11 months ago by r306797. Reviewers: hfinkel, chandlerc, whitequark, deadalnix Subscribers: llvm-commits Differential Revision: https://reviews.llvm.org/D47436 llvm-svn: 333378
* [NFC][VPlan] Wrap PlainCFGBuilder with an anonymous namespace.Andrei Elovikov2018-05-241-0/+2
| | | | | | | | | | | | | | | | Summary: It's internal to the VPlanHCFGBuilder and should not be visible outside of its translation unit. Reviewers: dcaballe, fhahn Reviewed By: fhahn Subscribers: rengolin, bollu, tschuett, llvm-commits, rkruppe Differential Revision: https://reviews.llvm.org/D47312 llvm-svn: 333187
* Remove DEBUG macro.Nicola Zaghen2018-05-232-3/+3
| | | | | | | | | | Now that the LLVM_DEBUG() macro landed on the various sub-projects the DEBUG macro can be removed. Also change the new uses of DEBUG to LLVM_DEBUG. Differential Revision: https://reviews.llvm.org/D46952 llvm-svn: 333091
* Fix warning from r332654 with LLVM_ATTRIBUTE_USEDDiego Caballero2018-05-211-2/+2
| | | | | | | | | r332654 tried to fix an unused function warning with a void cast. This approach worked for clang and gcc but not for MSVC. This commit replaces the void cast with the LLVM_ATTRIBUTE_USED approach. llvm-svn: 332910
* [VPlan] Reland r332654 and silence unused func warningDiego Caballero2018-05-219-42/+839
| | | | | | | | | | r332654 was reverted due to an unused function warning in release build. This commit includes the same code with the warning silenced. Differential Revision: https://reviews.llvm.org/D44338 llvm-svn: 332860
* Reverted r332654 as it has broken some buildbots and left unfixed for a long ↵Galina Kistanova2018-05-189-838/+42
| | | | | | | | | | | time. The introduced problem is: llvm.src/lib/Transforms/Vectorize/VPlanVerifier.cpp:29:13: error: unused function 'hasDuplicates' [-Werror,-Wunused-function] static bool hasDuplicates(const SmallVectorImpl<VPBlockBase *> &VPBlockVec) { ^ llvm-svn: 332747
* [LV][VPlan] Build plain CFG with simple VPInstructions for outer loops.Diego Caballero2018-05-179-42/+838
| | | | | | | | | | | | | | | | | | | Patch #3 from VPlan Outer Loop Vectorization Patch Series #1 (RFC: http://lists.llvm.org/pipermail/llvm-dev/2017-December/119523.html). Expected to be NFC for the current inner loop vectorization path. It introduces the basic algorithm to build the VPlan plain CFG (single-level CFG, no hierarchical CFG (H-CFG), yet) in the VPlan-native vectorization path using VPInstructions. It includes: - VPlanHCFGBuilder: Main class to build the VPlan H-CFG (plain CFG without nested regions, for now). - VPlanVerifier: Main class with utilities to check the consistency of a H-CFG. - VPlanBlockUtils: Main class with utilities to manipulate VPBlockBases in VPlan. Reviewers: rengolin, fhahn, mkuper, mssimpso, a.elovikov, hfinkel, aprantl. Differential Revision: https://reviews.llvm.org/D44338 llvm-svn: 332654
* Rename DEBUG macro to LLVM_DEBUG.Nicola Zaghen2018-05-145-309/+351
| | | | | | | | | | | | | | | | The DEBUG() macro is very generic so it might clash with other projects. The renaming was done as follows: - git grep -l 'DEBUG' | xargs sed -i 's/\bDEBUG\s\?(/LLVM_DEBUG(/g' - git diff -U0 master | ../clang/tools/clang-format/clang-format-diff.py -i -p1 -style LLVM - Manual change to APInt - Manually chage DOCS as regex doesn't match it. In the transition period the DEBUG() macro is still present and aliased to the LLVM_DEBUG() one. Differential Revision: https://reviews.llvm.org/D43624 llvm-svn: 332240
* [STLExtras] Add distance() for ranges, pred_size(), and succ_size()Vedant Kumar2018-05-101-1/+1
| | | | | | | | | | | This commit adds a wrapper for std::distance() which works with ranges. As it would be a common case to write `distance(predecessors(BB))`, this also introduces `pred_size()` and `succ_size()` helpers to make that easier to write. Differential Revision: https://reviews.llvm.org/D46668 llvm-svn: 332057
* [LV] Change MaxVectorSize bound to 256 in assertion, NFC otherwiseKrzysztof Parzyszek2018-05-091-2/+2
| | | | | | | It's possible to have a vector of 256 bytes in HVX code on Hexagon (vector pair in 128-byte mode). llvm-svn: 331885
* [LV] Fix for PR37248, Broadcast codegen incorrectly assumed vector loop body ↵Hideki Saito2018-05-081-5/+7
| | | | | | | | | | | | | | | | | | | is single basic block Summary: Broadcast code generation emitted instructions in pre-header, while the instruction they are dependent on in the vector loop body. This resulted in an IL verification error ---- value used before defined. Reviewers: rengolin, fhahn, hfinkel Reviewed By: rengolin, fhahn Subscribers: dcaballe, Ka-Ka, llvm-commits Differential Revision: https://reviews.llvm.org/D46302 llvm-svn: 331799
* Fix a bunch of places where operator-> was used directly on the return from ↵Craig Topper2018-05-051-5/+4
| | | | | | | | | | dyn_cast. Inspired by r331508, I did a grep and found these. Mostly just change from dyn_cast to cast. Some cases also showed a dyn_cast result being converted to bool, so those I changed to isa. llvm-svn: 331577
* Remove \brief commands from doxygen comments.Adrian Prantl2018-05-015-50/+50
| | | | | | | | | | | | | | | | We've been running doxygen with the autobrief option for a couple of years now. This makes the \brief markers into our comments redundant. Since they are a visual distraction and we don't want to encourage more \brief markers in new code either, this patch removes them all. Patch produced by for i in $(git grep -l '\\brief'); do perl -pi -e 's/\\brief //g' $i & done Differential Revision: https://reviews.llvm.org/D46290 llvm-svn: 331272
* [LV] Preserve inbounds on created GEPsDaniel Neilson2018-05-011-3/+22
| | | | | | | | | | | | | | | | | | | Summary: This is a fix for PR23997. The loop vectorizer is not preserving the inbounds property of GEPs that it creates. This is inhibiting some optimizations. This patch preserves the inbounds property in the case where a load/store is being fed by an inbounds GEP. Reviewers: mkuper, javed.absar, hsaito Reviewed By: hsaito Subscribers: dcaballe, hsaito, llvm-commits Differential Revision: https://reviews.llvm.org/D46191 llvm-svn: 331269
* [SLPVectorizer] Debug info shouldn't impact spill cost computation.Davide Italiano2018-04-301-1/+4
| | | | | | | | | | <rdar://problem/39794738> (Also, PR32761). Differential Revision: https://reviews.llvm.org/D46199 llvm-svn: 331199
* [LV] Use BB::instructionsWithoutDebug to skip DbgInfo (NFC).Florian Hahn2018-04-301-8/+3
| | | | | | | | | | | | | | This patch updates some code responsible the skip debug info to use BasicBlock::instructionsWithoutDebug. I think this makes things slightly simpler and more direct. Reviewers: mkuper, rengolin, dcaballe, aprantl, vsk Reviewed By: rengolin Differential Revision: https://reviews.llvm.org/D46254 llvm-svn: 331174
OpenPOWER on IntegriCloud