summaryrefslogtreecommitdiffstats
path: root/llvm/lib/Transforms/Vectorize/LoopVectorizationPlanner.h
Commit message (Collapse)AuthorAgeFilesLines
* [LV] Do not try to sink dead instructions.Florian Hahn2020-01-291-3/+4
| | | | | | | | | | | | | | | | | Dead instructions do not need to be sunk. Currently we try and record the recipies for them, but there are no recipes emitted for them and there's nothing to sink. They can be removed from SinkAfter while marking them for recording. Fixes PR44634. Reviewers: rengolin, hsaito, fhahn, Ayal, gilr Reviewed By: gilr Differential Revision: https://reviews.llvm.org/D73423 (cherry picked from commit a911fef3dd79e0a04b241be7b476dde7e99744c4)
* [LV] VPValues for memory operation pointers (NFCI)Gil Rapaport2020-01-101-0/+2
| | | | | | | | | | | | Memory instruction widening recipes use the pointer operand of their load/store ingredient for generating the needed GEPs, making it difficult to feed these recipes with pointers based on other ingredients or none at all. This patch modifies these recipes to use a VPValue for the pointer instead, in order to reduce ingredient def-use usage by ILV as a step towards full VPlan-based def-use relations. The recipes are constructed with VPValues bound to these ingredients, maintaining current behavior. Differential revision: https://reviews.llvm.org/D70865
* [LV] Apply sink-after & interleave-groups as VPlan transformations (NFCI)Gil Rapaport2019-11-091-2/+7
| | | | | | | | | This recommits 11ed1c0239fd51fd2f064311dc7725277ed0a994 (reverted in 9f08ce0d2197d4f163dfa4633eae2347ce8fc881 for failing an assert) with a fix: tryToWidenMemory() now first checks if the widening decision is to interleave, thus maintaining previous behavior where tryToInterleaveMemory() was called first, giving priority to interleave decisions over widening/scalarization. This commit adds the test case that exposed this bug as a LIT.
* Revert "[LV] Apply sink-after & interleave-groups as VPlan transformations ↵Gil Rapaport2019-11-081-7/+2
| | | | | | (NFCI)" This reverts commit 11ed1c0239fd51fd2f064311dc7725277ed0a994 - causes an assert failure.
* [LV] Apply sink-after & interleave-groups as VPlan transformations (NFCI)Gil Rapaport2019-11-081-2/+7
| | | | | | | | This recommits 100e797adb433724a17c9b42b6533cd634cb796b (reverted in 009e032634b3bd7fc32071ac2344b12142286477 for failing an assert). While the root cause was independently reverted in eaff3004019f97c64c88ab76da6b25106b659b30, this commit includes a LIT to make sure IVDescriptor's SinkAfter logic does not try to sink branch instructions.
* Temporarily Revert "[LV] Apply sink-after & interleave-groups as VPlan ↵Eric Christopher2019-11-061-7/+2
| | | | | | | | transformations (NFC)" as it's causing assert failures. This reverts commit 100e797adb433724a17c9b42b6533cd634cb796b.
* [LV] Apply sink-after & interleave-groups as VPlan transformations (NFC)Gil Rapaport2019-11-051-2/+7
| | | | | | | This recommits 2be17087f8c38934b7fc9208ae6cf4e9b4d44f4b (reverted in d3ec06d219788801380af1948c7f7ef9d3c6100b for heap-use-after-free) with a fix in IAI's reset() which was not clearing the set of interleave groups after deleting them.
* Revert "[LV] Apply sink-after & interleave-groups as VPlan transformations ↵Benjamin Kramer2019-11-041-7/+2
| | | | | | (NFC)" This reverts commit 2be17087f8c38934b7fc9208ae6cf4e9b4d44f4b. Fails ASAN.
* [LV] Apply sink-after & interleave-groups as VPlan transformations (NFC)Gil Rapaport2019-11-041-2/+7
| | | | | | | | | | The sink-after and interleave-group vectorization decisions were so far applied to VPlan during initial VPlan construction, which complicates VPlan construction – also because of their inter-dependence. This patch refactors buildVPlanWithRecipes() to construct a simpler initial VPlan and later apply both these vectorization decisions, in order, as VPlan-to-VPlan transformations. Differential Revision: https://reviews.llvm.org/D68577
* [LV] Scalar Epilogue Lowering. NFC.Sjoerd Meijer2019-07-251-2/+2
| | | | | | | | | | | | | | | | This refactors boolean 'OptForSize' that was passed around in a lot of places. It controlled folding of the tail loop, the scalar epilogue, into the main loop but code-size reasons may not be the only reason to do this. Thus, this is a first step to generalise the concept of tail-loop folding, and hence OptForSize has been renamed and is using an enum ScalarEpilogueStatus that holds the status how the epilogue should be lowered. This will be followed up by D65197, that picks up the predicate loop hint and performs the tail-loop folding. Differential Revision: https://reviews.llvm.org/D64916 llvm-svn: 366993
* [LV] Remove the redundant using LoopVectorizationPlanner:VPlanPtrFlorian Hahn2019-05-301-3/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | VPlan.h already contains the declaration of VPlanPtr type alias: using VPlanPtr = std::unique_ptr<VPlan>; The LoopVectorizationPlanner class also contains the same declaration of VPlanPtr and therefore LoopVectorize requires a long wording when its methods return VPlanPtr: LoopVectorizationPlanner::VPlanPtr LoopVectorizationPlanner::buildVPlanWithVPRecipes(...) but LoopVectorize.cpp includes VPlan.h (via LoopVectorizationPlanner.h) and can use VPlanPtr from that header. Patch by Pavel Samolysov. Reviewers: hsaito, rengolin, fhahn Reviewed By: fhahn Differential Revision: https://reviews.llvm.org/D62576 llvm-svn: 362126
* [VPlan] Determine Vector Width programmatically.Florian Hahn2019-03-281-0/+4
| | | | | | | | | | | | | | With this change, the VPlan native path is triggered with the directive: #pragma clang loop vectorize(enable) There is no need to specify the vectorize_width(N) clause. Patch by Francesco Petrogalli <francesco.petrogalli@arm.com> Differential Revision: https://reviews.llvm.org/D57598 llvm-svn: 357156
* [LV] Prevent interleaving if computeMaxVF returned None.Florian Hahn2019-02-071-2/+6
| | | | | | | | | | | | | | | As discussed in D57382, interleaving should be avoided if computeMaxVF returns None, same as we currently do for vectorization. Fixes https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=6477 Reviewers: Ayal, dcaballe, hsaito, mkuper, rengolin Reviewed By: Ayal Differential Revision: https://reviews.llvm.org/D57837 llvm-svn: 353461
* Update the file headers across all of the LLVM projects in the monorepoChandler Carruth2019-01-191-4/+3
| | | | | | | | | | | | | | | | | to reflect the new license. We understand that people may be surprised that we're moving the header entirely to discuss the new license. We checked this carefully with the Foundation's lawyer and we believe this is the correct approach. Essentially, all code in the project is now made available by the LLVM project under our new license, so you will see that the license headers include that license only. Some of our contributors have contributed code under our old license, and accordingly, we have retained a copy of our old license notice in the top-level files in each project and repository. llvm-svn: 351636
* [VPlan] Move recipe construction to VPRecipeBuilder.Florian Hahn2018-06-081-89/+7
| | | | | | | | | | | | | | This patch moves the recipe-creation functions out of LoopVectorizationPlanner, which should do the high-level orchestration of the transformations. Reviewers: dcaballe, rengolin, hsaito, Ayal Reviewed By: dcaballe Differential Revision: https://reviews.llvm.org/D47595 llvm-svn: 334305
* [VPlan] Move recipe based VPlan generation to separate function.Florian Hahn2018-06-081-2/+12
| | | | | | | | | | | | | | | | This first step separates VPInstruction-based and VPRecipe-based VPlan creation, which should make it easier to migrate to VPInstruction based code-gen step by step. Reviewers: Ayal, rengolin, dcaballe, hsaito, mkuper, mzolotukhin Reviewed By: dcaballe Subscribers: bollu, tschuett, rkruppe, llvm-commits Differential Revision: https://reviews.llvm.org/D47477 llvm-svn: 334284
* [VPlan] Reland r332654 and silence unused func warningDiego Caballero2018-05-211-5/+96
| | | | | | | | | | r332654 was reverted due to an unused function warning in release build. This commit includes the same code with the warning silenced. Differential Revision: https://reviews.llvm.org/D44338 llvm-svn: 332860
* Reverted r332654 as it has broken some buildbots and left unfixed for a long ↵Galina Kistanova2018-05-181-96/+5
| | | | | | | | | | | time. The introduced problem is: llvm.src/lib/Transforms/Vectorize/VPlanVerifier.cpp:29:13: error: unused function 'hasDuplicates' [-Werror,-Wunused-function] static bool hasDuplicates(const SmallVectorImpl<VPBlockBase *> &VPBlockVec) { ^ llvm-svn: 332747
* [LV][VPlan] Build plain CFG with simple VPInstructions for outer loops.Diego Caballero2018-05-171-5/+96
| | | | | | | | | | | | | | | | | | | Patch #3 from VPlan Outer Loop Vectorization Patch Series #1 (RFC: http://lists.llvm.org/pipermail/llvm-dev/2017-December/119523.html). Expected to be NFC for the current inner loop vectorization path. It introduces the basic algorithm to build the VPlan plain CFG (single-level CFG, no hierarchical CFG (H-CFG), yet) in the VPlan-native vectorization path using VPInstructions. It includes: - VPlanHCFGBuilder: Main class to build the VPlan H-CFG (plain CFG without nested regions, for now). - VPlanVerifier: Main class with utilities to check the consistency of a H-CFG. - VPlanBlockUtils: Main class with utilities to manipulate VPBlockBases in VPlan. Reviewers: rengolin, fhahn, mkuper, mssimpso, a.elovikov, hfinkel, aprantl. Differential Revision: https://reviews.llvm.org/D44338 llvm-svn: 332654
* Remove \brief commands from doxygen comments.Adrian Prantl2018-05-011-1/+1
| | | | | | | | | | | | | | | | We've been running doxygen with the autobrief option for a couple of years now. This makes the \brief markers into our comments redundant. Since they are a visual distraction and we don't want to encourage more \brief markers in new code either, this patch removes them all. Patch produced by for i in $(git grep -l '\\brief'); do perl -pi -e 's/\\brief //g' $i & done Differential Revision: https://reviews.llvm.org/D46290 llvm-svn: 331272
* [LV][VPlan] Detect outer loops for explicit vectorization.Diego Caballero2018-04-241-0/+4
| | | | | | | | | | | | | | | | | Patch #2 from VPlan Outer Loop Vectorization Patch Series #1 (RFC: http://lists.llvm.org/pipermail/llvm-dev/2017-December/119523.html). This patch introduces the basic infrastructure to detect, legality check and process outer loops annotated with hints for explicit vectorization. All these changes are protected under the feature flag -enable-vplan-native-path. This should make this patch NFC for the existing inner loop vectorizer. Reviewers: hfinkel, mkuper, rengolin, fhahn, aemerson, mssimpso. Differential Revision: https://reviews.llvm.org/D42447 llvm-svn: 330739
* Add missing includesDavid Blaikie2018-02-021-0/+3
| | | | llvm-svn: 324040
* [LV][VPlan] NFC patch to move LoopVectorizationPlanner class out of ↵Hal Finkel2018-01-071-0/+256
LoopVectorize.cpp Another small step forward to move VPlan stuff outside of LoopVectorize.cpp. VPlanBuilder.h is renamed to LoopVectorizationPlanner.h LoopVectorizationPlanner class is moved from LoopVectorize.cpp to LoopVectorizationPlanner.h LoopVectorizationCostModel::VectorizationFactor class is moved to LoopVectorizationPlanner.h (used by the planner class) --- this needs further streamlining work in later patches and thus all I did was take it out of the CostModel class and moved to the header file. The callback function had to stay inside LoopVectorize.cpp since it calls an InnerLoopVectorizer member function declared in it. Next Steps: Make InnerLoopVectorizer, LoopVectorizationCostModel, and other classes more modular and more aligned with VPlan direction, in small increments. Previous step was: r320900 (https://reviews.llvm.org/D41045) Patch by Hideki Saito, thanks! Differential Revision: https://reviews.llvm.org/D41420 llvm-svn: 321962
OpenPOWER on IntegriCloud