summaryrefslogtreecommitdiffstats
path: root/llvm/lib/Transforms/Vectorize
Commit message (Collapse)AuthorAgeFilesLines
* [SLP] Revert r312791 and other necessary commits, except for TTI andChandler Carruth2017-09-151-245/+49
| | | | | | | | | | | | | | | | | | | | | | | | | | CostModel. The original patch added support for horizontal min/max reductions to the SLP vectorizer. This patch causes LLVM to miscompile fairly simple signed min reductions. I have attached a test progrom to http://llvm.org/PR34635 that shows the behavior change after this patch. We found this in a test for the open source Eigen library, but also in other code. Unfortunately, the revert is moderately challenging. It required reverting: r313042: [SLP] Test with multiple uses of conditional op and wrong parent. r312853: [SLP] Fix buildbots, NFC. r312793: [SLP] Fix the warning about paths not returning the value, NFC. r312791: [SLP] Support for horizontal min/max reduction. And even then, I had to completely skip reverting the changes to TTI and CostModel because r312832 rewrote so much of this code. Plus, the cost modeling changes aren implicated in the miscompile, so they should be fine and will just not be used until this gets re-introduced. llvm-svn: 313409
* This patch fixes https://bugs.llvm.org/show_bug.cgi?id=32352 Vivek Pandya2017-09-151-9/+12
| | | | | | | | | | | It enables OptimizationRemarkEmitter::allowExtraAnalysis and MachineOptimizationRemarkEmitter::allowExtraAnalysis to return true not only for -fsave-optimization-record but when specific remarks are requested with command line options. The diagnostic handler used to be callback now this patch adds a class DiagnosticHandler. It has virtual method to provide custom diagnostic handler and methods to control which particular remarks are enabled. However LLVM-C API users can still provide callback function for diagnostic handler. llvm-svn: 313390
* This reverts r313381Vivek Pandya2017-09-151-12/+9
| | | | llvm-svn: 313387
* This patch fixes https://bugs.llvm.org/show_bug.cgi?id=32352 Vivek Pandya2017-09-151-9/+12
| | | | | | | | | | | It enables OptimizationRemarkEmitter::allowExtraAnalysis and MachineOptimizationRemarkEmitter::allowExtraAnalysis to return true not only for -fsave-optimization-record but when specific remarks are requested with command line options. The diagnostic handler used to be callback now this patch adds a class DiagnosticHandler. It has virtual method to provide custom diagnostic handler and methods to control which particular remarks are enabled. However LLVM-C API users can still provide callback function for diagnostic handler. llvm-svn: 313382
* Revert "[SLPVectorizer] Failure to beneficially vectorize 'copyable' ↵Ilya Biryukov2017-09-151-317/+142
| | | | | | | | | elements in integer binary ops." This reverts commit r313348. Reason: it caused buildbot failures. llvm-svn: 313352
* [SLPVectorizer] Failure to beneficially vectorize 'copyable' elements in ↵Dinar Temirbulatov2017-09-151-142/+317
| | | | | | | | | | | | | | | | | | | | | | integer binary ops. Patch tries to improve vectorization of the following code: void add1(int * __restrict dst, const int * __restrict src) { *dst++ = *src++; *dst++ = *src++ + 1; *dst++ = *src++ + 2; *dst++ = *src++ + 3; } Allows to vectorize even if the very first operation is not a binary add, but just a load. Reviewers: spatel, mzolotukhin, mkuper, hfinkel, RKSimon, filcab, ABataev, davide Subscribers: llvm-commits, RKSimon Differential Revision: https://reviews.llvm.org/D28907 llvm-svn: 313348
* [SLPVectorizer] Remove duplicated functionality code in initScheduleData ↵Dinar Temirbulatov2017-09-151-6/+0
| | | | | | function, NFCI. llvm-svn: 313341
* [LV] Fix maximum legal VF calculationAlon Kom2017-09-141-28/+18
| | | | | | | | | | | | | | | | This patch fixes pr34283, which exposed that the computation of maximum legal width for vectorization was wrong, because it relied on MaxInterleaveFactor to obtain the maximum stride used in the loop, however not all strided accesses in the loop have an interleave-group associated with them. Instead of recording the maximum stride in the loop, which can be over conservative (e.g. if the access with the maximum stride is not involved in the dependence limitation), this patch tracks the actual maximum legal width imposed by accesses that are involved in dependencies. Differential Revision: https://reviews.llvm.org/D37507 llvm-svn: 313237
* [SLPVectorizer] Prefer auto over explicit type for VL0, NFCI.Dinar Temirbulatov2017-09-141-1/+1
| | | | llvm-svn: 313228
* [LV] Avoid computing the register usage for default VF. NFCAnna Thomas2017-09-131-2/+4
| | | | | | | | | | | These are changes to reduce redundant computations when calculating a feasible vectorization factor: 1. early return when target has no vector registers 2. don't compute register usage for the default VF. Suggested during review for D37702. llvm-svn: 313176
* [LV] Fix PR34523 - avoid generating redundant selectsAyal Zaks2017-09-131-3/+3
| | | | | | | | | | | | | | | | | When converting a PHI into a series of 'select' instructions to combine the incoming values together according their edge masks, initialize the first value to the incoming value In0 of the first predecessor, instead of generating a redundant assignment 'select(Cond[0], In0, In0)'. The latter fails when the Cond[0] mask is null, representing a full mask, which can happen only when there's a single incoming value. No functional changes intended nor expected other than surviving null Cond[0]'s. This fix follows D35725, which introduced using null to represent full masks. Differential Revision: https://reviews.llvm.org/D37619 llvm-svn: 313119
* [LV] Clamp the VF to the trip countAnna Thomas2017-09-121-7/+12
| | | | | | | | | | | | | | | | | | | | | | | | | Summary: When the MaxVectorSize > ConstantTripCount, we should just clamp the vectorization factor to be the ConstantTripCount. This vectorizes loops where the TinyTripCountThreshold >= TripCount < MaxVF. Earlier we were finding the maximum vector width, which could be greater than the trip count itself. The Loop vectorizer does all the work for generating a vectorizable loop, but in the end we would always choose the scalar loop (since the VF > trip count). This allows us to choose the VF keeping in mind the trip count if available. This is a fix on top of rL312472. Reviewers: Ayal, zvi, hfinkel, dneilson Reviewed by: Ayal Subscribers: llvm-commits Differential Revision: https://reviews.llvm.org/D37702 llvm-svn: 313046
* [SLP] Fix for PHINode during horizontal reduction scanning, NFC.Alexey Bataev2017-09-121-1/+1
| | | | | | Reduces number of loops during instructions analysis. llvm-svn: 313035
* [SLP] Fix buildbots, NFC.Alexey Bataev2017-09-091-2/+2
| | | | llvm-svn: 312853
* [SLPVectorizer] Add struct InstructionsState that holds information about ↵Dinar Temirbulatov2017-09-081-88/+120
| | | | | | | | | | | | analysis of vector to be vectorized. Reviewers: spatel, mzolotukhin, mkuper, hfinkel, RKSimon, filcab, ABataev, davide Subscribers: llvm-commits, rengolin Differential Revision: https://reviews.llvm.org/D37212 llvm-svn: 312802
* [SLP] Fix the warning about paths not returning the value, NFC.Alexey Bataev2017-09-081-2/+4
| | | | llvm-svn: 312793
* [SLP] Support for horizontal min/max reduction.Alexey Bataev2017-09-081-49/+243
| | | | | | | | | | | | | SLP vectorizer supports horizontal reductions for Add/FAdd binary operations. Patch adds support for horizontal min/max reductions. Function getReductionCost() is split to getArithmeticReductionCost() for binary operation reductions and getMinMaxReductionCost() for min/max reductions. Patch fixes PR26956. Differential revision: https://reviews.llvm.org/D27846 llvm-svn: 312791
* LoopVectorize: MaxVF should not be larger than the loop trip countZvi Rackover2017-09-041-4/+8
| | | | | | | | | | | | | | | | | | | | | | | Summary: Improve how MaxVF is computed while taking into account that MaxVF should not be larger than the loop's trip count. Other than saving on compile-time by pruning the possible MaxVF candidates, this patch fixes pr34438 which exposed the following flow: 1. Short trip count identified -> Don't bail out, set OptForSize:=True to avoid tail-loop and runtime checks. 2. Compute MaxVF returned 16 on a target supporting AVX512. 3. OptForSize -> choose VF:=MaxVF. 4. Bail out because TripCount = 8, VF = 16, TripCount % VF !=0 means we need a tail loop. With this patch step 2. will choose MaxVF=8 based on TripCount. Reviewers: Ayal, dorit, mkuper, hfinkel Reviewed By: hfinkel Subscribers: hfinkel, llvm-commits Differential Revision: https://reviews.llvm.org/D37425 llvm-svn: 312472
* [LoopVectorize] Turn static DenseSet into switch.Benjamin Kramer2017-09-021-16/+47
| | | | | | LLVM transforms this into a bit test which is a lot faster and smaller. llvm-svn: 312417
* [Analysis, Transforms] Fix some Clang-tidy modernize and Include What You ↵Eugene Zelenko2017-09-011-90/+143
| | | | | | Use warnings; other minor fixes (NFC). llvm-svn: 312383
* [LoopVectorizer] Use two step casting for float to pointer types.Manoj Gupta2017-09-011-3/+40
| | | | | | | | | | | | | | | | | | | | Summary: LoopVectorizer is creating casts between vec<ptr> and vec<float> types on ARM when compiling OpenCV. Since, tIs is illegal to directly cast a floating point type to a pointer type even if the types have same size causing a crash. Fix the crash using a two-step casting by bitcasting to integer and integer to pointer/float. Fixes PR33804. Reviewers: mkuper, Ayal, dlj, rengolin, srhines Reviewed By: rengolin Subscribers: aemerson, kristof.beyls, mkazantsev, Meinersbur, rengolin, mzolotukhin, llvm-commits Differential Revision: https://reviews.llvm.org/D35498 llvm-svn: 312331
* [SLPVectorizer] Move out Entry->NeedToGather check and assert of inner loop ↵Dinar Temirbulatov2017-08-311-5/+6
| | | | | | as invariant, NFCI. llvm-svn: 312242
* [Instruction] add moveAfter() convenience function; NFCISanjay Patel2017-08-291-2/+1
| | | | | | | | | As suggested in D37121, here's a wrapper for removeFromParent() + insertAfter(), but implemented using moveBefore() for symmetry/efficiency. Differential Revision: https://reviews.llvm.org/D37239 llvm-svn: 312001
* [LV] Fix PR34248 - recommit D32871 after revert r311304Ayal Zaks2017-08-274-523/+2247
| | | | | | | | | | | Original commit r311077 of D32871 was reverted in r311304 due to failures reported in PR34248. This recommit fixes PR34248 by restricting the packing of predicated scalars into vectors only when vectorizing, avoiding doing so when unrolling w/o vectorizing. Added a test derived from the reproducer of PR34248. llvm-svn: 311849
* Revert r311077: [LV] Using VPlan ...Chandler Carruth2017-08-204-2251/+523
| | | | | | | This causes LLVM to assert fail on PPC64 and crash / infloop in other cases. Filed http://llvm.org/PR34248 with reproducer attached. llvm-svn: 311304
* [Loop Vectorize] Added a separate metadataAditya Kumar2017-08-201-9/+18
| | | | | | | | | | | Added a separate metadata to indicate when the loop has already been vectorized instead of setting width and count to 1. Patch written by Divya Shanmughan and Aditya Kumar Differential Revision: https://reviews.llvm.org/D36220 llvm-svn: 311281
* [SLP] Fix an unused variable warning in non-asserts builds.Chandler Carruth2017-08-191-0/+1
| | | | llvm-svn: 311227
* [SLPVectorizer] Tighten up VLeft, VRight declaration, remove unnecessary ↵Dinar Temirbulatov2017-08-191-4/+2
| | | | | | testcase test/Transforms/SLPVectorizer/X86/reorder.ll, NFCI. llvm-svn: 311223
* [SLPVectorizer] Add opcode parameter to reorderAltShuffleOperands, ↵Dinar Temirbulatov2017-08-191-30/+43
| | | | | | | | | | | | reorderInputsAccordingToOpcode functions. Reviewers: mkuper, RKSimon, ABataev, mzolotukhin, spatel, filcab Subscribers: llvm-commits, rengolin Differential Revision: https://reviews.llvm.org/D36766 llvm-svn: 311221
* [LV] Using VPlan to model the vectorized code and drive its transformationAyal Zaks2017-08-174-523/+2251
| | | | | | | | | | | | | | | | | | | | | | | | | | | | VPlan is an ongoing effort to refactor and extend the Loop Vectorizer. This patch introduces the VPlan model into LV and uses it to represent the vectorized code and drive the generation of vectorized IR. In this patch VPlan models the vectorized loop body: the vectorized control-flow is represented using VPlan's Hierarchical CFG, with predication refactored from being a post-vectorization-step into a vectorization planning step modeling if-then VPRegionBlocks, and generating code inline with non-predicated code. The vectorized code within each VPBasicBlock is represented as a sequence of Recipes, each responsible for modelling and generating a sequence of IR instructions. To keep the size of this commit manageable the Recipes in this patch are coarse-grained and capture large chunks of LV's code-generation logic. The constructed VPlans are dumped in dot format under -debug. This commit retains current vectorizer output, except for minor instruction reorderings; see associated modifications to lit tests. For further details on the VPlan model see docs/Proposals/VectorizationPlan.rst and its references. Authors: Gil Rapaport and Ayal Zaks Differential Revision: https://reviews.llvm.org/D32871 llvm-svn: 311077
* Fix -Wunused-lambda-capture for Release build.Rui Ueyama2017-08-151-2/+2
| | | | | | | `I` and `this` are used only in assert or DEBUG, so they are unused in Release build. llvm-svn: 310934
* [LV] Minor savings to Sink casts to unravel first order recurrenceAyal Zaks2017-08-151-1/+1
| | | | | | | | | Two minor savings: avoid copying the SinkAfter map and avoid moving a cast if it is not needed. Differential Revision: https://reviews.llvm.org/D36408 llvm-svn: 310910
* [SLPVectorizer] Replace VL[0] to VL0 with assert, add propagateIRFlags extra ↵Dinar Temirbulatov2017-08-151-7/+7
| | | | | | | | parameter VL0, replace E->Scalars[0] to VL0, NFCI. llvm-svn: 310904
* [SLPVectorizer] Schedule bundle with different opcodes.Dinar Temirbulatov2017-08-141-52/+140
| | | | | | | | | | | | This change let us schedule a bundle with different opcodes in it, for example : [ load, add, add, add ] Reviewers: mkuper, RKSimon, ABataev, mzolotukhin, spatel, filcab Subscribers: llvm-commits, rengolin Differential Revision: https://reviews.llvm.org/D36518 llvm-svn: 310847
* [LoopVectorize] Fix assertion failure in Fcmp vectorizationAnna Thomas2017-08-081-1/+3
| | | | | | | | | | | | | | | | | | | | | Summary: When vectorizing fcmps we can trip on incorrect cast assertion when setting the FastMathFlags after generating the vectorized FCmp. This can happen if the FCmp can be folded to true or false directly. The fix here is to set the FastMathFlag using the FastMathFlagBuilder *before* creating the FCmp Instruction. This is what's done by other optimizations such as InstCombine. Added a test case which trips on cast assertion without this patch. Reviewers: Ayal, mssimpso, mkuper, gilr Reviewed by: Ayal, mssimpso Subscribers: llvm-commits, mzolotukhin Differential Revision: https://reviews.llvm.org/D36244 llvm-svn: 310389
* [SLP] General improvements of SLP vectorization process.Alexey Bataev2017-08-071-106/+107
| | | | | | | | | | | | | | | | | | Patch tries to improve two-pass vectorization analysis, existing in SLP vectorizer. What it does: 1. Defines key nodes, that are the vectorization roots. Previously vectorization started if StoreInst or ReturnInst is found. For now, the vectorization started for all Instructions with no users and void types (Terminators, StoreInst) + CallInsts. 2. CmpInsts, InsertElementInsts and InsertValueInsts are stored in the array. This array is processed only after the vectorization of the first-after-these instructions key node is finished. Vectorization goes in reverse order to try to vectorize as much code as possible. Reviewers: mzolotukhin, Ayal, mkuper, gilr, hfinkel, RKSimon Subscribers: ashahid, anemet, RKSimon, mssimpso, llvm-commits Differential Revision: https://reviews.llvm.org/D29826 llvm-svn: 310260
* Revert "[SLP] General improvements of SLP vectorization process."Alexey Bataev2017-08-071-109/+106
| | | | | | This reverts commit r310255. llvm-svn: 310257
* [SLP] General improvements of SLP vectorization process.Alexey Bataev2017-08-071-106/+109
| | | | | | | | | | | | | | | Summary: Patch tries to improve two-pass vectorization analysis, existing in SLP vectorizer. What it does: 1. Defines key nodes, that are the vectorization roots. Previously vectorization started if StoreInst or ReturnInst is found. For now, the vectorization started for all Instructions with no users and void types (Terminators, StoreInst) + CallInsts. 2. CmpInsts, InsertElementInsts and InsertValueInsts are stored in the array. This array is processed only after the vectorization of the first-after-these instructions key node is finished. Vectorization goes in reverse order to try to vectorize as much code as possible. Reviewers: mzolotukhin, Ayal, mkuper, gilr, hfinkel, RKSimon Subscribers: ashahid, anemet, RKSimon, mssimpso, llvm-commits Differential Revision: https://reviews.llvm.org/D29826 llvm-svn: 310255
* [SLPVectorizer] Add extra parameter to setInsertPointAfterBundle to handle ↵Dinar Temirbulatov2017-08-051-23/+54
| | | | | | | | different opcodes, NFCI. Differential Revision: https://reviews.llvm.org/D35769 llvm-svn: 310183
* LV: Don't insert runtime ptr checks on divergent targetsMatt Arsenault2017-08-021-0/+12
| | | | llvm-svn: 309890
* [SLPVectorizer] Generalize interface of functions, NFC.Alexey Bataev2017-08-021-11/+15
| | | | llvm-svn: 309816
* [SLP] Fix for PR31880: shuffle and vectorize repeated scalar ops on ↵Alexey Bataev2017-08-021-0/+133
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | extracted elements Summary: Currently most of the time vectors of extractelement instructions are treated as scalars that must be gathered into vectors. But in some cases, like when we have extractelement instructions from single vector with different constant indeces or from 2 vectors of the same size, we can treat this operations as shuffle of a single vector or blending of 2 vectors. ``` define <2 x i8> @g(<2 x i8> %x, <2 x i8> %y) { %x0 = extractelement <2 x i8> %x, i32 0 %y1 = extractelement <2 x i8> %y, i32 1 %x0x0 = mul i8 %x0, %x0 %y1y1 = mul i8 %y1, %y1 %ins1 = insertelement <2 x i8> undef, i8 %x0x0, i32 0 %ins2 = insertelement <2 x i8> %ins1, i8 %y1y1, i32 1 ret <2 x i8> %ins2 } ``` can be converted to something like ``` define <2 x i8> @g(<2 x i8> %x, <2 x i8> %y) { %1 = shufflevector <2 x i8> %x, <2 x i8> %y, <2 x i32> <i32 0, i32 3> %2 = mul <2 x i8> %1, %1 ret <2 x i8> %2 } ``` Currently this type of conversion is considered as high cost transformation. Reviewers: mzolotukhin, delena, mkuper, hfinkel, RKSimon Subscribers: ashahid, RKSimon, spatel, llvm-commits Differential Revision: https://reviews.llvm.org/D30200 llvm-svn: 309812
* [SLPVectorizer] Unbreak the build with -Werror.Davide Italiano2017-07-311-3/+3
| | | | | | | GCC was complaining about `&&` within `||` without explicit parentheses. NFCI. llvm-svn: 309606
* [SLP] Initial rework for min/max horizontal reduction vectorization, NFC.Alexey Bataev2017-07-311-47/+158
| | | | | | | | | | | | Summary: All getReductionCost() functions are renamed to getArithmeticReductionCost() + added basic infrastructure to handle non-binary reduction operations. Reviewers: spatel, mzolotukhin, Ayal, mkuper, gilr, hfinkel Subscribers: RKSimon, llvm-commits Differential Revision: https://reviews.llvm.org/D29402 llvm-svn: 309566
* [Cost] Rename getReductionCost() to getArithmeticReductionCost(), NFC.Alexey Bataev2017-07-311-2/+4
| | | | llvm-svn: 309563
* [LV] Avoid redundant operations manipulating masksAyal Zaks2017-07-311-36/+37
| | | | | | | | | | | | | | | | The Loop Vectorizer generates redundant operations when manipulating masks: AND with true, OR with false, compare equal to true. Instead of relying on a subsequent pass to clean them up, this patch avoids generating them. Use null (no-mask) to represent all-one full masks, instead of a constant all-one vector, following the convention of masked gathers and scatters. Preparing for a follow-up VPlan patch in which these mask manipulating operations are modeled using recipes. Differential Revision: https://reviews.llvm.org/D35725 llvm-svn: 309558
* [SLP] Allow vectorization of the instruction from the same basic blocks ↵Alexey Bataev2017-07-281-3/+8
| | | | | | | | | | | | | | | | only, NFC. Summary: After some changes in SLP vectorizer we missed some additional checks to limit the instructions for vectorization. We should not perform analysis of the instructions if the parent of instruction is not the same as the parent of the first instruction in the tree or it was analyzed already. Subscribers: mzolotukhin Differential Revision: https://reviews.llvm.org/D34881 llvm-svn: 309425
* [SLP] Outline code for the check that instruction users are part ofAlexey Bataev2017-07-271-4/+11
| | | | | | vectorization tree, NFC. llvm-svn: 309284
* [SLPVectorizer] Replace E->Scalars to VL0 at vectorizeTree and move comment, ↵Dinar Temirbulatov2017-07-211-4/+3
| | | | | | NFCI. llvm-svn: 308750
* [SLPVectorizer] buildTree_rec replace cast<Instruction>(VL[0]) to VL0, NFCI.Dinar Temirbulatov2017-07-211-4/+4
| | | | llvm-svn: 308745
OpenPOWER on IntegriCloud