summaryrefslogtreecommitdiffstats
path: root/llvm/lib/Transforms/Vectorize
Commit message (Collapse)AuthorAgeFilesLines
...
* Recommit the patch "Use uniforms set to populate VecValuesToIgnore".Wei Mi2016-07-191-49/+40
| | | | | | | | | | | | | | | | | | For instructions in uniform set, they will not have vector versions so add them to VecValuesToIgnore. For induction vars, those only used in uniform instructions or consecutive ptrs instructions have already been added to VecValuesToIgnore above. For those induction vars which are only used in uniform instructions or non-consecutive/non-gather scatter ptr instructions, the related phi and update will also be added into VecValuesToIgnore set. The change will make the vector RegUsages estimation less conservative. Differential Revision: https://reviews.llvm.org/D20474 The recommit fixed the testcase global_alias.ll. llvm-svn: 275936
* Revert rL275912.Wei Mi2016-07-181-40/+49
| | | | llvm-svn: 275915
* Use uniforms set to populate VecValuesToIgnore.Wei Mi2016-07-181-49/+40
| | | | | | | | | | | | | | | | For instructions in uniform set, they will not have vector versions so add them to VecValuesToIgnore. For induction vars, those only used in uniform instructions or consecutive ptrs instructions have already been added to VecValuesToIgnore above. For those induction vars which are only used in uniform instructions or non-consecutive/non-gather scatter ptr instructions, the related phi and update will also be added into VecValuesToIgnore set. The change will make the vector RegUsages estimation less conservative. Differential Revision: https://reviews.llvm.org/D20474 llvm-svn: 275912
* [LV] Swap A and B in interleaved access analysis (NFC)Matthew Simpson2016-07-151-77/+87
| | | | | | | | | This patch swaps A and B in the interleaved access analysis and clarifies related comments. The algorithm is more intuitive if we let access A precede access B in program order rather than the reverse. This change was requested in the review of D19984. llvm-svn: 275567
* [LV] Rename StrideAccesses to AccessStrideInfo (NFC)Matthew Simpson2016-07-141-12/+12
| | | | | | | We now collect all accesses with a constant stride, not just the ones with a stride greater than one. This change was requested in the review of D19984. llvm-svn: 275473
* [LV] Allow interleaved accesses in loops with predicated blocksMatthew Simpson2016-07-141-34/+30
| | | | | | | | | | This patch allows the formation of interleaved access groups in loops containing predicated blocks. However, the predicated accesses are prevented from forming groups. Differential Revision: https://reviews.llvm.org/D19694 llvm-svn: 275471
* [LV] Avoid unnecessary IV scalar-to-vector-to-scalar conversionsMatthew Simpson2016-07-141-45/+69
| | | | | | | | | | | | This patch prevents increases in the number of instructions, pre-instcombine, due to induction variable scalarization. An increase in instructions can lead to an increase in the compile-time required to simplify the induction variables. We now maintain a new map for scalarized induction variables to prevent us from converting between the scalar and vector forms. This patch should resolve compile-time regressions seen after r274627. llvm-svn: 275419
* Extended LoadStoreVectorizer to vectorize subchains.Alina Sbirlea2016-07-131-64/+118
| | | | | | | | | | | | | | Summary: LSV used to abort vectorizing a chain for interleaved load/store accesses that alias. Allow a valid prefix of the chain to be vectorized, mark just the prefix and retry vectorizing the remaining chain. Reviewers: llvm-commits, jlebar, arsenm Subscribers: mzolotukhin Differential Revision: http://reviews.llvm.org/D22119 llvm-svn: 275317
* [LoopVectorize] Further cleanupsDavid Majnemer2016-07-131-4/+2
| | | | | | No functional change is intended, just a minor cleanup. llvm-svn: 275243
* [LV] Do not invalidate use-lists we're iterating over.Michael Kuperstein2016-07-121-14/+15
| | | | | | Should make sanitizers happier. llvm-svn: 275230
* [LV] Remove wrong assumption about LCSSAMichael Kuperstein2016-07-121-5/+0
| | | | | | | | | The LCSSA pass itself will not generate several redundant PHI nodes in a single exit block. However, such redundant PHI nodes don't violate LCSSA form, and may be introduced by passes that preserve LCSSA, and/or preserved by the LCSSA pass itself. So, assuming a single PHI node per exit block is not safe. llvm-svn: 275217
* [LoopVectorize] Assorted cleanupsDavid Majnemer2016-07-121-223/+184
| | | | | | | | | Use range-based for loops instead of doing everything manually. Use auto when appropriate. No functional change is intended. llvm-svn: 275205
* Correct ordering of loads/stores.Alina Sbirlea2016-07-111-15/+33
| | | | | | | | | | | | | | | | | | | | | | | | | | Summary: Aiming to correct the ordering of loads/stores. This patch changes the insert point for loads to the position of the first load. It updates the ordering method for loads to insert before, rather than after. Before this patch the following sequence: "load a[1], store a[1], store a[0], load a[2]" Would incorrectly vectorize to "store a[0,1], load a[1,2]". The correctness check was assuming the insertion point for loads is at the position of the first load, when in practice it was at the last load. An alternative fix would have been to invert the correctness check. The current fix changes insert position but also requires reordering of instructions before the vectorized load. Updated testcases to reflect the changes. Reviewers: tstellarAMD, llvm-commits, jlebar, arsenm Subscribers: mzolotukhin Differential Revision: http://reviews.llvm.org/D22071 llvm-svn: 275117
* Add TLI.allowsMisalignedMemoryAccesses to LoadStoreVectorizerAlina Sbirlea2016-07-111-24/+34
| | | | | | | | | | | | | Summary: Extend TTI to access TLI.allowsMisalignedMemoryAccesses(). Check condition when vectorizing load and store chains. Add additional parameters: AddressSpace, Alignment, Fast. Reviewers: llvm-commits, jlebar Subscribers: arsenm, mzolotukhin Differential Revision: http://reviews.llvm.org/D21935 llvm-svn: 275100
* Give helper classes/functions internal linkage. NFC.Benjamin Kramer2016-07-101-0/+2
| | | | llvm-svn: 275014
* [PM] Port LoopVectorize to the new PM.Sean Silva2016-07-091-338/+371
| | | | llvm-svn: 275000
* [PM] Fix a think-o. mv {Scalar,Vectorize}/SLPVectorize.hSean Silva2016-07-091-1/+1
| | | | llvm-svn: 274960
* Rename LoopAccessAnalysis to LoopAccessLegacyAnalysis /NFCXinliang David Li2016-07-081-6/+6
| | | | llvm-svn: 274927
* Remove duplicate inclusion /NFCXinliang David Li2016-07-081-1/+0
| | | | llvm-svn: 274921
* Add a missing semicolon.Rui Ueyama2016-07-071-1/+1
| | | | llvm-svn: 274794
* Clang-format LoadStoreVectorizerAlina Sbirlea2016-07-071-35/+33
| | | | | | | | | | Reviewers: llvm-commits, jlebar, arsenm Subscribers: mzolotukhin Differential Revision: http://reviews.llvm.org/D22107 llvm-svn: 274792
* Fixed a bug in vectorizing GEP before gather/scatter intrinsic.Elena Demikhovsky2016-07-071-8/+11
| | | | | | | | | | Vectorizing GEP was incorrect and broke SSA in some cases. The patch fixes PR27997 https://llvm.org/bugs/show_bug.cgi?id=27997. Differential revision: http://reviews.llvm.org/D22035 llvm-svn: 274735
* [LV] Don't widen trivial induction variablesMatthew Simpson2016-07-061-8/+58
| | | | | | | | | | | | | | | | | | | | We currently always vectorize induction variables. However, if an induction variable is only used for counting loop iterations or computing addresses with getelementptr instructions, we don't need to do this. Vectorizing these trivial induction variables can create vector code that is difficult to simplify later on. This is especially true when the unroll factor is greater than one, and we create vector arithmetic when computing step vectors. With this patch, we check if an induction variable is only used for counting iterations or computing addresses, and if so, scalarize the arithmetic when computing step vectors instead. This allows for greater simplification. This patch addresses the suboptimal pointer arithmetic sequence seen in PR27881. Reference: https://llvm.org/bugs/show_bug.cgi?id=27881 Differential Revision: http://reviews.llvm.org/D21620 llvm-svn: 274627
* [LV] Refactor integer induction widening (NFC)Matthew Simpson2016-07-051-87/+86
| | | | | | | | | This patch also removes the SCEV variants of getStepVector() since they have no uses after the refactoring. Differential Revision: http://reviews.llvm.org/D21903 llvm-svn: 274558
* LoadStoreVectorizer: Fix warning about extra semicolonMatt Arsenault2016-07-011-2/+2
| | | | llvm-svn: 274406
* Address two correctness issues in LoadStoreVectorizerAlina Sbirlea2016-07-011-3/+10
| | | | | | | | | | | | | | Summary: GetBoundryInstruction returns the last instruction as the instruction which follows or end(). Otherwise the last instruction in the boundry set is not being tested by isVectorizable(). Partially solve reordering of instructions. More extensive solution to follow. Reviewers: tstellarAMD, llvm-commits, jlebar Subscribers: escha, arsenm, mzolotukhin Differential Revision: http://reviews.llvm.org/D21934 llvm-svn: 274389
* [PM] refactor LoopAccessInfo code part-2Xinliang David Li2016-07-011-1/+1
| | | | | | Differential Revision: http://reviews.llvm.org/D21636 llvm-svn: 274334
* LoadStoreVectorizer: improvements: better pointer analysisMatt Arsenault2016-07-011-5/+26
| | | | | | | | | If OpB has an ADD NSW/NUW, we can use that to prove that adding 1 to OpA won't wrap if OpA + 1 == OpB. Patch by Fiona Glaser llvm-svn: 274324
* LoadStoreVectorizer: Don't increase alignment with no align setMatt Arsenault2016-07-011-2/+18
| | | | | | | If no alignment was set on the load/stores, it would vectorize to the new type even though this increases the default alignment. llvm-svn: 274323
* LoadStoreVectorizer: Check TTI for vec reg bit widthMatt Arsenault2016-07-011-16/+28
| | | | llvm-svn: 274322
* LoadStoreVectorizer: Fix assert when merging pointer opsMatt Arsenault2016-07-011-3/+17
| | | | | | | This needs to use inttoptr/ptrtoint if combining an int and pointer load. If a pointer is used always do an integer load. llvm-svn: 274321
* LoadStoreVectorizer: Use AA metadataMatt Arsenault2016-07-011-8/+5
| | | | | | | This was not passing the full instruction with metadata to the alias query. llvm-svn: 274318
* LoadStoreVectorizer: if one element of a vector is integer, default toMatt Arsenault2016-07-011-2/+18
| | | | | | | | | | | | integer. Fixes issues on some architectures where we use arithmetic ops to build vectors, which can cause bad things to happen for loads/stores of mixed types. Patch by Fiona Glaser llvm-svn: 274307
* LoadStoreVectorizer: Fix crashes on sub-byte typesMatt Arsenault2016-07-011-2/+14
| | | | llvm-svn: 274306
* LoadStoreVectorizer: Check skipFunction first.Matt Arsenault2016-06-301-4/+4
| | | | | | Also add test I forgot to add to r274296. llvm-svn: 274299
* LoadStoreVectorizer: Skip optnone functionsMatt Arsenault2016-06-301-1/+1
| | | | llvm-svn: 274296
* Add LoadStoreVectorizer passMatt Arsenault2016-06-303-1/+827
| | | | | | | This was contributed by Apple, and I've been working on minimal cleanups and generalizing it. llvm-svn: 274293
* Don't use unchecked dyn_castMatt Arsenault2016-06-301-1/+1
| | | | llvm-svn: 274282
* SLPVectorizer: Move propagateMetadata to VectorUtilsMatt Arsenault2016-06-302-68/+7
| | | | | | | | This will be re-used by the LoadStoreVectorizer. Fix handling of range metadata and testcase by Justin Lebar. llvm-svn: 274281
* Refine the set of UniformAfterVectorization instructions.Wei Mi2016-06-301-26/+71
| | | | | | | | | | Except the seed uniform instructions (conditional branch and consecutive ptr instructions), dependencies to be added into uniform set should only be used by existing uniform instructions or intructions outside of current loop. Differential Revision: http://reviews.llvm.org/D21755 llvm-svn: 274262
* [LV] Improve accuracy and formatting of function commentAdam Nemet2016-06-291-2/+3
| | | | llvm-svn: 274182
* Reverted patch 273864Elena Demikhovsky2016-06-291-8/+85
| | | | llvm-svn: 274115
* [Diag] Add getter shouldAlwaysPrint. NFCAdam Nemet2016-06-291-1/+1
| | | | | | | | | | For the new hotness attribute, the API will take the pass rather than the pass name so we can no longer play the trick of AlwaysPrint being a special pass name. This adds a getter to help the transition. There is also a corresponding clang patch. llvm-svn: 274100
* Fixed crash of SLP Vectorizer on KNLElena Demikhovsky2016-06-271-0/+2
| | | | | | | The bug is connected to vector GEPs. https://llvm.org/bugs/show_bug.cgi?id=28313 llvm-svn: 273919
* Fixed consecutive memory access detection in Loop Vectorizer.Elena Demikhovsky2016-06-271-85/+8
| | | | | | | | | | | | | | | | It did not handle correctly cases without GEP. The following loop wasn't vectorized: for (int i=0; i<len; i++) *to++ = *from++; I use getPtrStride() to find Stride for memory access and return 0 is the Stride is not 1 or -1. Re-commit rL273257 - revision: http://reviews.llvm.org/D20789 llvm-svn: 273864
* Apply clang-tidy's modernize-loop-convert to most of lib/Transforms.Benjamin Kramer2016-06-262-46/+42
| | | | | | Only minor manual fixes. No functionality change intended. llvm-svn: 273808
* [LV] Preserve order of dependences in interleaved accesses analysisMatthew Simpson2016-06-241-54/+205
| | | | | | | | | | | | | | | | | | | | | | | | | | The interleaved access analysis currently assumes that the inserted run-time pointer aliasing checks ensure the absence of dependences that would prevent its instruction reordering. However, this is not the case. Issues can arise from how code generation is performed for interleaved groups. For a load group, all loads in the group are essentially moved to the location of the first load in program order, and for a store group, all stores in the group are moved to the location of the last store. For groups having members involved in a dependence relation with any other instruction in the loop, this reordering can violate the dependence. This patch teaches the interleaved access analysis how to avoid breaking such dependences, and should fix PR27626. An assumption of the original analysis was that the accesses had been collected in "program order". The analysis was then simplified by visiting the accesses bottom-up. However, this ordering was never guaranteed for anything other than single basic block loops. Thus, this patch also enforces the desired ordering. Reference: https://llvm.org/bugs/show_bug.cgi?id=27626 Differential Revision: http://reviews.llvm.org/D19984 llvm-svn: 273687
* reverted the prev commit due to assertion failureElena Demikhovsky2016-06-211-7/+80
| | | | llvm-svn: 273258
* Fixed consecutive memory access detection in Loop Vectorizer.Elena Demikhovsky2016-06-211-80/+7
| | | | | | | | | | | | | | | It did not handle correctly cases without GEP. The following loop wasn't vectorized: for (int i=0; i<len; i++) *to++ = *from++; I use getPtrStride() to find Stride for memory access and return 0 is the Stride is not 1 or -1. Differential revision: http://reviews.llvm.org/D20789 llvm-svn: 273257
* [LAA] Enable symbolic stride speculation for all LAA clientsAdam Nemet2016-06-171-16/+1
| | | | | | | | | | | | | | | This is a functional change for LLE and LDist. The other clients (LV, LVerLICM) already had this explicitly enabled. The temporary boolean parameter to LAA is removed that allowed turning off speculation of symbolic strides. This makes LAA's caching interface LAA::getInfo only take the loop as the parameter. This makes the interface more friendly to the new Pass Manager. The flag -enable-mem-access-versioning is moved from LV to a LAA which now allows turning off speculation globally. llvm-svn: 273064
OpenPOWER on IntegriCloud