summaryrefslogtreecommitdiffstats
path: root/llvm/lib/Transforms/Vectorize
Commit message (Collapse)AuthorAgeFilesLines
* Vectorize intrinsic math function calls in SLPVectorizer.Karthik Bhat2014-05-032-143/+22
| | | | | | | This patch adds support to recognize and vectorize intrinsic math functions in SLPVectorizer. Review: http://reviews.llvm.org/D3560 and http://reviews.llvm.org/D3559 llvm-svn: 207901
* Clean up constructor logic and member access for LoopVectorizeHints.Eric Christopher2014-05-021-34/+39
| | | | | | | | | There are public functions that mutate various members as well as another private member already, so make all the members private to avoid the discontinuity and add accessors for the values. Should be no functional change. llvm-svn: 207868
* Revert r205965, which essentially reverts r205018 for the second time.Chandler Carruth2014-05-011-65/+30
| | | | | | | | | | | | | | | | =[ Turns out that this was the root cause of PR19621. We found a crasher only recently (likely due to improvements elsewhere in the SLP vectorizer) but the reduced test case failed all the way back to here. I've confirmed that reverting this patch both fixes the reduced test case in PR19621 and the actual source file that led to it, so it seems to really be rooted here. I've replied to the commit thread with discussion of my (feeble) attempts to debug this. Didn't make it very far, so reverting now that we have a good test case so that things can get back to healthy while the debugging carries on. llvm-svn: 207746
* Add a <tuple> include to more files that aren't getting it transitively on MSVC.Benjamin Kramer2014-04-301-0/+1
| | | | llvm-svn: 207617
* Fix vectorization remarks.Diego Novillo2014-04-291-6/+13
| | | | | | | | | This patch changes the vectorization remarks to also inform when vectorization is possible but not beneficial. Added tests to exercise some loop remarks. llvm-svn: 207574
* Continue slp vectorization even the BB already has vectorized store ↵Yi Jiang2014-04-291-1/+1
| | | | | | radar://16641956 llvm-svn: 207572
* Add optimization remarks to the loop unroller and vectorizer.Diego Novillo2014-04-291-0/+6
| | | | | | | | | | | | | | | Summary: This calls emitOptimizationRemark from the loop unroller and vectorizer at the point where they make a positive transformation. For the vectorizer, it reports vectorization and interleave factors. For the loop unroller, it reports all the different supported types of unrolling. Subscribers: llvm-commits Differential Revision: http://reviews.llvm.org/D3456 llvm-svn: 207528
* [BUG] Fix -Wunused-variable warning in Release mode. Thnx to Kostya ↵Zinovy Nis2014-04-291-2/+3
| | | | | | Serebryany for pointing. llvm-svn: 207516
* fix -Wunused-variable warning in Release modeKostya Serebryany2014-04-291-0/+1
| | | | llvm-svn: 207514
* [OPENMP][LV][D3423] Respect Hints.Force meta-data for loops in LoopVectorizerZinovy Nis2014-04-291-27/+57
| | | | llvm-svn: 207512
* [C++] Use 'nullptr'.Craig Topper2014-04-281-1/+1
| | | | llvm-svn: 207394
* [C++] Use 'nullptr'. Transforms edition.Craig Topper2014-04-253-97/+103
| | | | llvm-svn: 207196
* Allow vectorization of bit intrinsics in BB Vectorizer.Karthik Bhat2014-04-251-8/+21
| | | | | | This patch adds support for vectorization of bit intrinsics such as bswap,ctpop,ctlz,cttz. llvm-svn: 207174
* Allow vectorization of few missed llvm intrinsic calls in BBVectorizor by ↵Karthik Bhat2014-04-241-0/+8
| | | | | | handling them in isVectorizableIntrinsic function. llvm-svn: 207085
* [LV] Statistics numbers for LoopVectorize introduced: a number of analyzed ↵Alexander Musman2014-04-231-0/+7
| | | | | | | | | | | loops & a number of vectorized loops. Use -stats to see how many loops were analyzed for possible vectorization and how many of them were actually vectorized. Patch by Zinovy Nis Differential Revision: http://reviews.llvm.org/D3438 llvm-svn: 206956
* [Modules] Fix potential ODR violations by sinking the DEBUG_TYPEChandler Carruth2014-04-223-7/+8
| | | | | | | | | | | | | | | | | definition below all of the header #include lines, lib/Transforms/... edition. This one is tricky for two reasons. We again have a couple of passes that define something else before the includes as well. I've sunk their name macros with the DEBUG_TYPE. Also, InstCombine contains headers that need DEBUG_TYPE, so now those headers #define and #undef DEBUG_TYPE around their code, leaving them well formed modular headers. Fixing these headers was a large motivation for all of these changes, as "leaky" macros of this form are hard on the modules implementation. llvm-svn: 206844
* D3348 - [BUG] "Rotate Loop" pass kills "llvm.vectorizer.enable" metadataAlexey Bataev2014-04-151-6/+14
| | | | llvm-svn: 206266
* Reapply "SLPVectorizer: Ignore users that are insertelements we can ↵Arnold Schwaighofer2014-04-101-30/+65
| | | | | | | | | reschedule them" This commit reapplies 205018. After 205855 we should correctly vectorize intrinsics. llvm-svn: 205965
* SLPVectorizer: Only vectorize intrinsics whose operands are widened equallyArnold Schwaighofer2014-04-092-26/+10
| | | | | | | | | The vectorizer only knows how to vectorize intrinics by widening all operands by the same factor. Patch by Tyler Nowicki! llvm-svn: 205855
* Add NDEBUG markers around debug only function.Eric Christopher2014-04-071-0/+4
| | | | llvm-svn: 205706
* Add debug location information to the vectorizer debug statements.Eric Christopher2014-04-071-5/+29
| | | | | | Patch by Zinovy Nis. llvm-svn: 205705
* Fixing typo.David Blaikie2014-04-051-1/+1
| | | | | | Differential Revision: http://reviews.llvm.org/D3154 llvm-svn: 205674
* SLPVectorizer: compare entire intrinsic for SLP compatibility.Tim Northover2014-04-021-2/+2
| | | | | | | | | Some Intrinsics are overloaded to the extent that return type equality (all that's been checked up to now) does not guarantee that the arguments are the same. In these cases SLP vectorizer should not recurse into the operands, which can be achieved by comparing them as "Function *" rather than simply the ID. llvm-svn: 205424
* [LoopVectorizer] Count dependencies of consecutive pointers as uniformsHal Finkel2014-04-021-0/+10
| | | | | | | | | | | | | | | | | | | | | For the purpose of calculating the cost of the loop at various vectorization factors, we need to count dependencies of consecutive pointers as uniforms (which means that the VF = 1 cost is used for all overall VF values). For example, the TSVC benchmark function s173 has: ... %3 = add nsw i64 %indvars.iv, 16000 %arrayidx8 = getelementptr inbounds %struct.GlobalData* @global_data, i64 0, i32 0, i64 %3 ... and we must realize that the add will be a scalar in order to correctly deduce it to be profitable to vectorize this on PowerPC with VSX enabled. In fact, all dependencies of a consecutive pointer must be a scalar (uniform), and so we simply need to add all consecutive pointers to the worklist that currently detects collects uniforms. Fixes PR19296. llvm-svn: 205387
* Revert "SLPVectorizer: Ignore users that are insertelements we can ↵Arnold Schwaighofer2014-03-311-65/+30
| | | | | | | | | | | | | | reschedule them" This reverts commit r205018. Conflicts: lib/Transforms/Vectorize/SLPVectorizer.cpp test/Transforms/SLPVectorizer/X86/insert-element-build-vector.ll This is breaking libclc build. llvm-svn: 205260
* SLPVectorizer: Take credit for free extractelement instructionsArnold Schwaighofer2014-03-281-2/+11
| | | | | | | | | Extract element instructions that will be removed when vectorzing lower the cost. Patch by Arch D. Robison! llvm-svn: 205020
* SLPVectorizer: Fix typosArnold Schwaighofer2014-03-281-2/+2
| | | | | | Patch by Arch D. Robison! llvm-svn: 205019
* SLPVectorizer: Ignore users that are insertelements we can reschedule themArnold Schwaighofer2014-03-281-30/+65
| | | | | | Patch by Arch D. Robison! llvm-svn: 205018
* SLP vectorizer: Don't hoist vector extracts of phis.Andrew Trick2014-03-251-6/+1
| | | | | | | | | | | | | Extracts coming from phis were being hoisted, while all others were sunk to their uses. This was inconsistent and didn't seem to serve a purpose. Changing all extracts to be sunk to uses is a prerequisite for adding block frequency to the SLP vectorizer's cost model. I benchmarked the change in isolation (without block frequency). I only saw noise on x86 and some potentially significant improvements on ARM. No major regressions is good enough for me. llvm-svn: 204699
* [LV] While I'm here, use range based for loops which are so much cleanerChandler Carruth2014-03-181-7/+7
| | | | | | for this kind of walk. llvm-svn: 204188
* [LV] The actual change I intended to commit in r204148. Sorry for theChandler Carruth2014-03-181-8/+1
| | | | | | | | | | | | | noise. Original commit log: Replace some dead code with an assert. When I first ported this pass from a loop pass to a function pass I did so in the naive, recursive way. It doesn't actually work, we need a worklist instead. When I switched to the worklist I didn't delete the naive recursion. That recursion was also buggy because it was dead and never really exercised. llvm-svn: 204187
* [LV] Replace some dead code with an assert. When I first ported thisChandler Carruth2014-03-181-2/+2
| | | | | | | | | | pass from a loop pass to a function pass I did so in the naive, recursive way. It doesn't actually work, we need a worklist instead. When I switched to the worklist I didn't delete the naive recursion. That recursion was also buggy because it was dead and never really exercised. llvm-svn: 204184
* Resubmit "[SLPV] Recognize vectorizable intrinsics during SLP vectorization ..."Raul E. Silvera2014-03-121-0/+83
| | | | | | | This reverts commit 86cb795388643710dab34941ddcb5a9470ac39d8. The problems previously found have been resolved through other CLs. llvm-svn: 203707
* Fix build break.Ahmed Charles2014-03-091-0/+2
| | | | llvm-svn: 203366
* [C++11] Add range based accessors for the Use-Def chain of a Value.Chandler Carruth2014-03-093-92/+88
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This requires a number of steps. 1) Move value_use_iterator into the Value class as an implementation detail 2) Change it to actually be a *Use* iterator rather than a *User* iterator. 3) Add an adaptor which is a User iterator that always looks through the Use to the User. 4) Wrap these in Value::use_iterator and Value::user_iterator typedefs. 5) Add the range adaptors as Value::uses() and Value::users(). 6) Update *all* of the callers to correctly distinguish between whether they wanted a use_iterator (and to explicitly dig out the User when needed), or a user_iterator which makes the Use itself totally opaque. Because #6 requires churning essentially everything that walked the Use-Def chains, I went ahead and added all of the range adaptors and switched them to range-based loops where appropriate. Also because the renaming requires at least churning every line of code, it didn't make any sense to split these up into multiple commits -- all of which would touch all of the same lies of code. The result is still not quite optimal. The Value::use_iterator is a nice regular iterator, but Value::user_iterator is an iterator over User*s rather than over the User objects themselves. As a consequence, it fits a bit awkwardly into the range-based world and it has the weird extra-dereferencing 'operator->' that so many of our iterators have. I think this could be fixed by providing something which transforms a range of T&s into a range of T*s, but that *can* be separated into another patch, and it isn't yet 100% clear whether this is the right move. However, this change gets us most of the benefit and cleans up a substantial amount of code around Use and User. =] llvm-svn: 203364
* LoopVectorizer: Preserve fast-math flagsArnold Schwaighofer2014-03-051-5/+21
| | | | | | Fixes PR19045. llvm-svn: 203008
* [C++11] Add 'override' keyword to virtual methods that override their base ↵Craig Topper2014-03-053-11/+12
| | | | | | class. llvm-svn: 202953
* [Modules] Move ValueHandle into the IR library where Value itself lives.Chandler Carruth2014-03-042-2/+2
| | | | | | | | | | | Move the test for this class into the IR unittests as well. This uncovers that ValueMap too is in the IR library. Ironically, the unittest for ValueMap is useless in the Support library (honestly, so was the ValueHandle test) and so it already lives in the IR unittests. Mmmm, tasty layering. llvm-svn: 202821
* [Modules] Move the LLVM IR pattern match header into the IR library, itChandler Carruth2014-03-041-1/+1
| | | | | | obviously is coupled to the IR. llvm-svn: 202818
* [C++11] Replace llvm::tie with std::tie.Benjamin Kramer2014-03-021-2/+2
| | | | | | The old implementation is no longer needed in C++11. llvm-svn: 202644
* [C++11] Replace llvm::next and llvm::prior with std::next and std::prev.Benjamin Kramer2014-03-023-9/+9
| | | | | | Remove the old functions. llvm-svn: 202636
* Now that we have C++11, turn simple functors into lambdas and remove a ton ↵Benjamin Kramer2014-03-011-11/+4
| | | | | | | | of boilerplate. No intended functionality change. llvm-svn: 202588
* Make DataLayout a plain object, not a pass.Rafael Espindola2014-02-253-4/+8
| | | | | | | Instead, have a DataLayoutPass that holds one. This will allow parts of LLVM don't don't handle passes to also use DataLayout. llvm-svn: 202168
* Make a few more DataLayout variables const.Rafael Espindola2014-02-251-4/+4
| | | | llvm-svn: 202155
* Make some DataLayout pointers const.Rafael Espindola2014-02-242-20/+20
| | | | | | No functionality change. Just reduces the noise of an upcoming patch. llvm-svn: 202087
* SLPVectorizer: Try vectorizing 'splat' storesArnold Schwaighofer2014-02-241-3/+7
| | | | | | | | | Vectorize sequential stores of a broadcasted value. 5% on eon. radar://16124699 llvm-svn: 202067
* Rename many DataLayout variables from TD to DL.Rafael Espindola2014-02-211-9/+9
| | | | | | | | | I am really sorry for the noise, but the current state where some parts of the code use TD (from the old name: TargetData) and other parts use DL makes it hard to write a patch that changes where those variables come from and how they are passed along. llvm-svn: 201827
* fix for null VectorizedValue assertion in the SLP Vectorizer (in function ↵Gerolf Hoflehner2014-02-171-2/+4
| | | | | | vectorizeTree()). radar://16064178 llvm-svn: 201501
* fixed typo in comment as my test commitGerolf Hoflehner2014-02-161-1/+1
| | | | llvm-svn: 201486
* Reduce code duplication resulting from the ConstantVector/ConstantDataVector ↵Benjamin Kramer2014-02-132-22/+9
| | | | | | | | split. No intended functionality change. llvm-svn: 201344
OpenPOWER on IntegriCloud