summaryrefslogtreecommitdiffstats
path: root/llvm/lib/Transforms/Utils/LoopVersioning.cpp
Commit message (Collapse)AuthorAgeFilesLines
* [LoopVersioning] Don't modify the list that we iterate over in addPHINodesBjorn Pettersson2018-05-221-3/+6
| | | | | | | | | | | | | | | | | | | | | | | | | | | Summary: In LoopVersioning::addPHINodes we need to iterate over all users for a value "Inst", and if the user is outside of the VersionedLoop we should replace the use of "Inst" by using the value "PN" instead. Replacing the use of "Inst" for a user of "Inst" also means that Inst->users() is modified. So it is not safe to do the replace while iterating over Inst->users() as we used to do. This patch splits the task into two steps. First we iterate over Inst->users() to find all users that should be updated. Those users are saved into a local data structure on the stack. And then, in the second step, we do the actual updates. This time iterating over the local data structure. Reviewers: mzolotukhin, anemet Reviewed By: mzolotukhin Subscribers: llvm-commits Differential Revision: https://reviews.llvm.org/D47134 llvm-svn: 332958
* Remove \brief commands from doxygen comments.Adrian Prantl2018-05-011-1/+1
| | | | | | | | | | | | | | | | We've been running doxygen with the autobrief option for a couple of years now. This makes the \brief markers into our comments redundant. Since they are a visual distraction and we don't want to encourage more \brief markers in new code either, this patch removes them all. Patch produced by for i in $(git grep -l '\\brief'); do perl -pi -e 's/\\brief //g' $i & done Differential Revision: https://reviews.llvm.org/D46290 llvm-svn: 331272
* [LoopVersioning] Require loop-simplify form for loop versioning.Florian Hahn2016-12-191-3/+3
| | | | | | | | | | | | | | | | Summary: Requiring loop-simplify form for loop versioning ensures that the runtime check block always dominates the exit block. This patch closes #30958 (https://llvm.org/bugs/show_bug.cgi?id=30958). Reviewers: silviu.baranga, hfinkel, anemet, ashutosh.nema Subscribers: ashutosh.nema, mzolotukhin, efriedma, hfinkel, llvm-commits Differential Revision: https://reviews.llvm.org/D27469 llvm-svn: 290116
* Rename LoopAccessAnalysis to LoopAccessLegacyAnalysis /NFCXinliang David Li2016-07-081-3/+3
| | | | llvm-svn: 274927
* [PM] refactor LoopAccessInfo code part-2Xinliang David Li2016-07-011-3/+3
| | | | | | Differential Revision: http://reviews.llvm.org/D21636 llvm-svn: 274334
* [LAA] Default getInfo to not speculate symbolic strides. NFCAdam Nemet2016-06-161-1/+1
| | | | | | | Soon we won't be passing Strides to getInfo and then we'll have fewer call sites to update. llvm-svn: 272878
* [LoopVer] Remove an assert that's redundant now. NFCAdam Nemet2016-06-141-4/+1
| | | | | | | | Ensuring that the PHI are all single-operand is not performed in the second pass added by the previous pass. This removes the assert from the first pass. llvm-svn: 272650
* [LoopVer] Update all existing PHIs in the exit blockAdam Nemet2016-06-141-6/+18
| | | | | | | | | | | | | | | | | | | | | | We only used to add the edge from the cloned loop to PHIs that corresponded to values defined by the loop. We need to do this for all PHIs obviously since we need a PHI operand for each incoming edge. This includes things like PHIs with a constant value or with values defined before the original loop (see the testcases). After the patch the PHIs are added to the exit block in two passes. In the first pass we ensure there is a single-operand (LCSSA) PHI for each value defined by the loop. In the second pass we loop through each (single-operand) PHI and add the value for the edge from the cloned loop. If the value is defined in the loop we'll use the cloned instruction from the cloned loop. Fixes PR28037 llvm-svn: 272649
* Fix a typo in loop versioning.Vikram TV2016-06-131-1/+1
| | | | | | | | | | Reviewers: ashutosh.nema Subscribers: llvm-commits Differential Revision: http://reviews.llvm.org/D21281 llvm-svn: 272545
* Revert r272194 No need for it if loop Analysis Manager is usedXinliang David Li2016-06-091-1/+1
| | | | llvm-svn: 272243
* [PM] Refector LoopAccessInfo analysis code Xinliang David Li2016-06-081-1/+1
| | | | | | | | This is the preparation patch to port the analysis to new PM Differential Revision: http://reviews.llvm.org/D20560 llvm-svn: 272194
* Remove some 'const' specifiers that do nothing but prevent moving the argument.Benjamin Kramer2016-05-291-1/+1
| | | | | | | Found by clang-tidy's misc-move-const-arg. While there drop some obsolete c_str() calls. llvm-svn: 271181
* [LoopVersioning] Relax an assert for LCSSA PHIsAdam Nemet2016-03-221-3/+4
| | | | | | | | | | | | | When you have multiple LCSSA (single-operand) PHIs that are converted into two-operand PHIs due to versioning, only assert that the PHI currently being converted has a single operand. I.e. we don't want to check PHIs that were converted earlier in the loop. Fixes PR27023. Thanks to Karl-Johan Karlsson for the minimized testcase! llvm-svn: 264081
* [LoopVectorize] Annotate versioned loop with noalias metadataAdam Nemet2016-03-171-9/+13
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Summary: Use the new LoopVersioning facility (D16712) to add noalias metadata in the vector loop if we versioned with memchecks. This can enable some optimization opportunities further down the pipeline (see the included test or the benchmark improvement quoted in D16712). The test also covers the bug I had in the initial version in D16712. The vectorizer did not previously use LoopVersioning. The reason is that the vectorizer performs its transformations in single shot. It creates an empty single-block vector loop that it then populates with the widened, if-converted instructions. Thus creating an intermediate versioned scalar loop seems wasteful. So this patch (rather than bringing in LoopVersioning fully) adds a special interface to LoopVersioning to allow the vectorizer to add no-alias annotation while still performing its own versioning. As the vectorizer propagates metadata from the instructions in the original loop to the vector instructions we also check the pointer in the original instruction and see if LoopVersioning can add no-alias metadata based on the issued memchecks. Reviewers: hfinkel, nadav, mzolotukhin Subscribers: mzolotukhin, llvm-commits Differential Revision: http://reviews.llvm.org/D17191 llvm-svn: 263744
* [LoopVersioning] Annotate versioned loop with noalias metadataAdam Nemet2016-03-171-0/+89
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Summary: If we decide to version a loop to benefit a transformation, it makes sense to record the now non-aliasing accesses in the newly versioned loop. This allows non-aliasing information to be used by subsequent passes. One example is 456.hmmer in SPECint2006 where after loop distribution, we vectorize one of the newly distributed loops. To vectorize we version this loop to fully disambiguate may-aliasing accesses. If we add the noalias markers, we can use the same information in a later DSE pass to eliminate some dead stores which amounts to ~25% of the instructions of this hot memory-pipeline-bound loop. The overall performance improves by 18% on our ARM64. The scoped noalias annotation is added in LoopVersioning. The patch then enables this for loop distribution. A follow-on patch will enable it for the vectorizer. Eventually this should be run by default when versioning the loop but first I'd like to get some feedback whether my understanding and application of scoped noalias metadata is correct. Essentially my approach was to have a separate alias domain for each versioning of the loop. For example, if we first version in loop distribution and then in vectorization of the distributed loops, we have a different set of memchecks for each versioning. By keeping the scopes in different domains they can conveniently be defined independently since different alias domains don't affect each other. As written, I also have a separate domain for each loop. This is not necessary and we could save some metadata here by using the same domain across the different loops. I don't think it's a big deal either way. Probably the best is to review the tests first to see if I mapped this problem correctly to scoped noalias markers. I have plenty of comments in the tests. Note that the interface is prepared for the vectorizer which needs the annotateInstWithNoAlias API. The vectorizer does not use LoopVersioning so we need a way to pass in the versioned instructions. This is also why the maps have to become part of the object state. Also currently, we only have an AA-aware DSE after the vectorizer if we also run the LTO pipeline. Depending how widely this triggers we may want to schedule a DSE toward the end of the regular pass pipeline. Reviewers: hfinkel, nadav, ashutosh.nema Subscribers: mssimpso, aemerson, llvm-commits, mcrosier Differential Revision: http://reviews.llvm.org/D16712 llvm-svn: 263743
* [SCEV][LAA] Re-commit r260085 and r260086, this time with a fix for the memorySilviu Baranga2016-02-081-1/+0
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | sanitizer issue. The PredicatedScalarEvolution's copy constructor wasn't copying the Generation value, and was leaving it un-initialized. Original commit message: [SCEV][LAA] Add no wrap SCEV predicates and use use them to improve strided pointer detection Summary: This change adds no wrap SCEV predicates with: - support for runtime checking - support for expression rewriting: (sext ({x,+,y}) -> {sext(x),+,sext(y)} (zext ({x,+,y}) -> {zext(x),+,sext(y)} Note that we are sign extending the increment of the SCEV, even for the zext case. This is needed to cover the fairly common case where y would be a (small) negative integer. In order to do this, this change adds two new flags: nusw and nssw that are applicable to AddRecExprs and permit the transformations above. We also change isStridedPtr in LAA to be able to make use of these predicates. With this feature we should now always be able to work around overflow issues in the dependence analysis. Reviewers: mzolotukhin, sanjoy, anemet Subscribers: mzolotukhin, sanjoy, llvm-commits, rengolin, jmolloy, hfinkel Differential Revision: http://reviews.llvm.org/D15412 llvm-svn: 260112
* Revert r260086 and r260085. They have broken the memorySilviu Baranga2016-02-081-0/+1
| | | | | | sanitizer bots. llvm-svn: 260087
* [LoopVersioning] Don't assert when there are no memchecksSilviu Baranga2016-02-081-1/+0
| | | | | | | | | | | | We shouldn't assert when there are no memchecks, since we can have SCEV checks. There is already an assert covering the case where there are no SCEV checks or memchecks. This also changes the LAA pointer wrapping versioning test to use the loop versioning pass (this was how I managed to trigger the assert in the loop versioning pass). llvm-svn: 260086
* [LoopVersioning] Expose loop versioning as a pass tooAdam Nemet2016-02-031-0/+74
| | | | | | | | | | | | | | | | | | | | | | | | | | | Summary: LoopVersioning is a transform utility that transform passes can use to run-time disambiguate may-aliasing accesses. I'd like to also expose as pass to allow it to be unit-tested. I am planning to add support for non-aliasing annotation in LoopVersioning and I'd like to be able to write tests directly using this pass. (After that feature is done, the pass could also be used to look for optimization opportunities that are hidden behind incomplete alias information at compile time.) The pass drives LoopVersioning in its default way which is to fully disambiguate may-aliasing accesses no matter how many checks are required. Reviewers: hfinkel, ashutosh.nema, sbaranga Subscribers: zzheng, mssimpso, llvm-commits, sanjoy Differential Revision: http://reviews.llvm.org/D16612 llvm-svn: 259610
* Re-commit r255115, with the PredicatedScalarEvolution class moved toSilviu Baranga2015-12-091-2/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | ScalarEvolution.h, in order to avoid cyclic dependencies between the Transform and Analysis modules: [LV][LAA] Add a layer over SCEV to apply run-time checked knowledge on SCEV expressions Summary: This change creates a layer over ScalarEvolution for LAA and LV, and centralizes the usage of SCEV predicates. The SCEVPredicatedLayer takes the statically deduced knowledge by ScalarEvolution and applies the knowledge from the SCEV predicates. The end goal is that both LAA and LV should use this interface everywhere. This also solves a problem involving the result of SCEV expression rewritting when the predicate changes. Suppose we have the expression (sext {a,+,b}) and two predicates P1: {a,+,b} has nsw P2: b = 1. Applying P1 and then P2 gives us {a,+,1}, while applying P2 and the P1 gives us sext({a,+,1}) (the AddRec expression was changed by P2 so P1 no longer applies). The SCEVPredicatedLayer maintains the order of transformations by feeding back the results of previous transformations into new transformations, and therefore avoiding this issue. The SCEVPredicatedLayer maintains a cache to remember the results of previous SCEV rewritting results. This also has the benefit of reducing the overall number of expression rewrites. Reviewers: mzolotukhin, anemet Subscribers: jmolloy, sanjoy, llvm-commits Differential Revision: http://reviews.llvm.org/D14296 llvm-svn: 255122
* Revert r255115 until we figure out how to fix the bot failures.Silviu Baranga2015-12-091-2/+2
| | | | llvm-svn: 255117
* [LV][LAA] Add a layer over SCEV to apply run-time checked knowledge on SCEV ↵Silviu Baranga2015-12-091-2/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | expressions Summary: This change creates a layer over ScalarEvolution for LAA and LV, and centralizes the usage of SCEV predicates. The SCEVPredicatedLayer takes the statically deduced knowledge by ScalarEvolution and applies the knowledge from the SCEV predicates. The end goal is that both LAA and LV should use this interface everywhere. This also solves a problem involving the result of SCEV expression rewritting when the predicate changes. Suppose we have the expression (sext {a,+,b}) and two predicates P1: {a,+,b} has nsw P2: b = 1. Applying P1 and then P2 gives us {a,+,1}, while applying P2 and the P1 gives us sext({a,+,1}) (the AddRec expression was changed by P2 so P1 no longer applies). The SCEVPredicatedLayer maintains the order of transformations by feeding back the results of previous transformations into new transformations, and therefore avoiding this issue. The SCEVPredicatedLayer maintains a cache to remember the results of previous SCEV rewritting results. This also has the benefit of reducing the overall number of expression rewrites. Reviewers: mzolotukhin, anemet Subscribers: jmolloy, sanjoy, llvm-commits Differential Revision: http://reviews.llvm.org/D14296 llvm-svn: 255115
* [Utils] Put includes in correct order. NFC.Weiming Zhao2015-11-241-1/+0
| | | | | | | | | | | | | | | | | | | Summary: Followed the guidelines in: http://llvm.org/docs/CodingStandards.html#include-style However, I noticed that uppercase named headers come before lowercase ones throughout the codebase. So kept them as is. Patch by Mandeep Singh Grang <mgrang@codeaurora.org> Reviewers: majnemer, davide, jmolloy, atrick Subscribers: sanjoy Differential Revision: http://reviews.llvm.org/D14939 llvm-svn: 254005
* Allow LLE/LD and the loop versioning infrastructure to use SCEV predicatesSilviu Baranga2015-11-091-22/+53
| | | | | | | | | | | | | | | | | | | Summary: LAA currently generates a set of SCEV predicates that must be checked by users. In the case of Loop Distribute/Loop Load Elimination, no such predicates could have been emitted, since we don't allow stride versioning. However, in the future there could be SCEV predicates that will need to be checked. This change adds support for SCEV predicate versioning in the Loop Distribute, Loop Load Eliminate and the loop versioning infrastructure. Reviewers: anemet Subscribers: mssimpso, sanjoy, llvm-commits Differential Revision: http://reviews.llvm.org/D14240 llvm-svn: 252467
* Move the canonical header to the top of its matching cpp file as per coding ↵David Blaikie2015-10-261-1/+2
| | | | | | | | | convention This ensures that the header will be verified to be standalone (and avoid mistakes like the one fixed in r251178) llvm-svn: 251326
* TransformUtils: Remove implicit ilist iterator conversions, NFCDuncan P. N. Exon Smith2015-10-131-1/+1
| | | | | | | | | | | Continuing the work from last week to remove implicit ilist iterator conversions. First related commit was probably r249767, with some more motivation in r249925. This edition gets LLVMTransformUtils compiling without the implicit conversions. No functional change intended. llvm-svn: 250142
* [LVer] Fix FIXME: hide addPHINodes, NFCAdam Nemet2015-08-201-1/+6
| | | | | | | | | | | | | | Since Ashutosh made findDefsUsedOutsideOfLoop public, we can clean this up. Now clients that don't compute DefsUsedOutsideOfLoop can just call versionLoop() and computing DefsUsedOutsideOfLoop will happen implicitly. With that there is no reason to expose addPHINodes anymore. Ashutosh, you can now drop the calls to findDefsUsedOutsideOfLoop and addPHINodes in LVerLICM and things should just work. llvm-svn: 245579
* [LVer] Remove unused Pass parameter from versionLoop, NFCAdam Nemet2015-08-141-1/+1
| | | | llvm-svn: 245032
* [LoopVer] Optionally allow using memchecks from LAAAdam Nemet2015-08-121-0/+9
| | | | | | | | | | | r243382 changed the behavior to always require a set of memchecks to be passed to LoopVer. This change restores the prior behavior as an alternative to the new behavior. This allows the checks to be implicitly taken from the LAA object. Patch by Ashutosh Nema! llvm-svn: 244763
* [LAA] Change name from addRuntimeCheck to addRuntimeChecks, NFCAdam Nemet2015-08-111-1/+1
| | | | | | This was requested by Hal in D11205. llvm-svn: 244540
* [LoopVer] Remove unused pointer partition argument, NFC.Adam Nemet2015-08-101-2/+1
| | | | llvm-svn: 244527
* [LoopVer] Remove unused needsRuntimeChecks(), NFCAdam Nemet2015-08-031-7/+2
| | | | | | | | The previous commits moved this functionality into the client. Also remove the now unused member variable. llvm-svn: 243920
* [LoopVer] Add missing std::moveAdam Nemet2015-07-301-1/+2
| | | | | | | | | | | | The reason I was passing this vector by value in the constructor so that I wouldn't have to copy when initializing the corresponding member but then I forgot the std::move. The use-case is LoopDistribution which filters the checks then std::moves it to LoopVersioning's constructor. With this interface we can avoid any copies. llvm-svn: 243616
* [LDist][LVer] Explicitly pass the set of memchecks to LoopVersioning, NFCAdam Nemet2015-07-281-5/+6
| | | | | | | | | | | | | | | | | | Before the patch, the checks were generated internally in addRuntimeCheck. Now, we use the new overloaded version of addRuntimeCheck that takes the ready-made set of checks as a parameter. The checks are now generated by the client (LoopDistribution) with the new RuntimePointerChecking::generateChecks API. Also the new printChecks API is used to print out the checks for debugging. This is to continue the transition over to the new model whereby clients will get the full set of checks from LAA, filter it and then pass it to LoopVersioning and in turn to addRuntimeCheck. llvm-svn: 243382
* [LAA] Lift RuntimePointerCheck out of LoopAccessInfo, NFCAdam Nemet2015-07-141-1/+1
| | | | | | | | | I am planning to add more nested classes inside RuntimePointerCheck so all these triple-nesting would be hard to follow. Also rename it to RuntimePointerChecking (i.e. append 'ing'). llvm-svn: 242218
* [LoopDist/LoopVer] Move LoopVersioning to a new module, NFCAdam Nemet2015-07-101-0/+106
Summary: The class will obviously need improvement down the road. For one, there is no reason that addPHINodes would have to be exposed like that. I will make this and other improvements in follow-up patches. The main goal is to be able to share this functionality. The LoopLoadElimination pass I am working on needs it too. Later we can move other clients as well (LV and Ashutosh's LICMVer). Reviewers: hfinkel, ashutosh.nema Subscribers: llvm-commits Differential Revision: http://reviews.llvm.org/D10577 llvm-svn: 241932
OpenPOWER on IntegriCloud