summaryrefslogtreecommitdiffstats
path: root/llvm/lib/Transforms/Scalar/LoopIdiomRecognize.cpp
Commit message (Collapse)AuthorAgeFilesLines
* [LIR] Set attributes on memset_pattern16.Ahmed Bougacha2016-04-271-0/+2
| | | | | | | | | "inferattrs" will deduce the attribute, but it will be too late for many optimizations. Set it ourselves when creating the call. Differential Revision: http://reviews.llvm.org/D17598 llvm-svn: 267762
* [LIR] Reuse variable. NFCI.Ahmed Bougacha2016-04-271-1/+1
| | | | llvm-svn: 267761
* Re-commit optimization bisect support (r267022) without new pass manager ↵Andrew Kaylor2016-04-221-1/+1
| | | | | | | | | | support. The original commit was reverted because of a buildbot problem with LazyCallGraph::SCC handling (not related to the OptBisect handling). Differential Revision: http://reviews.llvm.org/D19172 llvm-svn: 267231
* Revert "Initial implementation of optimization bisect support."Vedant Kumar2016-04-221-1/+1
| | | | | | | | This reverts commit r267022, due to an ASan failure: http://lab.llvm.org:8080/green/job/clang-stage2-cmake-RgSan_check/1549 llvm-svn: 267115
* Initial implementation of optimization bisect support.Andrew Kaylor2016-04-211-1/+1
| | | | | | | | | | | | This patch implements a optimization bisect feature, which will allow optimizations to be selectively disabled at compile time in order to track down test failures that are caused by incorrect optimizations. The bisection is enabled using a new command line option (-opt-bisect-limit). Individual passes that may be skipped call the OptBisect object (via an LLVMContext) to see if they should be skipped based on the bisect limit. A finer level of control (disabling individual transformations) can be managed through an addition OptBisect method, but this is not yet used. The skip checking in this implementation is based on (and replaces) the skipOptnoneFunction check. Where that check was being called, a new call has been inserted in its place which checks the bisect limit and the optnone attribute. A new function call has been added for module and SCC passes that behaves in a similar way. Differential Revision: http://reviews.llvm.org/D19172 llvm-svn: 267022
* [LPM] Factor all of the loop analysis usage updates into a common helperChandler Carruth2016-02-191-24/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | routine. We were getting this wrong in small ways and generally being very inconsistent about it across loop passes. Instead, let's have a common place where we do this. One minor downside is that this will require some analyses like SCEV in more places than they are strictly needed. However, this seems benign as these analyses are complete no-ops, and without this consistency we can in many cases end up with the legacy pass manager scheduling deciding to split up a loop pass pipeline in order to run the function analysis half-way through. It is very, very annoying to fix these without just being very pedantic across the board. The only loop passes I've not updated here are ones that use AU.setPreservesAll() such as IVUsers (an analysis) and the pass printer. They seemed less relevant. With this patch, almost all of the problems in PR24804 around loop pass pipelines are fixed. The one remaining issue is that we run simplify-cfg and instcombine in the middle of the loop pass pipeline. We've recently added some loop variants of these passes that would seem substantially cleaner to use, but this at least gets us much closer to the previous state. Notably, the seven loop pass managers is down to three. I've not updated the loop passes using LoopAccessAnalysis because that analysis hasn't been fully wired into LoopSimplify/LCSSA, and it isn't clear that those transforms want to support those forms anyways. They all run late anyways, so this is harmless. Similarly, LSR is left alone because it already carefully manages its forms and doesn't need to get fused into a single loop pass manager with a bunch of other loop passes. LoopReroll didn't use loop simplified form previously, and I've updated the test case to match the trivially different output. Finally, I've also factored all the pass initialization for the passes that use this technique as well, so that should be done regularly and reliably. Thanks to James for the help reviewing and thinking about this stuff, and Ben for help thinking about it as well! Differential Revision: http://reviews.llvm.org/D17435 llvm-svn: 261316
* [LIR] Avoid turning non-temporal stores into memsetHaicheng Wu2016-02-171-0/+4
| | | | | | This is to fix PR26645. llvm-svn: 261149
* [LIR] Allow merging of memsets in negatively strided loops.Chad Rosier2016-02-121-5/+7
| | | | | | Last part of PR25166. llvm-svn: 260732
* [LIR] Partially revert r252926(NFC), which introduced a very subtle change.Chad Rosier2016-02-121-8/+8
| | | | | | | | | | | | | In short, before r252926 we were comparing an unsigned (StoreSize) against an a APInt (Stride), which is fine and well. After we were zero extending the Stride and then converting to an unsigned, which is not the same thing. Obviously, Stides can also be negative. This commit just restores the original behavior. AFAICT, it's not possible to write a test case to expose the issue because the code already has checks to make sure the StoreSize can't overflow an unsigned (which prevents the Stride from overflowing an unsigned as well). llvm-svn: 260706
* [LIR] Add support for structs and hand unrolled loopsHaicheng Wu2016-01-261-46/+189
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | This is a recommit of r258620 which causes PR26293. The original message: Now LIR can turn following codes into memset: typedef struct foo { int a; int b; } foo_t; void bar(foo_t *f, unsigned n) { for (unsigned i = 0; i < n; ++i) { f[i].a = 0; f[i].b = 0; } } void test(foo_t *f, unsigned n) { for (unsigned i = 0; i < n; i += 2) { f[i] = 0; f[i+1] = 0; } } llvm-svn: 258777
* Speculatively revert r258620 as it is the likely culprid of PR26293.Quentin Colombet2016-01-251-200/+46
| | | | llvm-svn: 258703
* [LIR] Add support for structs and hand unrolled loopsHaicheng Wu2016-01-231-46/+200
| | | | | | | | | | | | | | | | | | | | | | | | | Now LIR can turn following codes into memset: typedef struct foo { int a; int b; } foo_t; void bar(foo_t *f, unsigned n) { for (unsigned i = 0; i < n; ++i) { f[i].a = 0; f[i].b = 0; } } void test(foo_t *f, unsigned n) { for (unsigned i = 0; i < n; i += 2) { f[i] = 0; f[i+1] = 0; } } llvm-svn: 258620
* [LIR] General refactoring to simplify code and the ease future code reviewHaicheng Wu2016-01-041-62/+120
| | | | | | | | | | This is a resubmission of r256336 which was reverted in r256361. The issue was the lack of the invariant check of the memset value in processLooMemSet(). The original message: Move several checks into isLegalStores. Also, delineate between those stores that are memset-able and those that are memcpy-able. llvm-svn: 256783
* Revert r256336, it caused PR25939Nico Weber2015-12-241-113/+61
| | | | llvm-svn: 256361
* [LIR] General refactoring to simplify code and the ease future code review.Chad Rosier2015-12-231-61/+113
| | | | | | | | | | Move several checks into isLegalStores. Also, delineate between those stores that are memset-able and those that are memcpy-able. http://reviews.llvm.org/D15683 Patch by Haicheng Wu <haicheng@codeaurora.org>! llvm-svn: 256336
* [LIR] Refactor code to enable future patch. NFC.Chad Rosier2015-12-211-41/+41
| | | | llvm-svn: 256159
* [SCEV] Add and use SCEVConstant::getAPInt; NFCISanjoy Das2015-12-171-2/+2
| | | | llvm-svn: 255921
* getParent() ^ 3 == getModule() ; NFCISanjay Patel2015-12-141-1/+1
| | | | llvm-svn: 255511
* [LIR] Push check into helper function. NFC.Chad Rosier2015-12-011-4/+4
| | | | llvm-svn: 254416
* [LIR] Put includes in correct order. NFC.Chad Rosier2015-11-231-1/+1
| | | | llvm-svn: 253915
* [LIR] Update some comments. NFC.Chad Rosier2015-11-191-5/+4
| | | | llvm-svn: 253603
* [LIR] Fix 80-column from previous commit.Chad Rosier2015-11-191-1/+2
| | | | llvm-svn: 253586
* [LIR] Sink checks into function to enable future refactoring. NFC.Chad Rosier2015-11-191-15/+17
| | | | | | | The purpose of this change is help delineate the memset and memcpy optimizations with the overall goal of resolving PR25520. llvm-svn: 253585
* [LIR] Use the more appropriate method. NFC.Chad Rosier2015-11-191-1/+1
| | | | llvm-svn: 253578
* Revert "Change memcpy/memset/memmove to have dest and source alignments."Pete Cooper2015-11-191-3/+3
| | | | | | | | | | This reverts commit r253511. This likely broke the bots in http://lab.llvm.org:8011/builders/clang-ppc64-elf-linux2/builds/20202 http://bb.pgr.jp/builders/clang-3stage-i686-linux/builds/3787 llvm-svn: 253543
* Change memcpy/memset/memmove to have dest and source alignments.Pete Cooper2015-11-181-3/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Note, this was reviewed (and more details are in) http://lists.llvm.org/pipermail/llvm-commits/Week-of-Mon-20151109/312083.html These intrinsics currently have an explicit alignment argument which is required to be a constant integer. It represents the alignment of the source and dest, and so must be the minimum of those. This change allows source and dest to each have their own alignments by using the alignment attribute on their arguments. The alignment argument itself is removed. There are a few places in the code for which the code needs to be checked by an expert as to whether using only src/dest alignment is safe. For those places, they currently take the minimum of src/dest alignments which matches the current behaviour. For example, code which used to read: call void @llvm.memcpy.p0i8.p0i8.i32(i8* %dest, i8* %src, i32 500, i32 8, i1 false) will now read: call void @llvm.memcpy.p0i8.p0i8.i32(i8* align 8 %dest, i8* align 8 %src, i32 500, i1 false) For out of tree owners, I was able to strip alignment from calls using sed by replacing: (call.*llvm\.memset.*)i32\ [0-9]*\,\ i1 false\) with: $1i1 false) and similarly for memmove and memcpy. I then added back in alignment to test cases which needed it. A similar commit will be made to clang which actually has many differences in alignment as now IRBuilder can generate different source/dest alignments on calls. In IRBuilder itself, a new argument was added. Instead of calling: CreateMemCpy(Dst, Src, getInt64(Size), DstAlign, /* isVolatile */ false) you now call CreateMemCpy(Dst, Src, getInt64(Size), DstAlign, SrcAlign, /* isVolatile */ false) There is a temporary class (IntegerAlignment) which takes the source alignment and rejects implicit conversion from bool. This is to prevent isVolatile here from passing its default parameter to the source alignment. Note, changes in future can now be made to codegen. I didn't change anything here, but this change should enable better memcpy code sequences. Reviewed by Hal Finkel. llvm-svn: 253511
* [LIR] Add support for creating memcpys from loops with a negative stride.Chad Rosier2015-11-131-12/+21
| | | | | | | | | | | | | | | This allows us to transform the below loop into a memcpy. void test(unsigned *__restrict__ a, unsigned *__restrict__ b) { for (int i = 2047; i >= 0; --i) { a[i] = b[i]; } } This is the memcpy version of r251518, which added support for memset with negative strided loops. llvm-svn: 253091
* Add a comment that should have made my last commit.Chad Rosier2015-11-131-0/+1
| | | | llvm-svn: 253063
* [LIR] Factor out the code to compute base ptr for negative strided loops.Chad Rosier2015-11-131-10/+15
| | | | | | This will allow for the code to be reused in the memcpy optimization. llvm-svn: 253061
* [LIR] Minor refactoring. NFCI.Chad Rosier2015-11-121-21/+44
| | | | | | | This change prevents uninteresting stores from being inserted into the list of candidate stores for memset/memcpy conversion. llvm-svn: 252926
* [LIR] General refactor to improve compile-time and simplify code.Chad Rosier2015-11-111-16/+26
| | | | | | | | | | | First create a list of candidates, then transform. This simplifies the code in that you have don't have to worry that you may be using an invalidated iterator. Previously, each time we created a memset/memcpy we would reevaluate the entire loop potentially resulting in lots of redundant work for large basic blocks. llvm-svn: 252817
* Simplify. NFC.Chad Rosier2015-11-091-4/+1
| | | | llvm-svn: 252491
* [LIR] Simplify code by making DataLayout globally accessible. NFC.Chad Rosier2015-11-061-11/+10
| | | | llvm-svn: 252317
* Typo.Chad Rosier2015-10-281-1/+1
| | | | llvm-svn: 251521
* Reapply: [LIR] Add support for creating memsets from loops with a negative ↵Chad Rosier2015-10-281-24/+32
| | | | | | | | stride. The simple fix is to prevent forming memcpy from loops with a negative stride. llvm-svn: 251518
* Revert "[LIR] Add support for creating memsets from loops with a negative ↵Chad Rosier2015-10-281-28/+24
| | | | | | | | stride." This reverts commit r251512. This is causing LNT/chomp to fail. llvm-svn: 251513
* [LIR] Add support for creating memsets from loops with a negative stride.Chad Rosier2015-10-281-24/+28
| | | | | | http://reviews.llvm.org/D14125 llvm-svn: 251512
* Typo.Chad Rosier2015-10-131-1/+1
| | | | llvm-svn: 250224
* Scalar: Remove remaining ilist iterator implicit conversionsDuncan P. N. Exon Smith2015-10-131-7/+7
| | | | | | | | | | | | | | | | | | | Remove remaining `ilist_iterator` implicit conversions from LLVMScalarOpts. This change exposed some scary behaviour in lib/Transforms/Scalar/SCCP.cpp around line 1770. This patch changes a call from `Function::begin()` to `&Function::front()`, since the return was immediately being passed into another function that takes a `Function*`. `Function::front()` started to assert, since the function was empty. Note that `Function::end()` does not point at a legal `Function*` -- it points at an `ilist_half_node` -- so the other function was getting garbage before. (I added the missing check for `Function::isDeclaration()`.) Otherwise, no functionality change intended. llvm-svn: 250211
* [SCEV] Introduce ScalarEvolution::getOne and getZero.Sanjoy Das2015-09-231-2/+2
| | | | | | | | | | | | | | | | | | Summary: It is fairly common to call SE->getConstant(Ty, 0) or SE->getConstant(Ty, 1); this change makes such uses a little bit briefer. I've refactored the call sites I could find easily to use getZero / getOne. Reviewers: hfinkel, majnemer, reames Subscribers: sanjoy, llvm-commits Differential Revision: http://reviews.llvm.org/D12947 llvm-svn: 248362
* [PM/AA] Rebuild LLVM's alias analysis infrastructure in a way compatibleChandler Carruth2015-09-091-5/+14
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | with the new pass manager, and no longer relying on analysis groups. This builds essentially a ground-up new AA infrastructure stack for LLVM. The core ideas are the same that are used throughout the new pass manager: type erased polymorphism and direct composition. The design is as follows: - FunctionAAResults is a type-erasing alias analysis results aggregation interface to walk a single query across a range of results from different alias analyses. Currently this is function-specific as we always assume that aliasing queries are *within* a function. - AAResultBase is a CRTP utility providing stub implementations of various parts of the alias analysis result concept, notably in several cases in terms of other more general parts of the interface. This can be used to implement only a narrow part of the interface rather than the entire interface. This isn't really ideal, this logic should be hoisted into FunctionAAResults as currently it will cause a significant amount of redundant work, but it faithfully models the behavior of the prior infrastructure. - All the alias analysis passes are ported to be wrapper passes for the legacy PM and new-style analysis passes for the new PM with a shared result object. In some cases (most notably CFL), this is an extremely naive approach that we should revisit when we can specialize for the new pass manager. - BasicAA has been restructured to reflect that it is much more fundamentally a function analysis because it uses dominator trees and loop info that need to be constructed for each function. All of the references to getting alias analysis results have been updated to use the new aggregation interface. All the preservation and other pass management code has been updated accordingly. The way the FunctionAAResultsWrapperPass works is to detect the available alias analyses when run, and add them to the results object. This means that we should be able to continue to respect when various passes are added to the pipeline, for example adding CFL or adding TBAA passes should just cause their results to be available and to get folded into this. The exception to this rule is BasicAA which really needs to be a function pass due to using dominator trees and loop info. As a consequence, the FunctionAAResultsWrapperPass directly depends on BasicAA and always includes it in the aggregation. This has significant implications for preserving analyses. Generally, most passes shouldn't bother preserving FunctionAAResultsWrapperPass because rebuilding the results just updates the set of known AA passes. The exception to this rule are LoopPass instances which need to preserve all the function analyses that the loop pass manager will end up needing. This means preserving both BasicAAWrapperPass and the aggregating FunctionAAResultsWrapperPass. Now, when preserving an alias analysis, you do so by directly preserving that analysis. This is only necessary for non-immutable-pass-provided alias analyses though, and there are only three of interest: BasicAA, GlobalsAA (formerly GlobalsModRef), and SCEVAA. Usually BasicAA is preserved when needed because it (like DominatorTree and LoopInfo) is marked as a CFG-only pass. I've expanded GlobalsAA into the preserved set everywhere we previously were preserving all of AliasAnalysis, and I've added SCEVAA in the intersection of that with where we preserve SCEV itself. One significant challenge to all of this is that the CGSCC passes were actually using the alias analysis implementations by taking advantage of a pretty amazing set of loop holes in the old pass manager's analysis management code which allowed analysis groups to slide through in many cases. Moving away from analysis groups makes this problem much more obvious. To fix it, I've leveraged the flexibility the design of the new PM components provides to just directly construct the relevant alias analyses for the relevant functions in the IPO passes that need them. This is a bit hacky, but should go away with the new pass manager, and is already in many ways cleaner than the prior state. Another significant challenge is that various facilities of the old alias analysis infrastructure just don't fit any more. The most significant of these is the alias analysis 'counter' pass. That pass relied on the ability to snoop on AA queries at different points in the analysis group chain. Instead, I'm planning to build printing functionality directly into the aggregation layer. I've not included that in this patch merely to keep it smaller. Note that all of this needs a nearly complete rewrite of the AA documentation. I'm planning to do that, but I'd like to make sure the new design settles, and to flesh out a bit more of what it looks like in the new pass manager first. Differential Revision: http://reviews.llvm.org/D12080 llvm-svn: 247167
* More clean up, still NFC. Remove dead variables now that the casts are gone.Nick Lewycky2015-08-191-5/+4
| | | | llvm-svn: 245420
* Clean up this file a little. Remove dead casts, casting Values to Values. ↵Nick Lewycky2015-08-191-8/+8
| | | | | | Adjust some comments for typos and whitespace. NFC. llvm-svn: 245419
* Fix three typos in comments; "easilly" -> "easily".Nick Lewycky2015-08-181-1/+1
| | | | llvm-svn: 245379
* [PM] Port ScalarEvolution to the new pass manager.Chandler Carruth2015-08-171-4/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This change makes ScalarEvolution a stand-alone object and just produces one from a pass as needed. Making this work well requires making the object movable, using references instead of overwritten pointers in a number of places, and other refactorings. I've also wired it up to the new pass manager and added a RUN line to a test to exercise it under the new pass manager. This includes basic printing support much like with other analyses. But there is a big and somewhat scary change here. Prior to this patch ScalarEvolution was never *actually* invalidated!!! Re-running the pass just re-wired up the various other analyses and didn't remove any of the existing entries in the SCEV caches or clear out anything at all. This might seem OK as everything in SCEV that can uses ValueHandles to track updates to the values that serve as SCEV keys. However, this still means that as we ran SCEV over each function in the module, we kept accumulating more and more SCEVs into the cache. At the end, we would have a SCEV cache with every value that we ever needed a SCEV for in the entire module!!! Yowzers. The releaseMemory routine would dump all of this, but that isn't realy called during normal runs of the pipeline as far as I can see. To make matters worse, there *is* actually a key that we don't update with value handles -- there is a map keyed off of Loop*s. Because LoopInfo *does* release its memory from run to run, it is entirely possible to run SCEV over one function, then over another function, and then lookup a Loop* from the second function but find an entry inserted for the first function! Ouch. To make matters still worse, there are plenty of updates that *don't* trip a value handle. It seems incredibly unlikely that today GVN or another pass that invalidates SCEV can update values in *just* such a way that a subsequent run of SCEV will incorrectly find lookups in a cache, but it is theoretically possible and would be a nightmare to debug. With this refactoring, I've fixed all this by actually destroying and recreating the ScalarEvolution object from run to run. Technically, this could increase the amount of malloc traffic we see, but then again it is also technically correct. ;] I don't actually think we're suffering from tons of malloc traffic from SCEV because if we were, the fact that we never clear the memory would seem more likely to have come up as an actual problem before now. So, I've made the simple fix here. If in fact there are serious issues with too much allocation and deallocation, I can work on a clever fix that preserves the allocations (while clearing the data) between each run, but I'd prefer to do that kind of optimization with a test case / benchmark that shows why we need such cleverness (and that can test that we actually make it faster). It's possible that this will make some things faster by making the SCEV caches have higher locality (due to being significantly smaller) so until there is a clear benchmark, I think the simple change is best. Differential Revision: http://reviews.llvm.org/D12063 llvm-svn: 245193
* [LIR] Re-instate r244880, reverted in r244884, factoring the handling ofChandler Carruth2015-08-141-3/+5
| | | | | | | | | | | | | AliasAnalysis in LoopIdiomRecognize. The previous commit to LIR, r244879, exposed some scary bug in the loop pass pipeline with an assert failure that showed up on several bots. This patch got reverted as part of getting that revision reverted, but they're actually independent and unrelated. This patch has no functional change and should be completely safe. It is also useful for my current work on the AA infrastructure. llvm-svn: 244993
* Revert "[LIR] Start leveraging the fundamental guarantees of a loop..."Renato Golin2015-08-131-15/+12
| | | | | | | This reverts commit r244879, as it broke the test-suite on SingleSource/Regression/C/2004-03-15-IndirectGoto in AArch64. llvm-svn: 244885
* Revert "[LIR] Handle access to AliasAnalysis the same way as the other ↵Renato Golin2015-08-131-5/+3
| | | | | | | | | analysis in LoopIdiomRecognize." This reverts commit r244880, as it broke the test-suite on SingleSource/Regression/C/2004-03-15-IndirectGoto in AArch64. llvm-svn: 244884
* [LIR] Handle access to AliasAnalysis the same way as the other analysisChandler Carruth2015-08-131-3/+5
| | | | | | | in LoopIdiomRecognize. This is what started me staring at this code. Now migrating it with the new AA stuff will be trivial. llvm-svn: 244880
* [LIR] Start leveraging the fundamental guarantees of a loop inChandler Carruth2015-08-131-12/+15
| | | | | | | | | | | simplified form to remove redundant checks and simplify the code for popcount recognition. We don't actually need to handle all of these cases. I've left a FIXME for one in particular until I finish inspecting to make sure we don't actually *rely* on the predicate in any way. llvm-svn: 244879
OpenPOWER on IntegriCloud