summaryrefslogtreecommitdiffstats
path: root/llvm/lib/Transforms
Commit message (Collapse)AuthorAgeFilesLines
* Move function to obtain branch weights into the BranchInst class. NFC.Juergen Ributzka2014-12-091-26/+6
| | | | | | Make this function available to other parts of LLVM. llvm-svn: 223784
* Teach instcombine to canonicalize "element extraction" from a load of anChandler Carruth2014-12-091-41/+349
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | integer and "element insertion" into a store of an integer into actual element extraction, element insertion, and vector loads and stores. Previously various parts of LLVM (including instcombine itself) would introduce integer loads and stores into the code as a way of opaquely loading and storing "bits". In some cases (such as a memcpy of std::complex<float> object) we will eventually end up using those bits in non-integer types. In order for SROA to effectively promote the allocas involved, it splits these "store a bag of bits" integer loads and stores up into the constituent parts. However, for non-alloca loads and tsores which remain, it uses integer math to recombine the values into a large integer to load or store. All of this would be "fine", except that it forces LLVM to go through integer math to combine and split up values. While this makes perfect sense for integers (and in fact is critical for bitfields to end up lowering efficiently) it is *terrible* for non-integer types, especially floating point types. We have a much more canonical way of representing the act of concatenating the bits of two SSA values in LLVM: a vector and insertelement. This patch teaching InstCombine to use this representation. With this patch applied, LLVM will no longer introduce integer math into the critical path of every loop over std::complex<float> operations such as those that make up the hot path of ... oh, most HPC code, Eigen, and any other heavy linear algebra library. For the record, I looked *extensively* at fixing this in other parts of the compiler, but it just doesn't work: - We really do want to canonicalize memcpy and other bit-motion to integer loads and stores. SSA values are tremendously more powerful than "copy" intrinsics. Not doing this regresses massive amounts of LLVM's scalar optimizer. - We really do need to split up integer loads and stores of this form in SROA or every memcpy of a trivially copyable struct will prevent SSA formation of the members of that struct. It essentially turns off SROA. - The closest alternative is to actually split the loads and stores when partitioning with SROA, but this has all of the downsides historically discussed of splitting up loads and stores -- the wide-store information is fundamentally lost. We would also see performance regressions for bitfield-heavy code and other places where the integers aren't really intended to be split without seemingly arbitrary logic to treat integers totally differently. - We *can* effectively fix this in instcombine, so it isn't that hard of a choice to make IMO. Differential Revision: http://reviews.llvm.org/D6548 llvm-svn: 223764
* InstrProf: An intrinsic and lowering for instrumentation based profilingJustin Bogner2014-12-083-0/+311
| | | | | | | | | | | | | | | | | | | | | Introduce the ``llvm.instrprof_increment`` intrinsic and the ``-instrprof`` pass. These provide the infrastructure for writing counters for profiling, as in clang's ``-fprofile-instr-generate``. The implementation of the instrprof pass is ported directly out of the CodeGenPGO classes in clang, and with the followup in clang that rips that code out to use these new intrinsics this ends up being NFC. Doing the instrumentation this way opens some doors in terms of improving the counter performance. For example, this will make it simple to experiment with alternate lowering strategies, and allows us to try handling profiling specially in some optimizations if we want to. Finally, this drastically simplifies the frontend and puts all of the lowering logic in one place. llvm-svn: 223672
* LLVMInstrumentation requires MC since r223532.NAKAMURA Takumi2014-12-061-1/+1
| | | | llvm-svn: 223573
* Utils: Style cleanups, NFCDuncan P. N. Exon Smith2014-12-061-7/+7
| | | | llvm-svn: 223556
* Utils: Avoid RAUW on metadata in CloneFunction()Duncan P. N. Exon Smith2014-12-061-4/+4
| | | | llvm-svn: 223555
* Recommit of r223513 and r223514.Kuba Brecka2014-12-051-34/+48
| | | | | | Reviewed at http://reviews.llvm.org/D6488 llvm-svn: 223532
* Reverting r223513 and r223514.Kuba Brecka2014-12-051-48/+34
| | | | llvm-svn: 223520
* [DFSAN][MIPS][LLVM] Defining ShadowPtrMask variable for MIPS64Peter Collingbourne2014-12-051-1/+12
| | | | | | | | | | | Patch by Kumar Sukhani! corresponding compiler-rt patch: http://reviews.llvm.org/D6437 clang patch: http://reviews.llvm.org/D6147 Differential Revision: http://reviews.llvm.org/D6459 llvm-svn: 223516
* AddressSanitizer - Don't instrument globals from cstring_literals sections. ↵Kuba Brecka2014-12-051-34/+48
| | | | | | | | (llvm part) Reviewed at http://reviews.llvm.org/D6488 llvm-svn: 223513
* [msan] Avoid extra origin address realignment.Evgeniy Stepanov2014-12-051-21/+24
| | | | | | | | | Do not realign origin address if the corresponding application address is at least 4-byte-aligned. Saves 2.5% code size in track-origins mode. llvm-svn: 223464
* [InstCombine] Minor optimization for bswap with binary opsSimon Pilgrim2014-12-042-0/+65
| | | | | | | | | | | | | | | | | Added instcombine optimizations for BSWAP with AND/OR/XOR ops: OP( BSWAP(x), BSWAP(y) ) -> BSWAP( OP(x, y) ) OP( BSWAP(x), CONSTANT ) -> BSWAP( OP(x, BSWAP(CONSTANT) ) ) Since its just a one liner, I've also added BSWAP to the DAGCombiner equivalent as well: fold (OP (bswap x), (bswap y)) -> (bswap (OP x, y)) Refactored bswap-fold tests to use FileCheck instead of just checking that the bswaps had gone. Differential Revision: http://reviews.llvm.org/D6407 llvm-svn: 223349
* [msan] allow -fsanitize-coverage=N together with -fsanitize=memory, llvm partKostya Serebryany2014-12-032-2/+5
| | | | llvm-svn: 223312
* correct spelling, NFCMatthias Braun2014-12-031-3/+3
| | | | llvm-svn: 223274
* [SimplifyLibCalls] Improve double->float shrinking to consider constantsMatthias Braun2014-12-031-14/+33
| | | | | | | | | | This allows cases like float x; fmin(1.0, x); to be optimized to fminf(1.0f, x); rdar://19049359 Differential Revision: http://reviews.llvm.org/D6496 llvm-svn: 223270
* [SimplifyLibCalls] Enable double to float shrinking for copysignMatthias Braun2014-12-031-0/+1
| | | | | | | | rdar://19049359 Differential Revision: http://reviews.llvm.org/D6495 llvm-svn: 223269
* msan] Add compile-time checks for missing origins.Evgeniy Stepanov2014-12-031-10/+13
| | | | | | | | | | | | This change makes MemorySanitizer instrumentation a bit more strict about instructions that have no origin id assigned to them. This would have caught the bug that was fixed in r222918. This is re-commit of r222997, reverted in r223211, with 3 more missing origins added. llvm-svn: 223236
* InstCombine: simplify signed range checksErik Eckstein2014-12-032-0/+74
| | | | | | | | | Try to convert two compares of a signed range check into a single unsigned compare. Examples: (icmp sge x, 0) & (icmp slt x, n) --> icmp ult x, n (icmp slt x, 0) | (icmp sgt x, n) --> icmp ugt x, n llvm-svn: 223224
* Revert r222997. The newly added compile-time checks are finding missing ↵Nick Lewycky2014-12-031-10/+9
| | | | | | origins, testcase is being reduced and a PR will be posted shortly. llvm-svn: 223211
* LoopVectorize: Remove unnecessary RAUWDuncan P. N. Exon Smith2014-12-031-2/+0
| | | | | | | | | | Remove an unnecessary `MDNode::replaceAllUsesWith()`. In the preceding line, `TheLoop->setLoopID()` visits all backedges and sets the new loop ID. This sufficiently updates the loop metadata. Metadata RAUW is going away as part of PR21532. llvm-svn: 223210
* StructurizeCFG: Use LoopInfo analysis for better loop detectionTom Stellard2014-12-031-1/+6
| | | | | | | | We were assuming that each back-edge in a region represented a unique loop, which is not always the case. We need to use LoopInfo to correctly determine which back-edges are loops. llvm-svn: 223199
* Emit the entry block first and the exit block second, then all the blocks in ↵Nick Lewycky2014-12-031-3/+7
| | | | | | between afterwards. This is what gcc always does, and some out of tree tools depend on that. llvm-svn: 223193
* Prologue supportPeter Collingbourne2014-12-031-0/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Patch by Ben Gamari! This redefines the `prefix` attribute introduced previously and introduces a `prologue` attribute. There are a two primary usecases that these attributes aim to serve, 1. Function prologue sigils 2. Function hot-patching: Enable the user to insert `nop` operations at the beginning of the function which can later be safely replaced with a call to some instrumentation facility 3. Runtime metadata: Allow a compiler to insert data for use by the runtime during execution. GHC is one example of a compiler that needs this functionality for its tables-next-to-code functionality. Previously `prefix` served cases (1) and (2) quite well by allowing the user to introduce arbitrary data at the entrypoint but before the function body. Case (3), however, was poorly handled by this approach as it required that prefix data was valid executable code. Here we redefine the notion of prefix data to instead be data which occurs immediately before the function entrypoint (i.e. the symbol address). Since prefix data now occurs before the function entrypoint, there is no need for the data to be valid code. The previous notion of prefix data now goes under the name "prologue data" to emphasize its duality with the function epilogue. The intention here is to handle cases (1) and (2) with prologue data and case (3) with prefix data. References ---------- This idea arose out of discussions[1] with Reid Kleckner in response to a proposal to introduce the notion of symbol offsets to enable handling of case (3). [1] http://lists.cs.uiuc.edu/pipermail/llvmdev/2014-May/073235.html Test Plan: testsuite Differential Revision: http://reviews.llvm.org/D6454 llvm-svn: 223189
* PR21302. Vectorize only bottom-tested loops.Michael Zolotukhin2014-12-021-0/+9
| | | | | | rdar://problem/18886083 llvm-svn: 223171
* [Statepoints 3/4] Statepoint infrastructure for garbage collection: ↵Philip Reames2014-12-021-0/+9
| | | | | | | | | | | | | | | | | | SelectionDAGBuilder This is the third patch in a small series. It contains the CodeGen support for lowering the gc.statepoint intrinsic sequences (223078) to the STATEPOINT pseudo machine instruction (223085). The change also includes the set of helper routines and classes for working with gc.statepoints, gc.relocates, and gc.results since the lowering code uses them. With this change, gc.statepoints should be functionally complete. The documentation will follow in the fourth change, and there will likely be some cleanup changes, but interested parties can start experimenting now. I'm not particularly happy with the amount of code or complexity involved with the lowering step, but at least it's fairly well isolated. The statepoint lowering code is split into it's own files and anyone not working on the statepoint support itself should be able to ignore it. During the lowering process, we currently spill aggressively to stack. This is not entirely ideal (and we have plans to do better), but it's functional, relatively straight forward, and matches closely the implementations of the patchpoint intrinsics. Most of the complexity comes from trying to keep relocated copies of values in the same stack slots across statepoints. Doing so avoids the insertion of pointless load and store instructions to reshuffle the stack. The current implementation isn't as effective as I'd like, but it is functional and 'good enough' for many common use cases. In the long term, I'd like to figure out how to integrate the statepoint lowering with the register allocator. In principal, we shouldn't need to eagerly spill at all. The register allocator should do any spilling required and the statepoint should simply record that fact. Depending on how challenging that turns out to be, we may invest in a smarter global stack slot assignment mechanism as a stop gap measure. Reviewed by: atrick, ributzka llvm-svn: 223137
* [SwitchLowering] Handle destinations on multiple phi instructionsBruno Cardoso Lopes2014-12-021-2/+3
| | | | | | | | | Follow up from r222926. Also handle multiple destinations from merged cases on multiple and subsequent phi instructions. rdar://problem/19106978 llvm-svn: 223135
* [LICM] Avoind store sinking if no preheader is availableBruno Cardoso Lopes2014-12-021-2/+4
| | | | | | | | | Load instructions are inserted into loop preheaders when sinking stores and later removed if not used by the SSA updater. Avoid sinking if the loop has no preheader and avoid crashes. This fixes one more side effect of not handling indirectbr instructions properly on LoopSimplify. llvm-svn: 223119
* Revert r223049, r223050 and r223051 while investigating test failures.Hans Wennborg2014-12-011-8/+56
| | | | | | I didn't foresee affecting the Clang test suite :/ llvm-svn: 223054
* SimplifyCFG: Omit range checks for switch lookup tables when default is ↵Hans Wennborg2014-12-011-7/+8
| | | | | | | | unreachable They would get optimized away later, but we might as well not emit them. llvm-svn: 223051
* SimplifyCFG: don't remove unreachable default switch destinationsHans Wennborg2014-12-011-49/+0
| | | | | | | | | | | | | An unreachable default destination can be exploited by other optimizations, and SDag lowering is now prepared to handle them efficiently. For example, branches to the unreachable destination will be optimized away, such as in the case of range checks for switch lookup tables. On 64-bit Linux, this reduces the size of a clang bootstrap by 80 kB (and Chromium by 30 kB). llvm-svn: 223050
* [msan] Add compile-time checks for missing origins.Evgeniy Stepanov2014-12-011-9/+10
| | | | | | | | | | | This change makes MemorySanitizer instrumentation a bit more strict about instructions that have no origin id assigned to them. This would have caught the bug that was fixed in r222918. No functional change. llvm-svn: 222997
* [asan] Change dynamic alloca instrumentation to only consider allocas that ↵Yury Gribov2014-12-011-3/+35
| | | | | | | | are dominating all exits from function. Reviewed in http://reviews.llvm.org/D6412 llvm-svn: 222991
* DebugIR: Delete -debug-irDuncan P. N. Exon Smith2014-11-293-716/+0
| | | | llvm-svn: 222945
* Revert "Masked Vector Load and Store Intrinsics."Duncan P. N. Exon Smith2014-11-281-83/+15
| | | | | | | | | | | This reverts commit r222632 (and follow-up r222636), which caused a host of LNT failures on an internal bot. I'll respond to the commit on the list with a reproduction of one of the failures. Conflicts: lib/Target/X86/X86TargetTransformInfo.cpp llvm-svn: 222936
* InstCombine: FoldOrOfICmps harderDavid Majnemer2014-11-281-2/+25
| | | | | | | | | | | | We may be in a situation where the icmps might not be near each other in a tree of or instructions. Try to dig out related compare instructions and see if they combine. N.B. This won't fire on deep trees of compares because rewritting the tree might end up creating a net increase of IR. We may have to resort to something more sophisticated if this is a real problem. llvm-svn: 222928
* [LICM] Store sink and indirectbr instructionsBruno Cardoso Lopes2014-11-281-0/+6
| | | | | | | | | | | | | | Loop simplify skips exit-block insertion when exits contain indirectbr instructions. This leads to an assertion in LICM when trying to sink stores out of non-dedicated loop exits containing indirectbr instructions. This patch fix this issue by re-checking for dedicated exits in LICM prior to store sink attempts. Differential Revision: http://reviews.llvm.org/D6414 rdar://problem/18943047 llvm-svn: 222927
* [SwitchLowering] Handle multiple destinations on condensed case stmtsBruno Cardoso Lopes2014-11-281-12/+29
| | | | | | | | | | | | | | Switch cases statements with sequential values that branch to the same destination BB may often be handled together in a single new source BB. In this scenario we need to remove remaining incoming values from PHI instructions in the destination BB, as to match the number of source branches. Differential Revision: http://reviews.llvm.org/D6415 rdar://problem/19040894 llvm-svn: 222926
* [msan] Fix origin propagation for select of floats.Evgeniy Stepanov2014-11-281-3/+4
| | | | | | | | | | MSan does not assign origin for instrumentation temps (i.e. the ones that do not come from the application code), but "select" instrumentation erroneously tried to use one of those. https://code.google.com/p/memory-sanitizer/issues/detail?id=78 llvm-svn: 222918
* Removed extra line from a comment to test first commit. NFC.Ankur Garg2014-11-281-1/+0
| | | | llvm-svn: 222916
* reinstate r222872: Peephole optimization in switch table lookup: reuse the ↵Erik Eckstein2014-11-271-7/+100
| | | | | | | | | | | | | | | | | | | | | | | | | | | guarding table comparison if possible. Fixed missing dominance check. Original commit message: This optimization tries to reuse the generated compare instruction, if there is a comparison against the default value after the switch. Example: if (idx < tablesize) r = table[idx]; // table does not contain default_value else r = default_value; if (r != default_value) ... Is optimized to: cond = idx < tablesize; if (cond) r = table[idx]; else r = default_value; if (cond) ... Jump threading will then eliminate the second if(cond). llvm-svn: 222891
* [msan] Remove indirect call wrapping code.Evgeniy Stepanov2014-11-271-89/+1
| | | | | | This functionality was only used in MSanDR, which is deprecated. llvm-svn: 222889
* Revert "Peephole optimization in switch table lookup: reuse the guarding ↵Erik Eckstein2014-11-271-89/+7
| | | | | | | | table comparison if possible." It is breaking the clang bootstrag. llvm-svn: 222877
* Peephole optimization in switch table lookup: reuse the guarding table ↵Erik Eckstein2014-11-271-7/+89
| | | | | | | | | | | | | | | | | | | | | | | | | comparison if possible. This optimization tries to reuse the generated compare instruction, if there is a comparison against the default value after the switch. Example: if (idx < tablesize) r = table[idx]; // table does not contain default_value else r = default_value; if (r != default_value) ... Is optimized to: cond = idx < tablesize; if (cond) r = table[idx]; else r = default_value; if (cond) ... \endcode Jump threading will then eliminate the second if(cond). llvm-svn: 222872
* InstCombine: Restore optimizations lost in r210006David Majnemer2014-11-271-0/+28
| | | | | | | | This restores our ability to optimize: (X & C) == 0 ? X ^ C : X into X | C (X & C) != 0 ? X ^ C : X into X & ~C llvm-svn: 222871
* Revert "Added inst combine transforms for single bit tests from Chris's note"David Majnemer2014-11-261-28/+1
| | | | | | | | | | | This reverts commit r210006, it miscompiled libapr which is used in who knows how many projects. A test has been added to ensure that we don't regress again. I'll work on a rewrite of what the optimization was trying to do later. llvm-svn: 222856
* [InstCombine] Change LLVM To canonicalize toward the value type beingChandler Carruth2014-11-251-100/+72
| | | | | | | | | | | | | | | | | | | | | | | | | | stored rather than the pointer type. This change is analogous to r220138 which changed the canonicalization for loads. The rationale is the same: memory does not have a type, operations (and thus the values they produce) have a type. We should match that type as closely as possible rather than reading some form of semantics into the pointer type. With this change, loads and stores should no longer be made with nonsensical types for the values that tehy load and store. This is particularly important when trying to match specific loaded and stored types in the process of doing other instcombines, which is what led me down this twisty maze of miscanonicalization. I've put quite some effort into looking through IR to find places where LLVM's optimizer was being unreasonably conservative in the face of mismatched load and store types, however it is possible (let's say, likely!) I have missed some. If you see regressions here, or from r220138, the likely cause is some part of LLVM failing to cope with load and store types differing. Test cases appreciated, it is important that we root all of these out of LLVM. llvm-svn: 222748
* Revert r220349 to re-instate r220277 with a fix for PR21330 -- quiteChandler Carruth2014-11-252-5/+6
| | | | | | | | | | | | | | | | | | | | | | | | | clearly only exactly equal width ptrtoint and inttoptr casts are no-op casts, it says so right there in the langref. Make the code agree. Original log from r220277: Teach the load analysis to allow finding available values which require inttoptr or ptrtoint cast provided there is datalayout available. Eventually, the datalayout can just be required but in practice it will always be there today. To go with the ability to expose available values requiring a ptrtoint or inttoptr cast, helpers are added to perform one of these three casts. These smarts are necessary to finish canonicalizing loads and stores to the operational type requirements without regressing fundamental combines. I've added some test cases. These should actually improve as the load combining and store combining improves, but they may fundamentally be highlighting some missing combines for select in addition to exercising the specific added logic to load analysis. llvm-svn: 222739
* Bug 21610: Canonicalize min/max fcmp selects to use ordered comparisonsMatt Arsenault2014-11-241-1/+30
| | | | llvm-svn: 222705
* [asan/coverage] change the way asan coverage instrumentation is done: ↵Kostya Serebryany2014-11-241-10/+7
| | | | | | instead of setting the guard to 1 in the generated code, pass the pointer to guard to __sanitizer_cov and set it there. No user-visible functionality change expected llvm-svn: 222675
* InstCombine: Don't create an unused instructionDavid Majnemer2014-11-241-2/+1
| | | | | | | | | | We would create an instruction but not inserting it. Not inserting the unused instruction would lead us to verification failure. This fixes PR21653. llvm-svn: 222659
OpenPOWER on IntegriCloud