summaryrefslogtreecommitdiffstats
path: root/llvm/lib/Transforms
Commit message (Collapse)AuthorAgeFilesLines
* HotColdSplitting: check that target supports cold calling conventionSebastian Pop2018-09-101-4/+13
| | | | | | | | | Before tagging a function with coldcc make sure the target supports cold calling convention. Without this patch HotColdSplitting pass fails on aarch64 with: fatal error: error in backend: Unsupported calling convention. llvm-svn: 341838
* add flag instead of using a constant [NFC]Sebastian Pop2018-09-101-1/+5
| | | | llvm-svn: 341837
* make flag name more specific to gvn [NFC]Sebastian Pop2018-09-101-2/+2
| | | | llvm-svn: 341836
* InstCombine: move hasOneUse check to the top of foldICmpAddConstantTim Northover2018-09-101-3/+3
| | | | | | | | | | | | There were two combines not covered by the check before now, neither of which actually differed from normal in the benefit analysis. The most recent seems to be because it was just added at the top of the function (naturally). The older is from way back in 2008 (r46687) when we just didn't put those checks in so routinely, and has been diligently maintained since. llvm-svn: 341831
* Don't create a temporary vector of loop blocks just to iterate over them.Benjamin Kramer2018-09-101-2/+1
| | | | | | Loop's getBlocks returns an ArrayRef. llvm-svn: 341821
* [GVN] Invalidate cached info for values replaced by equality propagationJohn Brawn2018-09-101-0/+6
| | | | | | | | | When GVN propagates an equality by replacing one value with another it also needs to invalidate the cached information for the value being replaced. Differential Revision: https://reviews.llvm.org/D51218 llvm-svn: 341820
* [IndVars] Set Changed if rewriteFirstIterationLoopExitValues changes IR. PR38863Max Kazantsev2018-09-101-3/+6
| | | | | | | | | | | Currently, `rewriteFirstIterationLoopExitValues` does not set Changed flag even if it makes changes in the IR. There is no clear evidence that it can cause a crash, but it looks highly suspicious and likely invalid. Differential Revision: https://reviews.llvm.org/D51779 Reviewed By: skatkov llvm-svn: 341779
* [IndVars] Set Changed if sinkUnusedInvariants changes IR. PR38863Max Kazantsev2018-09-101-5/+9
| | | | | | | | | | | Currently, `sinkUnusedInvariants` does not set Changed flag even if it makes changes in the IR. There is no clear evidence that it can cause a crash, but it looks highly suspicious and likely invalid. Differential Revision: https://reviews.llvm.org/D51777 Reviewed By: skatkov llvm-svn: 341777
* Move a transformation routine from LoopUtils to LoopVectorize.Vikram TV2018-09-102-72/+84
| | | | | | | | | | | | | | | | Summary: Move InductionDescriptor::transform() routine from LoopUtils to its only uses in LoopVectorize.cpp. Specifically, the function is renamed as InnerLoopVectorizer::emitTransformedIndex(). This is a child to D51153. Reviewers: dmgreen, llvm-commits Reviewed By: dmgreen Differential Revision: https://reviews.llvm.org/D51837 llvm-svn: 341776
* Move createMinMaxOp() out of RecurrenceDescriptor.Vikram TV2018-09-102-50/+49
| | | | | | | | | | Reviewers: dmgreen, llvm-commits Reviewed By: dmgreen Differential Revision: https://reviews.llvm.org/D51838 llvm-svn: 341773
* [SimplifyIndVar] Avoid generating truncate instructions with non-hoisted ↵Abderrazek Zaafrani2018-09-071-0/+152
| | | | | | | | Laod operand. Differential Revision: https://reviews.llvm.org/D49151 llvm-svn: 341726
* [MemorySSA] Update MemoryPhi wiring for block splitting to consider if ↵Alina Sbirlea2018-09-071-1/+2
| | | | | | | | | | | | | | | | | identical edges were merged. Summary: Block splitting is done with either identical edges being merged, or not. Only critical edges can be split without merging identical edges based on an option. Teach the memoryssa updater to take this into account: for the same edge between two blocks only move one entry from the Phi in Old to the new Phi in New. Reviewers: george.burgess.iv Subscribers: sanjoy, jlebar, Prazek, llvm-commits Differential Revision: https://reviews.llvm.org/D51563 llvm-svn: 341709
* [InstCombine] narrow vector select with padded condition and extracted ↵Sanjay Patel2018-09-071-0/+38
| | | | | | | | | | | | | | | | | | result (PR38691) shuf (sel (shuf NarrowCond, undef, WideMask), X, Y), undef, NarrowMask) --> sel NarrowCond, (shuf X, undef, NarrowMask), (shuf Y, undef, NarrowMask) The motivating case from: https://bugs.llvm.org/show_bug.cgi?id=38691 ...is the last regression test. In that case, we're just left with the narrow select. Note that if we do create new shuffles, they use the existing extraction identity mask, so there's no danger that this transform creates arbitrary shuffles. Differential Revision: https://reviews.llvm.org/D51496 llvm-svn: 341708
* [PGO] Fix some style issue of ControlHeightReductionFangrui Song2018-09-071-33/+30
| | | | | | | | | | | | Reviewers: yamauchi Reviewed By: yamauchi Subscribers: llvm-commits Differential Revision: https://reviews.llvm.org/D51811 llvm-svn: 341702
* [PGO][CHR] Build/warning fixHiroshi Yamauchi2018-09-071-0/+2
| | | | llvm-svn: 341692
* NFC: remove magic bool in LoopIdiomRecognizeJF Bastien2018-09-071-9/+9
| | | | | | Use an enum class instead. llvm-svn: 341684
* [PGO][CHR] Small cleanup.Hiroshi Yamauchi2018-09-071-36/+17
| | | | | | | | | | | | | | | | Summary: Do away with demangling. It wasn't really necessary. Declared some local functions to be static. Reviewers: davidxl Reviewed By: davidxl Subscribers: llvm-commits Differential Revision: https://reviews.llvm.org/D51740 llvm-svn: 341681
* [InstCombine] Fold (min/max ~X, Y) -> ~(max/min X, ~Y) when Y is freely ↵Craig Topper2018-09-072-13/+46
| | | | | | | | | | | | invertible If the ~X wasn't able to simplify above the max/min, we might be able to simplify it by moving it below the max/min. I had to modify the ~(min/max ~X, Y) transform to prevent getting stuck in a loop when we saw the new ~(max/min X, ~Y) before the ~Y had been folded away to remove the new not. Differential Revision: https://reviews.llvm.org/D51398 llvm-svn: 341674
* [LV] Fix code gen for conditionally executed loads and storesAnna Thomas2018-09-071-8/+41
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Fix a latent bug in loop vectorizer which generates incorrect code for memory accesses that are executed conditionally. As pointed in review, this bug definitely affects uniform loads and may affect conditional stores that should have turned into scatters as well). The code gen for conditionally executed uniform loads on architectures that support masked gather instructions is broken. Without this patch, we were unconditionally executing the *conditional* load in the vectorized version. This patch does the following: 1. Uniform conditional loads on architectures with gather support will have correct code generated. In particular, the cost model (setCostBasedWideningDecision) is fixed. 2. For the recipes which are handled after the widening decision is set, we use the isScalarWithPredication(I, VF) form which is added in the patch. 3. Fix the vectorization cost model for scalarization (getMemInstScalarizationCost): implement and use isPredicatedInst to identify *all* predicated instructions, not just scalar+predicated. So, now the cost for scalarization will be increased for maskedloads/stores and gather/scatter operations. In short, we should be choosing the gather/scatter in place of scalarization on archs where it is profitable. 4. We needed to weaken the assert in useEmulatedMaskMemRefHack. Reviewers: Ayal, hsaito, mkuper Differential Revision: https://reviews.llvm.org/D51313 llvm-svn: 341673
* Hot cold splitting passAditya Kumar2018-09-074-0/+379
| | | | | | | | | | | | Find cold blocks based on profile information (or optionally with static analysis). Forward propagate profile information to all cold-blocks. Outline a cold region. Set calling conv and prof hint for the callsite of the outlined function. Worked in collaboration with: Sebastian Pop <s.pop@samsung.com> Differential Revision: https://reviews.llvm.org/D50658 llvm-svn: 341669
* [InstCombine] Do not fold scalar ops over select with vector condition.Florian Hahn2018-09-071-0/+8
| | | | | | | | | | | | | If OtherOpT or OtherOpF have scalar types and the condition is a vector, we would create an invalid select. Reviewers: spatel, john.brawn, mssimpso, craig.topper Reviewed By: spatel Differential Revision: https://reviews.llvm.org/D51781 llvm-svn: 341666
* [NewGVN] Mark function as changed if we erase instructions.Florian Hahn2018-09-071-2/+4
| | | | | | | | | | | | | | | | | | | | | | Currently eliminateInstructions only returns true if any instruction got replaced. In the test case for this patch, we eliminate the trivially dead calls, for which eliminateInstructions not do a replacement and the function is not marked as changed, which is why the inliner crashes while traversing the call graph. Alternatively we could also change eliminateInstructions to return true in case we mark instructions for deletion, but that's slightly more code and doing it at the place where the replacement happens seems safer. Fixes PR37517. Reviewers: davide, mcrosier, efriedma, bjope Reviewed By: bjope Differential Revision: https://reviews.llvm.org/D51169 llvm-svn: 341651
* [MSan] don't access MsanCtorFunction when using KMSANAlexander Potapenko2018-09-071-1/+1
| | | | | | MSan has found a use of uninitialized memory in MSan, fix it. llvm-svn: 341646
* [MSan] Add KMSAN instrumentation to MSan passAlexander Potapenko2018-09-071-72/+310
| | | | | | | | | | | | | | | | | | | | | | | | Introduce the -msan-kernel flag, which enables the kernel instrumentation. The main differences between KMSAN and MSan instrumentations are: - KMSAN implies msan-track-origins=2, msan-keep-going=true; - there're no explicit accesses to shadow and origin memory. Shadow and origin values for a particular X-byte memory location are read and written via pointers returned by __msan_metadata_ptr_for_load_X(u8 *addr) and __msan_store_shadow_origin_X(u8 *addr, uptr shadow, uptr origin); - TLS variables are stored in a single struct in per-task storage. A call to a function returning that struct is inserted into every instrumented function before the entry block; - __msan_warning() takes a 32-bit origin parameter; - local variables are poisoned with __msan_poison_alloca() upon function entry and unpoisoned with __msan_unpoison_alloca() before leaving the function; - the pass doesn't declare any global variables or add global constructors to the translation unit. llvm-svn: 341637
* [IndVars] Set Changed when we delete dead instructions. PR38855Max Kazantsev2018-09-071-1/+1
| | | | | | | | | | | IndVars does not set `Changed` flag when it eliminates dead instructions. As result, it may make IR modifications and report that it has done nothing. It leads to inconsistent preserved analyzes results. Differential Revision: https://reviews.llvm.org/D51770 Reviewed By: skatkov llvm-svn: 341633
* [SampleFDO] Make sample profile loader unaware of compact format change.Wei Mi2018-09-061-13/+10
| | | | | | | | | | | | | | | | The patch tries to make sample profile loader independent of profile format change. It moves compact format related code into FunctionSamples and SampleProfileReader classes, and sample profile loader only has to interact with those two classes and will be unaware of profile format changes. The cleanup also contain some fixes to further remove the difference between compactbinary format and binary format. After the cleanup using different formats originated from the same profile will generate the same binaries, which we verified by compiling two large server benchmarks w/wo thinlto. Differential Revision: https://reviews.llvm.org/D51643 llvm-svn: 341591
* [InstCombine] add xor+not foldsSanjay Patel2018-09-061-0/+16
| | | | | | | | | | This fold is needed to avoid a regression when we try to recommit rL300977. We can't see the most basic win currently because demanded bits changes the patterns: https://rise4fun.com/Alive/plpp llvm-svn: 341559
* [MSan] store origins for variadic function parameters in ↵Alexander Potapenko2018-09-061-4/+61
| | | | | | | | | | | | | | | | | __msan_va_arg_origin_tls Add the __msan_va_arg_origin_tls TLS array to keep the origins for variadic function parameters. Change the instrumentation pass to store parameter origins in this array. This is a reland of r341528. test/msan/vararg.cc doesn't work on Mips, PPC and AArch64 (because this patch doesn't touch them), XFAIL these arches. Also turned out Clang crashed on i80 vararg arguments because of incorrect origin type returned by getOriginPtrForVAArgument() - fixed it and added a test. llvm-svn: 341554
* [InstCombine] fix formatting in SimplifyDemandedVectorElts->Select; NFCISanjay Patel2018-09-061-12/+16
| | | | | | | | I'm preparing to add the same functionality both here and to the DAG version of this code in D51696 / D51433, so try to make those cases as similar as possible to avoid bugs. llvm-svn: 341545
* [MSan] revert r341528 to unbreak the botsAlexander Potapenko2018-09-061-61/+4
| | | | llvm-svn: 341541
* [LoopInterchange] Cleanup unused variables.Florian Hahn2018-09-061-8/+2
| | | | llvm-svn: 341537
* [LoopInterchange] Move preheader creation to transform stage and simplify.Florian Hahn2018-09-061-25/+18
| | | | | | | | | | | | | | | | | | | There is no need to create preheaders in the analysis stage, we only need them when adjusting the branches. Also, the only cases we need to create our own preheaders is when they have more than 1 predecessors or PHI nodes (even with only 1 predecessor, we could have an LCSSA phi node). I have simplified the conditions and added some assertions to be sure. Because we know the inner and outer loop need to be tightly nested, it is sufficient to check if the inner loop preheader is the outer loop header to check if we need to create a new preheader. Reviewers: efriedma, mcrosier, karthikthecool Reviewed By: efriedma Differential Revision: https://reviews.llvm.org/D51703 llvm-svn: 341533
* [MSan] store origins for variadic function parameters in ↵Alexander Potapenko2018-09-061-4/+61
| | | | | | | | | | __msan_va_arg_origin_tls Add the __msan_va_arg_origin_tls TLS array to keep the origins for variadic function parameters. Change the instrumentation pass to store parameter origins in this array. llvm-svn: 341528
* [MSan] Make sure variadic function arguments do not overflow __msan_va_arg_tlsAlexander Potapenko2018-09-061-23/+50
| | | | | | | | | | | | Turns out that calling a variadic function with too many (e.g. >100 i64's) arguments overflows __msan_va_arg_tls, which leads to smashing other TLS data with function argument shadow values. getShadow() already checks for kParamTLSSize and returns clean shadow if the argument does not fit, so just skip storing argument shadow for such arguments. llvm-svn: 341525
* Revert "[IndVars] Turn isValidRewrite into an assertion" because it seems wrongMax Kazantsev2018-09-061-8/+9
| | | | llvm-svn: 341517
* [IndVars] Turn isValidRewrite into an assertionMax Kazantsev2018-09-061-9/+8
| | | | | | | | | | | | | | | Function rewriteLoopExitValues contains a check on isValidRewrite which is needed to make sure that SCEV does not convert the pattern `gep Base, (&p[n] - &p[0])` into `gep &p[n], Base - &p[0]`. This problem has been fixed in SCEV long ago, so this check is just obsolete. This patch converts it into an assertion to make sure that the SCEV will not mess up this case in the future. Differential Revision: https://reviews.llvm.org/D51582 Reviewed By: atrick llvm-svn: 341516
* [ControlHeightReduction] Remove unused includesBenjamin Kramer2018-09-051-5/+5
| | | | | | Also clang-format them. llvm-svn: 341468
* [Aggressive InstCombine] Move C bindings to their own header file.Benjamin Kramer2018-09-051-1/+1
| | | | llvm-svn: 341461
* Prevent unsigned overflow.Richard Trieu2018-09-051-7/+9
| | | | | | | | | | The sum of the weights is caculated in an APInt, which has a width smaller than 64. In certain cases, the sum of the widths would overflow when calculations are done inside an APInt, but would not if done with uint64_t. Since the values will be passed as uint64_t in the function call anyways, do all the math in 64 bits. Also added an assert in case the probabilities overflow 64 bits. llvm-svn: 341444
* Fix -Wunused-function in release build after rL341386Fangrui Song2018-09-051-7/+10
| | | | llvm-svn: 341443
* [InstCombine] fix xor-or-xor fold to check uses and handle commutesSanjay Patel2018-09-041-32/+21
| | | | | | | | | | | | I'm probably missing some way to use m_Deferred to remove the code duplication, but that can be a follow-up. The improvement in demand_shrink_nsw.ll is an example of missing the fold because the pattern matching was deficient. I didn't try to follow the bits in that test, but Alive says it's correct: https://rise4fun.com/Alive/ugc llvm-svn: 341426
* Revert "Revert r341269: [Constant Hoisting] Hoisting Constant GEP Expressions"Zhaoshi Zheng2018-09-041-37/+137
| | | | | | | | | | | | | | | | | | | | | | | | | | | | Reland r341269. Use std::stable_sort when sorting constant condidates. Reverting commit, r341365: Revert r341269: [Constant Hoisting] Hoisting Constant GEP Expressions One of the tests is failing 50% of the time when expensive checks are enabled. Not sure how deep the problem is so just reverting while the author can investigate so that the bots stop repeatedly failing and blaming things incorrectly. Will respond with details on the original commit. Original commit, r341269: [Constant Hoisting] Hoisting Constant GEP Expressions Leverage existing logic in constant hoisting pass to transform constant GEP expressions sharing the same base global variable. Multi-dimensional GEPs are rewritten into single-dimensional GEPs. https://reviews.llvm.org/D51396 Differential Revision: https://reviews.llvm.org/D51654 llvm-svn: 341417
* [LV] First order recurrence phis should not be treated as uniformAnna Thomas2018-09-041-0/+5
| | | | | | | | | | | | | | This is fix for PR38786. First order recurrence phis were incorrectly treated as uniform, which caused them to be vectorized as uniform instructions. Patch by Ayal Zaks and Orivej Desh! Reviewed by: Anna Differential Revision: https://reviews.llvm.org/D51639 llvm-svn: 341416
* Fix a memory leak after rL341386.Hiroshi Yamauchi2018-09-041-1/+1
| | | | | | | | | | | | Reviewers: davidxl Reviewed By: davidxl Subscribers: llvm-commits Differential Revision: https://reviews.llvm.org/D51658 llvm-svn: 341412
* [InstCombine] make ((X & C) ^ C) form consistent for vectorsSanjay Patel2018-09-041-4/+2
| | | | | | It would be better to create a 'not' here, but that's not possible yet. llvm-svn: 341410
* [InstCombine] simplify code for xor folds; NFCISanjay Patel2018-09-041-40/+23
| | | | | | | | | This is just a cleanup step. The TODO comments show what is wrong with the 'and' version of the fold. Fixing this should be part of recommitting: rL300977 llvm-svn: 341405
* Fix unused variable warningReid Kleckner2018-09-041-2/+2
| | | | llvm-svn: 341400
* [SimpleLoopUnswitch] remove a chain of dead blocks at onceFedor Sergeev2018-09-041-19/+19
| | | | | | | | | | | | | | | | | | | | | | Recent change to deleteDeadBlocksFromLoop was not enough to fix all the problems related to dead blocks after nontrivial unswitching of switches. We need to delete all the dead blocks that were created during unswitching, otherwise we will keep having problems with phi's or dead blocks. This change removes all the dead blocks that are reachable from the loop, not trying to track whether these blocks are newly created by unswitching or not. While not completely correct, we are unlikely to get loose but reachable dead blocks that do not belong to our loop nest. It does fix all the failures currently known, in particular PR38778. Reviewed By: asbirlea Differential Revision: https://reviews.llvm.org/D51519 llvm-svn: 341398
* Fix build failures after rL341386.Hiroshi Yamauchi2018-09-041-0/+8
| | | | | | | | | | | | Reviewers: davidxl Reviewed By: davidxl Subscribers: llvm-commits Differential Revision: https://reviews.llvm.org/D51647 llvm-svn: 341391
* [PGO] Control Height ReductionHiroshi Yamauchi2018-09-044-0/+2020
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Summary: Control height reduction merges conditional blocks of code and reduces the number of conditional branches in the hot path based on profiles. if (hot_cond1) { // Likely true. do_stg_hot1(); } if (hot_cond2) { // Likely true. do_stg_hot2(); } -> if (hot_cond1 && hot_cond2) { // Hot path. do_stg_hot1(); do_stg_hot2(); } else { // Cold path. if (hot_cond1) { do_stg_hot1(); } if (hot_cond2) { do_stg_hot2(); } } This speeds up some internal benchmarks up to ~30%. Reviewers: davidxl Reviewed By: davidxl Subscribers: xbolva00, dmgreen, mehdi_amini, llvm-commits, mgorny Differential Revision: https://reviews.llvm.org/D50591 llvm-svn: 341386
OpenPOWER on IntegriCloud