summaryrefslogtreecommitdiffstats
path: root/llvm/test/Transforms/HotColdSplit
Commit message (Collapse)AuthorAgeFilesLines
* HotColdSplitting: Do not outline within noreturn functionsVedant Kumar2019-12-191-0/+20
| | | | | | | A function marked `noreturn` may contain unreachable terminators: these should not be considered cold, as the function may be a trampoline. rdar://58068594
* Fix failing testcase to check for the correct outputBill Wendling2019-12-021-7/+2
|
* Place the "cold" code piece into the same section as the original functionBill Wendling2019-12-021-0/+33
| | | | | | | | | | | | | | | | Summary: This cropped up in the Linux kernel where cold code was placed in an incompatible section. Reviewers: compnerd, vsk, tejohnson Reviewed By: vsk Subscribers: hiraditya, llvm-commits Tags: #llvm Differential Revision: https://reviews.llvm.org/D70925
* Invalidate assumption cache before outlining.Aditya Kumar2019-10-041-0/+12
| | | | | | | | | | | | | | Subscribers: llvm-commits Tags: #llvm Reviewers: compnerd, vsk, sebpop, fhahn, tejohnson Reviewed by: vsk Differential Revision: https://reviews.llvm.org/D68478 llvm-svn: 373807
* HotColdSplitting: invalidate the AssumptionCache on splitSaleem Abdulrasool2019-09-231-0/+38
| | | | | | | | | | | When a cold path is outlined, the value tracking in the assumption cache may be invalidated due to the code motion. We would previously trip an assertion in subsequent passes (but required the passes to happen in a single run as the assumption cache is shared across the passes). Invalidating the cache ensures that we get the correct information when needed with the legacy pass manager as well. llvm-svn: 372667
* [CodeExtractor] Fix sinking of allocas with multiple bitcast uses (PR42451)Vedant Kumar2019-07-101-0/+74
| | | | | | | | | | | | | | An alloca which can be sunk into the extraction region may have more than one bitcast use. Move these uses along with the alloca to prevent use-before-def. Testing: check-llvm, stage2 build of clang Fixes llvm.org/PR42451. Differential Revision: https://reviews.llvm.org/D64463 llvm-svn: 365660
* [lit] Delete empty lines at the end of lit.local.cfg NFCFangrui Song2019-06-171-1/+0
| | | | llvm-svn: 363538
* Revert "Temporarily Revert "Add basic loop fusion pass.""Eric Christopher2019-04-1733-0/+1886
| | | | | | | | The reversion apparently deleted the test/Transforms directory. Will be re-reverting again. llvm-svn: 358552
* Temporarily Revert "Add basic loop fusion pass."Eric Christopher2019-04-1733-1886/+0
| | | | | | | | As it's causing some bot failures (and per request from kbarton). This reverts commit r358543/ab70da07286e618016e78247e4a24fcb84077fda. llvm-svn: 358546
* [HotColdSplit] Disable splitting for sanitized functionsVedant Kumar2019-02-261-0/+72
| | | | | | | | | | Splitting can make sanitizer errors harder to understand, as the trapping instruction may not be in the function where the bug was detected. rdar://48142697 llvm-svn: 354931
* [CodeExtractor] Do not lift lifetime.end markers for region inputsVedant Kumar2019-02-152-8/+53
| | | | | | | | | | | | | | | | | | | | | | | If a lifetime.end marker occurs along one path through the extraction region, but not another, then it's still incorrect to lift the marker, because there is some path through the extracted function which would ordinarily not reach the marker. If the call to the extracted function is in a loop, unrolling can cause inputs to the function to become optimized out as undef after the first iteration. To prevent incorrect stack slot merging in the calling function, it should be sufficient to lift lifetime.start markers for region inputs. I've tested this theory out by doing a stage2 check-all with randomized splitting enabled. This is a follow-up to r353973, and there's additional context for this change in https://reviews.llvm.org/D57834. rdar://47896986 Differential Revision: https://reviews.llvm.org/D58253 llvm-svn: 354159
* [CodeExtractor] Only lift lifetime markers present in the extraction regionVedant Kumar2019-02-132-0/+135
| | | | | | | | | | | | | | | When CodeExtractor finds liftime markers referencing inputs to the extraction region, it lifts these markers out of the region and inserts them around the call to the extracted function (see r350420, PR39671). However, it should *only* lift lifetime markers that are actually present in the extraction region. I.e., if a start marker is present in the extraction region but a corresponding end marker isn't (or vice versa), only the start marker (or end marker, resp.) should be lifted. Differential Revision: https://reviews.llvm.org/D57834 llvm-svn: 353973
* [HotColdSplit] With PGO add profile entry metadata to split cold functionTeresa Johnson2019-02-071-0/+43
| | | | | | | | | | | | | | | | | | Summary: When compiling with profile data, ensure the split cold function gets cold function_entry_count metadata (just use 0 since it should be cold). Otherwise with function sections it will not be placed in the unlikely text section with other cold code. Reviewers: vsk Subscribers: sebpop, hiraditya, davidxl, llvm-commits Tags: #llvm Differential Revision: https://reviews.llvm.org/D57900 llvm-svn: 353434
* [HotColdSplit] Do not split out `resume` instructionsVedant Kumar2019-02-052-6/+18
| | | | | | | | | Resumes that are not reachable from a cleanup landing pad are considered to be unreachable. It’s not safe to split them out. rdar://47808235 llvm-svn: 353242
* [CodeExtractor] Add support for the `swifterror` attributeVedant Kumar2019-01-281-0/+43
| | | | | | | When passing a `swifterror` argument or alloca as an input to an extraction region, mark the input parameter `swifterror`. llvm-svn: 352408
* [HotColdSplit] Introduce a cost model to control splitting behaviorVedant Kumar2019-01-2510-93/+125
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The main goal of the model is to avoid *increasing* function size, as that would eradicate any memory locality benefits from splitting. This happens when: - There are too many inputs or outputs to the cold region. Argument materialization and reloads of outputs have a cost. - The cold region has too many distinct exit blocks, causing a large switch to be formed in the caller. - The code size cost of the split code is less than the cost of a set-up call. A secondary goal is to prevent excessive overall binary size growth. With the cost model in place, I experimented to find a splitting threshold that works well in practice. To make warm & cold code easily separable for analysis purposes, I moved split functions to a "cold" section. I experimented with thresholds between [0, 4] and set the default to the threshold which minimized geomean __text size. Experiment data from building LNT+externals for X86 (N = 639 programs, all sizes in bytes): | Configuration | __text geom size | __cold geom size | TEXT geom size | | **-Os** | 1736.3 | 0, n=0 | 10961.6 | | -Os, thresh=0 | 1740.53 | 124.482, n=134 | 11014 | | -Os, thresh=1 | 1734.79 | 57.8781, n=90 | 10978.6 | | -Os, thresh=2 | ** 1733.85 ** | 65.6604, n=61 | 10977.6 | | -Os, thresh=3 | 1733.85 | 65.3071, n=61 | 10977.6 | | -Os, thresh=4 | 1735.08 | 67.5156, n=54 | 10965.7 | | **-Oz** | 1554.4 | 0, n=0 | 10153 | | -Oz, thresh=2 | ** 1552.2 ** | 65.633, n=61 | 10176 | | **-O3** | 2563.37 | 0, n=0 | 13105.4 | | -O3, thresh=2 | ** 2559.49 ** | 71.1072, n=61 | 13162.4 | Picking thresh=2 reduces the geomean __text section size by 0.14% at -Os, -Oz, and -O3 and causes ~0.2% growth in the TEXT segment. Note that TEXT size is page-aligned, whereas section sizes are byte-aligned. Experiment data from building LNT+externals for ARM64 (N = 558 programs, all sizes in bytes): | Configuration | __text geom size | __cold geom size | TEXT geom size | | **-Os** | 1763.96 | 0, n=0 | 42934.9 | | -Os, thresh=2 | ** 1760.9 ** | 76.6755, n=61 | 42934.9 | Picking thresh=2 reduces the geomean __text section size by 0.17% at -Os and causes no growth in the TEXT segment. Measurements were done with D57082 (r352080) applied. Differential Revision: https://reviews.llvm.org/D57125 llvm-svn: 352228
* [HotColdSplit] Split more aggressively before/after cold invokesVedant Kumar2019-01-251-0/+31
| | | | | | | | | | | While a cold invoke itself and its unwind destination can't be extracted, code which unconditionally executes before/after the invoke may still be profitable to extract. With cost model changes from D57125 applied, this gives a 3.5% increase in split text across LNT+externals on arm64 at -Os. llvm-svn: 352160
* [HotColdSplit] Mark inherently cold functions as suchVedant Kumar2019-01-193-2/+68
| | | | | | | | | | If an inherently cold function is found, mark it as cold. For now this means applying the `cold` and `minsize` attributes. As a drive-by, revisit and clean up the criteria for considering a function for splitting. Add tests. llvm-svn: 351623
* [CodeExtractor] Emit lifetime markers around reloads of outputsVedant Kumar2019-01-193-8/+11
| | | | | | | | | | | | | | | | CodeExtractor permits extracting a region of blocks from a function even when values defined within the region are used outside of it. This is typically done by creating an alloca in the original function and reloading the alloca after a call to the extracted function. Wrap the reload in lifetime start/end markers to promote stack coloring. Suggested by Sergei Kachkov! Differential Revision: https://reviews.llvm.org/D56045 llvm-svn: 351621
* [HotColdSplit] Allow outlining with live outputsVedant Kumar2019-01-171-0/+68
| | | | | | | | | | | | | Prior to r348205, extracting code regions with live output values was disabled because of a miscompilation (PR39433). Lift the restriction as PR39433 has been addressed. Tested on LNT+externals, on a run of check-llvm in a stage2 build, and with a full build of iOS (with hot/cold splitting enabled). As a drive-by, remove an errant TODO. llvm-svn: 351492
* [HotColdSplit] Consider resume instructions to be coldVedant Kumar2019-01-171-0/+20
| | | | | | | | | | Resuming exception unwinding is roughly as unlikely as throwing an exception. Tested on LNT+externals (in particular, the C++ EH regression tests provide end-to-end test coverage), as well as with a full build of iOS. llvm-svn: 351491
* [HotColdSplit] Relax requirement that the cold sink block be extractableVedant Kumar2019-01-171-0/+3
| | | | | | | | | Relaxing this requirement creates opportunities to split code dominated by an EH pad. Tested on LNT+externals. llvm-svn: 351483
* [HotColdSplit] Simplify tests by lowering their splitting thresholdsVedant Kumar2019-01-1724-59/+30
| | | | | | | | This gets rid of the brittle/mysterious calls to @sink()/@sideeffect() peppered throughout the test cases. They are no longer needed to force splitting to occur. llvm-svn: 351480
* [CodeExtractor] Do not extract unsafe lifetime markersVedant Kumar2019-01-041-0/+66
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Lifetime markers which reference inputs to the extraction region are not safe to extract. Example ('rhs' will be extracted): ``` entry: +------------+ | x = alloca | | y = alloca | +------------+ / \ lhs: rhs: +-------------------+ +-------------------+ | lifetime_start(x) | | lifetime_start(x) | | use(x) | | lifetime_start(y) | | lifetime_end(x) | | use(x, y) | | lifetime_start(y) | | lifetime_end(y) | | use(y) | | lifetime_end(x) | | lifetime_end(y) | +-------------------+ +-------------------+ ``` Prior to extraction, the stack coloring pass sees that the slots for 'x' and 'y' are in-use at the same time. After extraction, the coloring pass infers that 'x' and 'y' are *not* in-use concurrently, because markers from 'rhs' are no longer available to help decide otherwise. This leads to a miscompile, because the stack slots actually are in-use concurrently in the extracted function. Fix this by moving lifetime start/end markers for memory regions defined in the calling function around the call to the extracted function. Fixes llvm.org/PR39671 (rdar://45939472). Differential Revision: https://reviews.llvm.org/D55967 llvm-svn: 350420
* [HotColdSplitting] Disable outlining landingpad instructions (PR39917)Vedant Kumar2018-12-111-5/+4
| | | | | | | | | | It's currently not safe to outline landingpad instructions (see llvm.org/PR39917). Like @llvm.eh.typeid.for, the order and content of previous landingpad instructions in a function alters the lowering of subsequent landingpads by renumbering type info ID's. Outlining a landingpad therefore breaks exception handling & unwinding. llvm-svn: 348870
* [HotColdSplitting] Refine definition of unlikelyExecutedVedant Kumar2018-12-073-6/+132
| | | | | | | | | | | | | | | | | | | | | | | The splitting pass uses its 'unlikelyExecuted' predicate to statically decide which blocks are cold. - Do not treat noreturn calls as if they are cold unless they are actually marked cold. This is motivated by functions like exit() and longjmp(), which are not beneficial to outline. - Do not treat inline asm as an outlining barrier. In practice asm("") is frequently used to inhibit basic block merging; enabling outlining in this case results in substantial memory savings. - Treat invokes of cold functions as cold. As a drive-by, remove the 'exceptionHandlingFunctions' predicate, because it's no longer needed. The pass can identify & outline blocks dominated by EH pads, so there's no need to special-case __cxa_begin_catch etc. Differential Revision: https://reviews.llvm.org/D54244 llvm-svn: 348640
* [HotColdSplitting] Outline more than once per functionVedant Kumar2018-12-0710-0/+538
| | | | | | | | | | | | | | | | | | Algorithm: Identify maximal cold regions and put them in a worklist. If a candidate region overlaps with another, discard it. While the worklist is full, remove a single-entry sub-region from the worklist and attempt to outline it. By the non-overlap property, this should not invalidate parts of the domtree pertaining to other outlining regions. Testing: LNT results on X86 are clean. With test-suite + externals, llvm outlines 134KB pre-patch, and 352KB post-patch (+ ~2.6x). The file 483.xalancbmk/src/Constants.cpp stands out as an extreme case where llvm outlines over 100 times in some functions (mostly EH paths). There was not a significant performance impact pre vs. post-patch. Differential Revision: https://reviews.llvm.org/D53887 llvm-svn: 348639
* [CodeExtractor] Do not marked outlined calls which may resume EH as noreturnVedant Kumar2018-12-051-0/+37
| | | | | | | | | | Treat terminators which resume exception propagation as returning instructions (at least, for the purposes of marking outlined functions `noreturn`). This is to avoid inserting traps after calls to outlined functions which unwind. rdar://46129950 llvm-svn: 348404
* [CodeExtractor] Split PHI nodes with incoming values from outlined region ↵Vedant Kumar2018-12-031-2/+2
| | | | | | | | | | | | | | | | | (PR39433) If a PHI node out of extracted region has multiple incoming values from it, split this PHI on two parts. First PHI has incomings only from region and extracts with it (they are placed to the separate basic block that added to the list of outlined), and incoming values in original PHI are replaced by first PHI. Similar solution is already used in CodeExtractor for PHIs in entry block (severSplitPHINodes method). It covers PR39433 bug. Patch by Sergei Kachkov! Differential Revision: https://reviews.llvm.org/D55018 llvm-svn: 348205
* [CodeExtractor] Mark functions noreturn when applicableVedant Kumar2018-11-081-0/+3
| | | | | | | | | | This eliminates the outlining penalty for llvm.trap/unreachable, because callers no longer have to emit cleanup/ret instructions after calling an outlined `noreturn` function. rdar://45523626 llvm-svn: 346421
* [CodeExtractor] Do not extract calls to eh_typeid_for (PR39545)Vedant Kumar2018-11-061-0/+26
| | | | | | | | | | | The lowering for a call to eh_typeid_for changes when it's moved from one function to another. There are several proposals for fixing this issue in llvm.org/PR39545. Until some solution is in place, do not allow CodeExtractor to extract calls to eh_typeid_for, as that results in serious miscompilations. llvm-svn: 346256
* [CodeExtractor] Erase use-without-def debug intrinsics in parent funcVedant Kumar2018-11-061-0/+53
| | | | | | | | | | | When CodeExtractor moves instructions to a new function, debug intrinsics referring to those instructions within the parent function become invalid. This results in the same verifier failure which motivated r344545, about function-local metadata being used in the wrong function. llvm-svn: 346255
* [HotColdSplitting] Use TTI to inform outlining thresholdVedant Kumar2018-11-045-1/+37
| | | | | | | | | | | | | | | Using TargetTransformInfo allows the splitting pass to factor in the code size cost of instructions as it decides whether or not outlining is profitable. This did not regress the overall amount of outlining seen on the handful of internal frameworks I tested. Thanks to Jun Bum Lim for suggesting this! Differential Revision: https://reviews.llvm.org/D53835 llvm-svn: 346108
* [HotColdSplitting] Allow outlining single-block cold regionsVedant Kumar2018-10-293-51/+70
| | | | | | | | | | | | | | | | It can be profitable to outline single-block cold regions because they may be large. Allow outlining single-block regions if they have over some threshold of non-debug, non-terminator instructions. I chose 3 as the threshold after experimenting with several internal frameworks. In practice, reducing the threshold further did not give much improvement, whereas increasing it resulted in substantial regressions. Differential Revision: https://reviews.llvm.org/D53824 llvm-svn: 345524
* [HotColdSplitting] Identify larger cold regions using domtree queriesVedant Kumar2018-10-245-4/+275
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The current splitting algorithm works in three stages: 1) Identify cold blocks, then 2) Use forward/backward propagation to mark hot blocks, then 3) Grow a SESE region of blocks *outside* of the set of hot blocks and start outlining. While testing this pass on Apple internal frameworks I noticed that some kinds of control flow (e.g. loops) are never outlined, even though they unconditionally lead to / follow cold blocks. I noticed two other issues related to how cold regions are identified: - An inconsistency can arise in the internal state of the hotness propagation stage, as a block may end up in both the ColdBlocks set and the HotBlocks set. Further inconsistencies can arise as these sets do not match what's in ProfileSummaryInfo. - It isn't necessary to limit outlining to single-exit regions. This patch teaches the splitting algorithm to identify maximal cold regions and outline them. A maximal cold region is defined as the set of blocks post-dominated by a cold sink block, or dominated by that sink block. This approach can successfully outline loops in the cold path. As a side benefit, it maintains less internal state than the current approach. Due to a limitation in CodeExtractor, blocks within the maximal cold region which aren't dominated by a single entry point (a so-called "max ancestor") are filtered out. Results: - X86 (LNT + -Os + externals): 134KB of TEXT were outlined compared to 47KB pre-patch, or a ~3x improvement. Did not see a performance impact across two runs. - AArch64 (LNT + -Os + externals + Apple-internal benchmarks): 149KB of TEXT were outlined. Ditto re: performance impact. - Outlining results improve marginally in the internal frameworks I tested. Follow-ups: - Outline more than once per function, outline large single basic blocks, & try to remove unconditional branches in outlined functions. Differential Revision: https://reviews.llvm.org/D53627 llvm-svn: 345209
* [hot-cold-split] Name split functions with ".cold" suffixTeresa Johnson2018-10-243-5/+5
| | | | | | | | | | | | | | | | | | | | | | | Summary: The current default of appending "_"+entry block label to the new extracted cold function breaks demangling. Change the deliminator from "_" to "." to enable demangling. Because the header block label will be empty for release compile code, use "extracted" after the "." when the label is empty. Additionally, add a mechanism for the client to pass in an alternate suffix applied after the ".", and have the hot cold split pass use "cold."+Count, where the Count is currently 1 but can be used to uniquely number multiple cold functions split out from the same function with D53588. Reviewers: sebpop, hiraditya Subscribers: llvm-commits, erik.pilkington Differential Revision: https://reviews.llvm.org/D53534 llvm-svn: 345178
* [HotColdSplitting] Attach MinSize to outlined codeVedant Kumar2018-10-231-0/+32
| | | | | | | | | | | | | | | | | | Outlined code is cold by assumption, so it makes sense to optimize it for minimal code size rather than performance. After r344869 moved the splitting pass to the end of the IR pipeline, this does not result in much of a code size reduction. This is probably because a comparatively small number backend transforms make use of the MinSize hint. Running LNT on x86_64, I see that 33/1020 binaries shrink for a total of 919 bytes of TEXT reduction. I didn't measure a significant performance impact. Differential Revision: https://reviews.llvm.org/D53518 llvm-svn: 345072
* [hot-cold-split] Add opt remark on successTeresa Johnson2018-10-221-2/+3
| | | | | | | | | | | | Summary: Emit optimization remark on successful hot cold split. Reviewers: sebpop, hiraditya Subscribers: llvm-commits Differential Revision: https://reviews.llvm.org/D53512 llvm-svn: 344938
* [hot-cold-split] Add missing FileCheck invocationsTeresa Johnson2018-10-221-2/+2
| | | | | | | | | | | | | | Summary: r344558 added some CHECK statements to split-cold-2.ll, but didn't add any invocations of FileCheck. Add those here. Reviewers: sebpop Subscribers: llvm-commits Differential Revision: https://reviews.llvm.org/D53505 llvm-svn: 344928
* [hot-cold-split] fix failing testcasesSebastian Pop2018-10-162-4/+12
| | | | llvm-svn: 344577
* [hot-cold-split] fix static analysis of cold regionsSebastian Pop2018-10-152-3/+25
| | | | | | | | | | | | | | | | | | | | | | Make the code of blockEndsInUnreachable to match the function blockEndsInUnreachable in CodeGen/BranchFolding.cpp. I also have added a note to make sure the code of this function will not be modified unless the back-end version is also modified. An early return before outlining has been added to avoid outlining the full function body when the first block in the function is marked cold. The static analysis of cold code has been amended to avoid marking the whole function as cold by back-propagation because the back-propagation would mark blocks with return statements as cold. The patch adds debug statements to help discover these problems. Differential Revision: https://reviews.llvm.org/D52904 llvm-svn: 344558
* [CodeExtractor] Erase debug intrinsics in outlined thunks (fix PR22900)Vedant Kumar2018-10-151-0/+51
| | | | | | | | | | | | | | | Variable updates within the outlined function are invisible to debuggers. This could be improved by defining a DISubprogram for the new function. For the moment, simply erase the debug intrinsics instead. This fixes verifier failures about function-local metadata being used in the wrong function, seen while testing the hot/cold splitting pass. rdar://45142482 Differential Revision: https://reviews.llvm.org/D53267 llvm-svn: 344545
* Add support for new pass managerAditya Kumar2018-10-032-0/+2
| | | | | | | | | | | Modified the testcases to use both pass managers Use single commandline flag for both pass managers. Differential Revision: https://reviews.llvm.org/D52708 Reviewers: sebpop, tejohnson, brzycki, SirishP Reviewed By: tejohnson, brzycki llvm-svn: 343662
* HotColdSplit: fix invalid SSA due to outliningSebastian Pop2018-09-142-9/+29
| | | | | | | | The test used to fail with an invalid phi node: the two predecessors were outlined and the SSA representation was left invalid. The patch adds the exit block to the cold region. llvm-svn: 342277
* HotColdSplitting: fix test failing because of last commitSebastian Pop2018-09-101-3/+1
| | | | llvm-svn: 341839
* Hot cold splitting passAditya Kumar2018-09-071-0/+34
Find cold blocks based on profile information (or optionally with static analysis). Forward propagate profile information to all cold-blocks. Outline a cold region. Set calling conv and prof hint for the callsite of the outlined function. Worked in collaboration with: Sebastian Pop <s.pop@samsung.com> Differential Revision: https://reviews.llvm.org/D50658 llvm-svn: 341669
OpenPOWER on IntegriCloud