summaryrefslogtreecommitdiffstats
path: root/llvm/test/Transforms/PGOProfile
Commit message (Collapse)AuthorAgeFilesLines
...
* Simplify irreducible loop metadata test code.Hiroshi Yamauchi2017-11-141-57/+10
| | | | | | | | | | | | | | | | Summary: Shorten the irreducible loop metadata test code by removing insignificant instructions. Reviewers: davidxl Reviewed By: davidxl Subscribers: llvm-commits Differential Revision: https://reviews.llvm.org/D40043 llvm-svn: 318182
* [LTO][ThinLTO] Use the linker resolutions to mark global values as dso_local.Sean Fertile2017-11-041-1/+1
| | | | | | | | | | | | | Now that we have a way to mark GlobalValues as local we can use the symbol resolutions that the linker plugin provides as part of lto/thinlto link step to refine the compilers view on what symbols will end up being local. Originally commited as r317374, but reverted in r317395 to update some missed tests. Differential Revision: https://reviews.llvm.org/D35702 llvm-svn: 317408
* Revert "[LTO][ThinLTO] Use the linker resolutions to mark global values ..."Sean Fertile2017-11-041-1/+1
| | | | | | | | | Changes more tests then expected on one of the build bots. reverting to investigate. This reverts https://llvm.org/svn/llvm-project/llvm/trunk@317374 llvm-svn: 317395
* [LTO][ThinLTO] Use the linker resolutions to mark global values as dso_local.Sean Fertile2017-11-031-1/+1
| | | | | | | | | | Now that we have a way to mark GlobalValues as local we can use the symbol resolutions that the linker plugin provides as part of lto/thinlto link step to refine the compilers view on what symbols will end up being local. Differential Revision: https://reviews.llvm.org/D35702 llvm-svn: 317374
* Irreducible loop metadata for more accurate block frequency under PGO.Hiroshi Yamauchi2017-11-022-0/+213
| | | | | | | | | | | | | | | | | | | | | | | Summary: Currently the block frequency analysis is an approximation for irreducible loops. The new irreducible loop metadata is used to annotate the irreducible loop headers with their header weights based on the PGO profile (currently this is approximated to be evenly weighted) and to help improve the accuracy of the block frequency analysis for irreducible loops. This patch is a basic support for this. Reviewers: davidxl Reviewed By: davidxl Subscribers: mehdi_amini, llvm-commits, eraman Differential Revision: https://reviews.llvm.org/D39028 llvm-svn: 317278
* [ThinLTO] Fix dead stripping analysis for SamplePGOTeresa Johnson2017-09-202-0/+95
| | | | | | | | | | | | | | | | | | Summary: The fix for dead stripping analysis in the case of SamplePGO indirect calls to local functions (r313151) introduced the possibility of an infinite loop. Make sure we check for the value being already live after we update it for SamplePGO indirect call handling. Reviewers: danielcdh Subscribers: mehdi_amini, inglorion, llvm-commits, eraman Differential Revision: https://reviews.llvm.org/D38086 llvm-svn: 313766
* Fix bot failures by requiring x86 target in new testTeresa Johnson2017-09-131-0/+2
| | | | | | | | The test added in r313151 requires a target triple since it is running through code generation. Fix bot failures by requiring an x86 target. llvm-svn: 313153
* [ThinLTO] For SamplePGO, need to handle ICP targets consistently in thin linkTeresa Johnson2017-09-133-0/+125
| | | | | | | | | | | | | | | | | | | | | | Summary: SamplePGO indirect call profiles record the target as the original GUID for statics. The importer had special handling to map to the normal GUID in that case. The dead global analysis needs the same treatment or inconsistencies arise, resulting in linker unsats due to some dead symbols being exported and kept, leaving in references to other dead symbols that are removed. This can happen when a SamplePGO profile collected by one binary is used for a different binary, so the indirect call profiles may not accurately reflect live targets. Reviewers: danielcdh Subscribers: mehdi_amini, inglorion, llvm-commits, eraman Differential Revision: https://reviews.llvm.org/D37783 llvm-svn: 313151
* Add null check for promoted direct callDehao Chen2017-08-291-0/+44
| | | | | | | | | | | | | | Summary: We originally assume that in pgo-icp, the promoted direct call will never be null after strip point casts. However, stripPointerCasts is so smart that it could possibly return the value of the function call if it knows that the return value is always an argument. In this case, the returned value cannot cast to Instruction. In this patch, null check is added to ensure null pointer will not be accessed. Reviewers: tejohnson, xur, davidxl, djasper Reviewed By: tejohnson Subscribers: llvm-commits, sanjoy Differential Revision: https://reviews.llvm.org/D37252 llvm-svn: 312005
* Create PHI node for the return value only when the return value has uses.Taewook Oh2017-08-282-2/+71
| | | | | | | | | | | | | | | | | | | Summary: Currently, a phi node is created in the normal destination to unify the return values from promoted calls and the original indirect call. This patch makes this phi node to be created only when the return value has uses. This patch is necessary to generate valid code, as compiler crashes with the attached test case without this patch. Without this patch, an illegal phi node that has no incoming value from `entry`/`catch` is created in `cleanup` block. I think existing implementation is good as far as there is at least one use of the original indirect call. `insertCallRetPHI` creates a new phi node in the normal destination block only when the original indirect call dominates its use and the normal destination block. Otherwise, `fixupPHINodeForNormalDest` will handle the unification of return values naturally without creating a new phi node. However, if there's no use, `insertCallRetPHI` still creates a new phi node even when the original indirect call does not dominate the normal destination block, because `getCallRetPHINode` returns false. Reviewers: xur, davidxl, danielcdh Reviewed By: xur Subscribers: llvm-commits Differential Revision: https://reviews.llvm.org/D37176 llvm-svn: 311906
* [PGO] Set edge weights for indirectbr instruction with profile countsRong Xu2017-08-232-0/+61
| | | | | | | | | | | Current PGO only annotates the edge weight for branch and switch instructions with profile counts. We should also annotate the indirectbr instruction as all the information is there. This patch enables the annotating for indirectbr instructions. Also uses this annotation in branch probability analysis. Differential Revision: https://reviews.llvm.org/D37074 llvm-svn: 311604
* Keep Optimization Remark Yaml in NewPMSam Elliott2017-08-201-0/+4
| | | | | | | | | | | | | | | | | | | Summary: The New Pass Manager infrastructure was forgetting to keep around the optimization remark yaml file that the compiler might have been producing. This meant setting the option to '-' for stdout worked, but setting it to a filename didn't give file output (presumably it was deleted because compilation didn't explicitly keep it). This change just ensures that the file is kept if compilation succeeds. So far I have updated one of the optimization remark output tests to add a version with the new pass manager. It is my intention for this patch to also include changes to all tests that use `-opt-remark-output=` but I wanted to get the code patch ready for review while I was making all those changes. Fixes https://bugs.llvm.org/show_bug.cgi?id=33951 Reviewers: anemet, chandlerc Reviewed By: anemet, chandlerc Subscribers: javed.absar, chandlerc, fhahn, llvm-commits Differential Revision: https://reviews.llvm.org/D36906 llvm-svn: 311271
* [PGO] Fixed assertion due to mismatched memcpy size type.Ana Pazos2017-08-181-0/+27
| | | | | | | | | | | | | | | | Summary: Memcpy intrinsics have size argument of any integer type, like i32 or i64. Fixed size type along with its value when cloning the intrinsic. Reviewers: davidxl, xur Reviewed By: davidxl Subscribers: mcrosier, llvm-commits Differential Revision: https://reviews.llvm.org/D36844 llvm-svn: 311188
* Make ICP uses PSI to check for hotness.Dehao Chen2017-08-084-12/+11
| | | | | | | | | | | | | | Summary: Currently, ICP checks the count against a fixed value to see if it is hot enough to be promoted. This does not work for SamplePGO because sampled count may be much smaller. This patch uses PSI to check if the count is hot enough to be promoted. Reviewers: davidxl, tejohnson, eraman Reviewed By: davidxl Subscribers: sanjoy, llvm-commits, mehdi_amini Differential Revision: https://reviews.llvm.org/D36341 llvm-svn: 310416
* Migrate PGOMemOptSizeOpt to use new OptimizationRemarkEmitter PassSam Elliott2017-07-301-1/+35
| | | | | | | | | | | | | | | | | Summary: Fixes PR33790. This patch still needs a yaml-style test, which I shall write tomorrow Reviewers: anemet Reviewed By: anemet Subscribers: anemet, llvm-commits Differential Revision: https://reviews.llvm.org/D35981 llvm-svn: 309497
* Separate the ICP total threshold and remaining threshold.Dehao Chen2017-07-281-2/+9
| | | | | | | | | | | | | | Summary: In the current implementation, isPromotionProfitable only checks if the call count to a direct target is no less than a certain percentage threshold of the remaining call counts that have not been promoted. This causes code size problems when the target count is small but greater than a large portion of remaining counts. E.g. target1 takes 99.9%, while target2 takes 0.1%. Both targets will be promoted and inlined, makes the function size too large, which potentially prevents it from further inlining into its callers. This patch adds another percentage threshold against the total indirect call count. If the target count needs to be no less than both thresholds in order to be promoted speculatively. Reviewers: davidxl, tejohnson Reviewed By: tejohnson Subscribers: sanjoy, llvm-commits Differential Revision: https://reviews.llvm.org/D35962 llvm-svn: 309345
* [ICP] Migrate to OptimizationRemarkEmitterAdam Nemet2017-07-271-1/+1
| | | | | | | | | | | | | | | | | | | | | This is a module pass so for the old PM, we can't use ORE, the function analysis pass. Instead ORE is created on the fly. A few notes: - isPromotionLegal is folded in the caller since we want to emit the Function in the remark but we can only do that if the symbol table look-up succeeded. - There was good test coverage for remarks in this pass. - promoteIndirectCall uses ORE conditionally since it's also used from SampleProfile which does not use ORE yet. Fixes PR33792. Differential Revision: https://reviews.llvm.org/D35929 llvm-svn: 309294
* [PGO] Move the PGOInstrumentation pass to new OptRemark API.Davide Italiano2017-07-201-2/+2
| | | | | | This fixes PR33791. llvm-svn: 308668
* [PGO] Enhance pgo counter promotionXinliang David Li2017-07-123-4/+169
| | | | | | | | | | | | | | | | | | | | | | | This is an incremental change to the promotion feature. There are two problems with the current behavior: 1) loops with multiple exiting blocks are totally disabled 2) a counter update can only be promoted one level up in the loop nest -- which does help much for short trip count inner loops inside a high trip-count outer loops. Due to this limitation, we still saw very large profile count fluctuations from run to run for the affected loops which are usually very hot. This patch adds the support for promotion counters iteratively across the loop nest. It also turns on the promotion for loops with multiple exiting blocks (with a limit). For single-threaded applications, the performance impact is flat on average. For instance, dealII improves, but povray regresses. llvm-svn: 307863
* [PGO] Implementate profile counter regiser promotionXinliang David Li2017-06-253-0/+222
| | | | | | Differential Revision: http://reviews.llvm.org/D34085 llvm-svn: 306231
* [PATCH] [PGO] Fixed cast operation in ↵Ana Pazos2017-06-191-0/+14
| | | | | | | | | | | | | | emIntrinsicVisitor::instrumentOneMemIntrinsic. Reviewers: xur, efriedma, davidxl Reviewed By: davidxl Subscribers: llvm-commits Differential Revision: https://reviews.llvm.org/D34293 llvm-svn: 305737
* [PGO] Update VP metadata after memory intrinsic optimizationTeresa Johnson2017-06-131-2/+7
| | | | | | | | | | | | | | | Summary: Leave an updated VP metadata on the fallback memcpy intrinsic after specialization. This can be used for later possible expansion based on the average of the remaining values. Reviewers: davidxl Subscribers: llvm-commits Differential Revision: https://reviews.llvm.org/D34164 llvm-svn: 305321
* [PartialInlining] Emit branch info and profile data as remarksXinliang David Li2017-06-011-0/+5
| | | | | | | | | This allows us to collect profile statistics to tune static branch prediction. Differential Revision: http://reviews.llvm.org/D33746 llvm-svn: 304452
* Memory intrinsic value profile optimization: Avoid divide by 0Teresa Johnson2017-04-281-0/+19
| | | | | | | | | | | | | | Summary: Skip memops if the total value profiled count is 0, we can't correctly scale up the counts and there is no point anyway. Reviewers: davidxl Subscribers: llvm-commits Differential Revision: https://reviews.llvm.org/D32624 llvm-svn: 301645
* Update profile during memory instrinsic optimizationTeresa Johnson2017-04-241-5/+16
| | | | | | | | | | | | | | | | Summary: Ensure that the new merge BB (which contains the rest of the original BB after the mem op being optimized) gets a profile frequency, in case there are additional mem ops later in the BB. Otherwise they get skipped as the merge BB looks cold. Reviewers: davidxl, xur Subscribers: llvm-commits Differential Revision: https://reviews.llvm.org/D32447 llvm-svn: 301244
* [PGO] Memory intrinsic calls optimization based on profiled sizeRong Xu2017-04-041-0/+100
| | | | | | | | | | | | | | | | | | | | | | | | This patch optimizes two memory intrinsic operations: memset and memcpy based on the profiled size of the operation. The high level transformation is like: mem_op(..., size) ==> switch (size) { case s1: mem_op(..., s1); goto merge_bb; case s2: mem_op(..., s2); goto merge_bb; ... default: mem_op(..., size); goto merge_bb; } merge_bb: Differential Revision: http://reviews.llvm.org/D28966 llvm-svn: 299446
* [PGO] Add omitted test cases.Rong Xu2017-03-171-0/+35
| | | | llvm-svn: 298115
* [PGO] Value profile for size of memory intrinsic callsRong Xu2017-03-172-0/+86
| | | | | | | | This patch annotates the valuesites profile to memory intrinsics. Differential Revision: http://reviews.llvm.org/D31002 llvm-svn: 298110
* Resubmit r297897: [PGO] Value profile for size of memory intrinsic callsRong Xu2017-03-161-2/+2
| | | | | | | R297897 inadvertently enabled annotation for memop profiling. This new patch fixed it. llvm-svn: 297996
* Revert "[PGO] Value profile for size of memory intrinsic calls"Eric Liu2017-03-161-2/+2
| | | | | | This commit reverts r297897 and r297909. llvm-svn: 297951
* [PGO] Value profile for size of memory intrinsic callsRong Xu2017-03-151-2/+2
| | | | | | | | | This patch adds the value profile support to profile the size parameter of memory intrinsic calls: memcpy, memcmp, and memmov. Differential Revision: http://reviews.llvm.org/D28965 llvm-svn: 297897
* SamplePGO ThinLTO ICP fix for local functions.Dehao Chen2017-03-142-0/+90
| | | | | | | | | | | | | | | | | | | | | | | | Summary: In SamplePGO, if the profile is collected from non-LTO binary, and used to drive ThinLTO, the indirect call promotion may fail because ThinLTO adjusts local function names to avoid conflicts. There are two places of where the mismatch can happen: 1. thin-link prepends SourceFileName to front of FuncName to build the GUID (GlobalValue::getGlobalIdentifier). Unlike instrumentation FDO, SamplePGO does not use the PGOFuncName scheme and therefore the indirect call target profile data contains a hash of the OriginalName. 2. backend compiler promotes some local functions to global and appends .llvm.{$ModuleHash} to the end of the FuncName to derive PromotedFunctionName This patch tries at the best effort to find the GUID from the original local function name (in profile), and use that in ICP promotion, and in SamplePGO matching that happens in the backend after importing/inlining: 1. in thin-link, it builds the map from OriginalName to GUID so that when thin-link reads in indirect call target profile (represented by OriginalName), it knows which GUID to import. 2. in backend compiler, if sample profile reader cannot find a profile match for PromotedFunctionName, it will try to find if there is a match for OriginalFunctionName. 3. in backend compiler, we build symbol table entry for OriginalFunctionName and pointer to the same symbol of PromotedFunctionName, so that ICP can find the correct target to promote. Reviewers: mehdi_amini, tejohnson Reviewed By: tejohnson Subscribers: llvm-commits, Prazek Differential Revision: https://reviews.llvm.org/D30754 llvm-svn: 297757
* [PGO] Directory name stripping in global identifier for static functionsRong Xu2017-02-251-3/+8
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Current internal option -static-func-full-module-prefix keeps all the directory path the profile counter names for static functions. The default of this option is false. This strips the directory names from the source filename which is problematic: (1) it creates linker errors for profile-generation compilation, exposed in our internal benchmarks. We are seeing messages like "warning: relocation refers to discarded section". This is due to the name conflicts after the stripping. (2) the stripping only applies to getPGOFuncName. Current Thin-LTO module importing for the indirect-calls assumes the source directory name not being stripped. Current default value for this option can potentially prevent some inter-module indirect-call-promotions. This patch turns the default value for -static-func-full-module-prefix to true. The second part of the patch is to have an alternative implementation under the internal option -static-func-strip-dirname-prefix=<value> This options specifies level of directories to be stripped from the source filename. Using a large value as the parameter has the same effect as -static-func-full-module-prefix. Differential Revision: http://reviews.llvm.org/D29512 llvm-svn: 296206
* Add call branch annotation for ICP promoted direct call in SamplePGO mode.Dehao Chen2017-02-231-0/+4
| | | | | | | | | | | | | | Summary: SamplePGO uses branch_weight annotation to represent callsite hotness. When ICP promotes an indirect call to direct call, we need to make sure the direct call is annotated with branch_weight in SamplePGO mode, so that downstream function inliner can use hot callsite heuristic. Reviewers: davidxl, eraman, xur Reviewed By: davidxl, xur Subscribers: mehdi_amini, llvm-commits Differential Revision: https://reviews.llvm.org/D30282 llvm-svn: 296028
* Re-apply "[profiling] Remove dead profile name vars after emitting name data"Vedant Kumar2017-02-141-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | This reverts 295092 (re-applies 295084), with a fix for dangling references from the array of coverage names passed down from frontends. I missed this in my initial testing because I only checked test/Profile, and not test/CoverageMapping as well. Original commit message: The profile name variables passed to counter increment intrinsics are dead after we emit the finalized name data in __llvm_prf_nm. However, we neglect to erase these name variables. This causes huge size increases in the __TEXT,__const section as well as slowdowns when linker dead stripping is disabled. Some affected projects are so massive that they fail to link on Darwin, because only the small code model is supported. Fix the issue by throwing away the name constants as soon as we're done with them. Differential Revision: https://reviews.llvm.org/D29921 llvm-svn: 295099
* Revert "[profiling] Remove dead profile name vars after emitting name data"Vedant Kumar2017-02-141-1/+1
| | | | | | | | This reverts commit r295084. There is a test failure on: http://lab.llvm.org:8011/builders/clang-atom-d525-fedora-rel/builds/2620/ llvm-svn: 295092
* [profiling] Remove dead profile name vars after emitting name dataVedant Kumar2017-02-141-1/+1
| | | | | | | | | | | | | | | | The profile name variables passed to counter increment intrinsics are dead after we emit the finalized name data in __llvm_prf_nm. However, we neglect to erase these name variables. This causes huge size increases in the __TEXT,__const section as well as slowdowns when linker dead stripping is disabled. Some affected projects are so massive that they fail to link on Darwin, because only the small code model is supported. Fix the issue by throwing away the name constants as soon as we're done with them. Differential Revision: https://reviews.llvm.org/D29921 llvm-svn: 295084
* Fix some broken CHECK lines.Benjamin Kramer2017-01-221-2/+2
| | | | | | The colon is important. llvm-svn: 292761
* Resubmit "[PGO] Turn off comdat renaming in IR PGO by default"Rong Xu2017-01-115-21/+81
| | | | | | This patch resubmits the changes in r291588. llvm-svn: 291696
* Revert "[PGO] Turn off comdat renaming in IR PGO by default"Rong Xu2017-01-105-81/+21
| | | | | | | This patch reverts r291588: [PGO] Turn off comdat renaming in IR PGO by default, as we are seeing some hash mismatches in our internal tests. llvm-svn: 291621
* [PGO] Turn off comdat renaming in IR PGO by defaultRong Xu2017-01-105-21/+81
| | | | | | | | | | | | | | | | | | | | | | | Summary: In IR PGO we append the function hash to comdat functions to avoid the potential hash mismatch. This turns out not legal in some cases: if the comdat function is address-taken and used in comparison. Renaming changes the semantic. This patch turns off comdat renaming by default. To alleviate the hash mismatch issue, we now rename the profile variable for comdat functions. Profile allows co-existing multiple versions of profiles with different hash value. The inlined copy will always has the correct profile counter. The out-of-line copy might not have the correct count. But we will not have the bogus mismatch warning. Reviewers: davidxl Subscribers: llvm-commits, xur Differential Revision: https://reviews.llvm.org/D28416 llvm-svn: 291588
* Fix the test cases committed in r289521. Rong Xu2016-12-131-10/+8
| | | | llvm-svn: 289556
* llvm/test/Transforms/PGOProfile/noreturncall.ll REQUIRES asserts due to ↵NAKAMURA Takumi2016-12-131-0/+1
| | | | | | -debug-only. llvm-svn: 289522
* [PGO] Fix insane counts due to nonreturn callsRong Xu2016-12-132-0/+57
| | | | | | | | | | | | | | | | Summary: Since we don't break BBs for function calls. We might get some insane counts (wrap of unsigned) in the presence of noreturn calls. This patch sets these counts to zero instead of the wrapped number. Reviewers: davidxl Subscribers: xur, eraman, llvm-commits Differential Revision: https://reviews.llvm.org/D27602 llvm-svn: 289521
* [PGO] Fix PGO use ICE when there are unreachable BBsRong Xu2016-12-022-0/+32
| | | | | | | | | | | | | | For -O0 there might be unreachable BBs, which breaks the assumption that all the BBs have an auxiliary data structure. In this patch, we add another interface called findBBInfo() so that a nullptr can be returned for the unreachable BBs (and the callers can ignore those BBs). This fixes the bug reported https://llvm.org/bugs/show_bug.cgi?id=31209 Differential Revision: https://reviews.llvm.org/D27280 llvm-svn: 288528
* [PGO] Fix select instruction annotationRong Xu2016-10-252-0/+48
| | | | | | | | | | | | | | | | Summary: Select instruction annotation in IR PGO uses the edge count to infer the branch count. It's currently placed in setInstrumentedCounts() where no all the BB counts have been computed. This leads to wrong branch weights. Move the annotation after all BB counts are populated. Reviewers: davidxl Subscribers: llvm-commits Differential Revision: https://reviews.llvm.org/D25961 llvm-svn: 285128
* [PGO] Create weak alias for the renamed Comdat functionRong Xu2016-10-061-0/+6
| | | | | | | | | | Add a weak alias to the renamed Comdat function in IR level instrumentation, using it's original name. This ensures the same behavior w/ and w/o IR instrumentation, even for non standard conforming code. Differential Revision: http://reviews.llvm.org/D25339 llvm-svn: 283490
* [Profile] Implement select instruction instrumentation in IR PGOXinliang David Li2016-09-182-0/+42
| | | | | | Differential Revision: http://reviews.llvm.org/D23727 llvm-svn: 281858
* [ThinLTO] Indirect call promotion fixes for promoted local functionsTeresa Johnson2016-08-292-3/+19
| | | | | | | | | | | | | | | | | | | Summary: Fix a couple issues limiting the application of indirect call promotion in ThinLTO mode: - Invoke indirect call promotion before globalopt, since it may eliminate imported functions which appear unreferenced. - Invoke indirect call promotion with InLTO=true so that the PGOFuncName metadata is used to get the name for locals which would have been renamed during promotion. Reviewers: davidxl, mehdi_amini Subscribers: Prazek, llvm-commits, mehdi_amini Differential Revision: https://reviews.llvm.org/D24004 llvm-svn: 280024
* [Profile] add test with large countsXinliang David Li2016-08-202-0/+15
| | | | llvm-svn: 279361
OpenPOWER on IntegriCloud