summaryrefslogtreecommitdiffstats
path: root/llvm/lib/Transforms/Instrumentation/PGOInstrumentation.cpp
Commit message (Collapse)AuthorAgeFilesLines
* The error message for mismatched value sites is very cryptic.Dmitry Mikulin2019-04-231-2/+10
| | | | | | | | Make it more readable for an average user. Differential Revision: https://reviews.llvm.org/D60896 llvm-svn: 359043
* [PGO] Better handling of profile hash mismatchRong Xu2019-04-111-6/+20
| | | | | | | | | | | We currently assume profile hash conflicts will be caught by an upfront check and we assert for the cases that escape the check. The assumption is not always true as there are chances of conflict. This patch prints a warning and skips annotating the function for the escaped cases,. Differential Revision: https://reviews.llvm.org/D60154 llvm-svn: 358225
* Change some dyn_cast to more apropriate isa. NFCFangrui Song2019-04-051-1/+1
| | | | llvm-svn: 357773
* [NFC] Fix PGO link error in shared libs buildJordan Rupprecht2019-03-041-1/+8
| | | | llvm-svn: 355346
* [PGO] Context sensitive PGO (part 2)Rong Xu2019-02-281-2/+6
| | | | | | | | | | | Part 2 of CSPGO changes (mostly related to ProfileSummary). Note that I use a default parameter in setProfileSummary() and getSummary(). This is to break the dependency in clang. I will make the parameter explicit after changing clang in a separated patch. Differential Revision: https://reviews.llvm.org/D54175 llvm-svn: 355131
* Recommit r354930 "[PGO] Context sensitive PGO (part 1)"Rong Xu2019-02-271-56/+133
| | | | | | Fixed UBSan failures. llvm-svn: 355005
* Revert "[PGO] Context sensitive PGO (part 1)"Vlad Tsyrklevich2019-02-271-147/+56
| | | | | | This reverts commit r354930, it was causing UBSan failures. llvm-svn: 354953
* [PGO] Context sensitive PGO (part 1)Rong Xu2019-02-261-56/+147
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Current PGO profile counts are not context sensitive. The branch probabilities for the inlined functions are kept the same for all call-sites, and they might be very different from the actual branch probabilities. These suboptimal profiles can greatly affect some downstream optimizations, in particular for the machine basic block placement optimization. In this patch, we propose to have a post-inline PGO instrumentation/use pass, which we called Context Sensitive PGO (CSPGO). For the users who want the best possible performance, they can perform a second round of PGO instrument/use on the top of the regular PGO. They will have two sets of profile counts. The first pass profile will be manly for inline, indirect-call promotion, and CGSCC simplification pass optimizations. The second pass profile is for post-inline optimizations and code-gen optimizations. A typical usage: // Regular PGO instrumentation and generate pass1 profile. > clang -O2 -fprofile-generate source.c -o gen > ./gen > llvm-profdata merge default.*profraw -o pass1.profdata // CSPGO instrumentation. > clang -O2 -fprofile-use=pass1.profdata -fcs-profile-generate -o gen2 > ./gen2 // Merge two sets of profiles > llvm-profdata merge default.*profraw pass1.profdata -o profile.profdata // Use the combined profile. Pass manager will invoke two PGO use passes. > clang -O2 -fprofile-use=profile.profdata -o use This change touches many components in the compiler. The reviewed patch (D54175) will committed in phrases. Differential Revision: https://reviews.llvm.org/D54175 llvm-svn: 354930
* Update the file headers across all of the LLVM projects in the monorepoChandler Carruth2019-01-191-4/+3
| | | | | | | | | | | | | | | | | to reflect the new license. We understand that people may be surprised that we're moving the header entirely to discuss the new license. We checked this carefully with the Foundation's lawyer and we believe this is the correct approach. Essentially, all code in the project is now made available by the LLVM project under our new license, so you will see that the license headers include that license only. Some of our contributors have contributed code under our old license, and accordingly, we have retained a copy of our old license notice in the top-level files in each project and repository. llvm-svn: 351636
* [CallSite removal] Port `IndirectCallSiteVisitor` to use `CallBase` andChandler Carruth2019-01-071-6/+6
| | | | | | | | | | | update client code. Also rename it to use the more generic term `call` instead of something that could be confused with a praticular type. Differential Revision: https://reviews.llvm.org/D56183 llvm-svn: 350508
* [PGO] Exit early if all count values are zeroRong Xu2018-11-071-3/+12
| | | | | | | | | If all the edge counts for a function are zero, skip count population and annotation, as nothing will happen. This can save some compile time. Differential Revision: https://reviews.llvm.org/D54212 llvm-svn: 346370
* [TI removal] Make variables declared as `TerminatorInst` and initializedChandler Carruth2018-10-151-4/+4
| | | | | | | | | | | | | by `getTerminator()` calls instead be declared as `Instruction`. This is the biggest remaining chunk of the usage of `getTerminator()` that insists on the narrow type and so is an easy batch of updates. Several files saw more extensive updates where this would cascade to requiring API updates within the file to use `Instruction` instead of `TerminatorInst`. All of these were trivial in nature (pervasively using `Instruction` instead just worked). llvm-svn: 344502
* Add a flag to remap manglings when reading profile data information.Richard Smith2018-10-101-6/+17
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This can be used to preserve profiling information across codebase changes that have widespread impact on mangled names, but across which most profiling data should still be usable. For example, when switching from libstdc++ to libc++, or from the old libstdc++ ABI to the new ABI, or even from a 32-bit to a 64-bit build. The user can provide a remapping file specifying parts of mangled names that should be treated as equivalent (eg, std::__1 should be treated as equivalent to std::__cxx11), and profile data will be treated as applying to a particular function if its name is equivalent to the name of a function in the profile data under the provided equivalences. See the documentation change for a description of how this is configured. Remapping is supported for both sample-based profiling and instruction profiling. We do not support remapping indirect branch target information, but all other profile data should be remapped appropriately. Support is only added for the new pass manager. If someone wants to also add support for this for the old pass manager, doing so should be straightforward. This is the LLVM side of Clang r344199. Reviewers: davidxl, tejohnson, dlj, erik.pilkington Subscribers: mehdi_amini, steven_wu, dexonsmith, llvm-commits Differential Revision: https://reviews.llvm.org/D51249 llvm-svn: 344200
* Rename DEBUG macro to LLVM_DEBUG.Nicola Zaghen2018-05-141-28/+30
| | | | | | | | | | | | | | | | The DEBUG() macro is very generic so it might clash with other projects. The renaming was done as follows: - git grep -l 'DEBUG' | xargs sed -i 's/\bDEBUG\s\?(/LLVM_DEBUG(/g' - git diff -U0 master | ../clang/tools/clang-format/clang-format-diff.py -i -p1 -style LLVM - Manual change to APInt - Manually chage DOCS as regex doesn't match it. In the transition period the DEBUG() macro is still present and aliased to the LLVM_DEBUG() one. Differential Revision: https://reviews.llvm.org/D43624 llvm-svn: 332240
* Remove \brief commands from doxygen comments.Adrian Prantl2018-05-011-1/+1
| | | | | | | | | | | | | | | | We've been running doxygen with the autobrief option for a couple of years now. This makes the \brief markers into our comments redundant. Since they are a visual distraction and we don't want to encourage more \brief markers in new code either, this patch removes them all. Patch produced by for i in $(git grep -l '\\brief'); do perl -pi -e 's/\\brief //g' $i & done Differential Revision: https://reviews.llvm.org/D46290 llvm-svn: 331272
* [NFC] fix trivial typos in document and commentsHiroshi Inoue2018-04-141-1/+1
| | | | | | "not not" -> "not" etc llvm-svn: 330083
* [PGO] Fix branch probability remarks assertRong Xu2018-03-271-7/+9
| | | | | | | | | Fixed counter/weight overflow that leads to an assertion. Also fixed the help string for pgo-emit-branch-prob option. Differential Revision: https://reviews.llvm.org/D44809 llvm-svn: 328653
* Fix Layering, move instrumentation transform headers into Instrumentation ↵David Blaikie2018-03-231-1/+1
| | | | | | subdirectory llvm-svn: 328379
* Add a ProfileCount class to represent entry counts.Easwaran Raman2018-01-171-1/+2
| | | | | | | | | | | | | | Summary: The class wraps a uint64_t and an enum to represent the type of profile count (real and synthetic) with some helper methods. Reviewers: davidxl Subscribers: llvm-commits Differential Revision: https://reviews.llvm.org/D41883 llvm-svn: 322771
* Split IndirectBr critical edges before PGO gen/use passes.Hiroshi Yamauchi2017-12-121-0/+6
| | | | | | | | | | | | | | | | | | | | | | Summary: The PGO gen/use passes currently fail with an assert failure if there's a critical edge whose source is an IndirectBr instruction and that edge needs to be instrumented. To avoid this in certain cases, split IndirectBr critical edges in the PGO gen/use passes. This works for blocks with single indirectbr predecessors, but not for those with multiple indirectbr predecessors (splitting an IndirectBr critical edge isn't always possible.) Reviewers: davidxl, xur Reviewed By: davidxl Subscribers: efriedma, llvm-commits, mehdi_amini Differential Revision: https://reviews.llvm.org/D40699 llvm-svn: 320511
* [PGO] change arg type to uint64_t to match member field typeXinliang David Li2017-12-101-2/+2
| | | | llvm-svn: 320285
* Revert r320104: infinite loop profiling bug fixXinliang David Li2017-12-081-12/+34
| | | | | | | | | | | Causes unexpected memory issue with New PM this time. The new PM invalidates BPI but not BFI, leaving the reference to BPI from BFI invalid. Abandon this patch. There is a more general solution which also handles runtime infinite loop (but not statically). llvm-svn: 320180
* [PGO] detect infinite loop and form MST properlyXinliang David Li2017-12-071-34/+12
| | | | | | Differential Revision: http://reviews.llvm.org/D40873 llvm-svn: 320104
* Revert r319794: [PGO] detect infinite loop and form MST properly: memory ↵Xinliang David Li2017-12-051-37/+13
| | | | | | leak problem llvm-svn: 319841
* [PGO] detect infinite loop and form MST properlyXinliang David Li2017-12-051-13/+37
| | | | | | Differential Revision: http://reviews.llvm.org/D40702 llvm-svn: 319794
* Add heuristics for irreducible loop metadata under PGOHiroshi Yamauchi2017-11-201-1/+12
| | | | | | | | | | | | | | | | | | | | | | | | | Summary: Add the following heuristics for irreducible loop metadata: - When an irreducible loop header is missing the loop header weight metadata, give it the minimum weight seen among other headers. - Annotate indirectbr targets with the loop header weight metadata (as they are likely to become irreducible loop headers after indirectbr tail duplication.) These greatly improve the accuracy of the block frequency info of the Python interpreter loop (eg. from ~3-16x off down to ~40-55% off) and the Python performance (eg. unpack_sequence from ~50% slower to ~8% faster than GCC) due to better register allocation under PGO. Reviewers: davidxl Reviewed By: davidxl Subscribers: llvm-commits Differential Revision: https://reviews.llvm.org/D39980 llvm-svn: 318693
* Irreducible loop metadata for more accurate block frequency under PGO.Hiroshi Yamauchi2017-11-021-2/+26
| | | | | | | | | | | | | | | | | | | | | | | Summary: Currently the block frequency analysis is an approximation for irreducible loops. The new irreducible loop metadata is used to annotate the irreducible loop headers with their header weights based on the PGO profile (currently this is approximated to be evenly weighted) and to help improve the accuracy of the block frequency analysis for irreducible loops. This patch is a basic support for this. Reviewers: davidxl Reviewed By: davidxl Subscribers: mehdi_amini, llvm-commits, eraman Differential Revision: https://reviews.llvm.org/D39028 llvm-svn: 317278
* [Transforms] Fix some Clang-tidy modernize and Include What You Use ↵Eugene Zelenko2017-10-211-61/+121
| | | | | | warnings; other minor fixes (NFC). llvm-svn: 316253
* [NFC] Convert OptimizationRemarkEmitter old emit() calls to new closureVivek Pandya2017-10-111-2/+4
| | | | | | | | | | | | | | parameterized emit() calls Summary: This is not functional change to adopt new emit() API added in r313691. Reviewed By: anemet Subscribers: llvm-commits Differential Revision: https://reviews.llvm.org/D38285 llvm-svn: 315476
* Rename OptimizationDiagnosticInfo.* to OptimizationRemarkEmitter.*Adam Nemet2017-10-091-1/+1
| | | | | | | Sync it up with the name of the class actually defined here. This has been bothering me for a while... llvm-svn: 315249
* Add options to dump PGO counts in text.Hiroshi Yamauchi2017-09-131-20/+38
| | | | | | | | | | | | | | | | | Summary: Added text options to -pgo-view-counts and -pgo-view-raw-counts that dump block frequency and branch probability info in text. This is useful when the graph is very large and complex (the dot command crashes, lines/edges too close to tell apart, hard to navigate without textual search) or simply when text is preferred. Reviewers: davidxl Reviewed By: davidxl Subscribers: llvm-commits Differential Revision: https://reviews.llvm.org/D37776 llvm-svn: 313159
* [PGO] Set edge weights for indirectbr instruction with profile countsRong Xu2017-08-231-1/+2
| | | | | | | | | | | Current PGO only annotates the edge weight for branch and switch instructions with profile counts. We should also annotate the indirectbr instruction as all the information is there. This patch enables the annotating for indirectbr instructions. Also uses this annotation in branch probability analysis. Differential Revision: https://reviews.llvm.org/D37074 llvm-svn: 311604
* [PGOInstr] Add a debug printXinliang David Li2017-07-211-0/+6
| | | | llvm-svn: 308785
* [PGO] Move the PGOInstrumentation pass to new OptRemark API.Davide Italiano2017-07-201-4/+4
| | | | | | This fixes PR33791. llvm-svn: 308668
* [Constants] If we already have a ConstantInt*, prefer to use ↵Craig Topper2017-07-061-1/+1
| | | | | | | | isZero/isOne/isMinusOne instead of isNullValue/isOneValue/isAllOnesValue inherited from Constant. NFCI Going through the Constant methods requires redetermining that the Constant is a ConstantInt and then calling isZero/isOne/isMinusOne. llvm-svn: 307292
* [PATCH] [PGO] Fixed cast operation in ↵Ana Pazos2017-06-191-1/+1
| | | | | | | | | | | | | | emIntrinsicVisitor::instrumentOneMemIntrinsic. Reviewers: xur, efriedma, davidxl Reviewed By: davidxl Subscribers: llvm-commits Differential Revision: https://reviews.llvm.org/D34293 llvm-svn: 305737
* [PartialInlining] Emit branch info and profile data as remarksXinliang David Li2017-06-011-0/+64
| | | | | | | | | This allows us to collect profile statistics to tune static branch prediction. Differential Revision: http://reviews.llvm.org/D33746 llvm-svn: 304452
* Fix a couple of typos in memory intrinsic optimization output (NFC)Teresa Johnson2017-05-241-1/+1
| | | | | | s/instrinsic/intrinsic llvm-svn: 303782
* [PGO] Memory intrinsic calls optimization based on profiled sizeRong Xu2017-04-041-15/+15
| | | | | | | | | | | | | | | | | | | | | | | | This patch optimizes two memory intrinsic operations: memset and memcpy based on the profiled size of the operation. The high level transformation is like: mem_op(..., size) ==> switch (size) { case s1: mem_op(..., s1); goto merge_bb; case s2: mem_op(..., s2); goto merge_bb; ... default: mem_op(..., size); goto merge_bb; } merge_bb: Differential Revision: http://reviews.llvm.org/D28966 llvm-svn: 299446
* [PGO] Change the internal options description. nfc.Rong Xu2017-03-171-3/+5
| | | | llvm-svn: 298120
* [PGO] Value profile for size of memory intrinsic callsRong Xu2017-03-171-5/+22
| | | | | | | | This patch annotates the valuesites profile to memory intrinsics. Differential Revision: http://reviews.llvm.org/D31002 llvm-svn: 298110
* Resubmit r297897: [PGO] Value profile for size of memory intrinsic callsRong Xu2017-03-161-2/+86
| | | | | | | R297897 inadvertently enabled annotation for memop profiling. This new patch fixed it. llvm-svn: 297996
* Revert "[PGO] Value profile for size of memory intrinsic calls"Eric Liu2017-03-161-83/+1
| | | | | | This commit reverts r297897 and r297909. llvm-svn: 297951
* Revert "Revert "[PGO] Minor cleanup for count instruction in ↵Vitaly Buka2017-03-151-1/+4
| | | | | | | | | | SelectInstVisitor."" Previously reverted wrong revision. This reverts commit r297910. llvm-svn: 297911
* Revert "[PGO] Minor cleanup for count instruction in SelectInstVisitor."Vitaly Buka2017-03-151-4/+1
| | | | | | | | Fails LLVMFuzzer.LLVMFuzzer.value-profile-strncmp.test This reverts commit r297892. llvm-svn: 297910
* [PGO] Value profile for size of memory intrinsic callsRong Xu2017-03-151-1/+83
| | | | | | | | | This patch adds the value profile support to profile the size parameter of memory intrinsic calls: memcpy, memcmp, and memmov. Differential Revision: http://reviews.llvm.org/D28965 llvm-svn: 297897
* [PGO] Minor cleanup for count instruction in SelectInstVisitor.Rong Xu2017-03-151-1/+4
| | | | | | | | | | | | | | | | | Summary: NSIs can be double-counted by different operations in SelectInstVisitor. Sink the the update to VM_counting mode only. Also reset the value for each counting operation. Reviewers: davidxl Reviewed By: davidxl Subscribers: xur, llvm-commits Differential Revision: https://reviews.llvm.org/D30999 llvm-svn: 297892
* [PGO] Refactor the code for value profile annotationRong Xu2017-03-151-29/+38
| | | | | | | | | This patch refactors the code for value profile annotation to facilitate of adding other kind of value profiles. Differential Revision: http://reviews.llvm.org/D30989 llvm-svn: 297870
* Minor format change. nfc.Rong Xu2017-03-091-5/+5
| | | | llvm-svn: 297400
* Fix variable name /NFCXinliang David Li2017-02-041-2/+2
| | | | llvm-svn: 294090
OpenPOWER on IntegriCloud