summaryrefslogtreecommitdiffstats
path: root/llvm/lib/Transforms/Instrumentation/InstrProfiling.cpp
Commit message (Collapse)AuthorAgeFilesLines
* [InstrProf] Use separate comdat group for data and countersReid Kleckner2019-02-271-19/+12
| | | | | | | | | | | | | | | | | | | | | | | Summary: I hadn't realized that instrumentation runs before inlining, so we can't use the function as the comdat group. Doing so can create relocations against discarded sections when references to discarded __profc_ variables are inlined into functions outside the function's comdat group. In the future, perhaps we should consider standardizing the comdat group names that ELF and COFF use. It will save object file size, since __profv_$sym won't appear in the symbol table again. Reviewers: xur, vsk Subscribers: eraman, hiraditya, cfe-commits, #sanitizers, llvm-commits Tags: #clang, #sanitizers, #llvm Differential Revision: https://reviews.llvm.org/D58737 llvm-svn: 355044
* Recommit r354930 "[PGO] Context sensitive PGO (part 1)"Rong Xu2019-02-271-10/+43
| | | | | | Fixed UBSan failures. llvm-svn: 355005
* Revert "[PGO] Context sensitive PGO (part 1)"Vlad Tsyrklevich2019-02-271-43/+10
| | | | | | This reverts commit r354930, it was causing UBSan failures. llvm-svn: 354953
* [PGO] Context sensitive PGO (part 1)Rong Xu2019-02-261-10/+43
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Current PGO profile counts are not context sensitive. The branch probabilities for the inlined functions are kept the same for all call-sites, and they might be very different from the actual branch probabilities. These suboptimal profiles can greatly affect some downstream optimizations, in particular for the machine basic block placement optimization. In this patch, we propose to have a post-inline PGO instrumentation/use pass, which we called Context Sensitive PGO (CSPGO). For the users who want the best possible performance, they can perform a second round of PGO instrument/use on the top of the regular PGO. They will have two sets of profile counts. The first pass profile will be manly for inline, indirect-call promotion, and CGSCC simplification pass optimizations. The second pass profile is for post-inline optimizations and code-gen optimizations. A typical usage: // Regular PGO instrumentation and generate pass1 profile. > clang -O2 -fprofile-generate source.c -o gen > ./gen > llvm-profdata merge default.*profraw -o pass1.profdata // CSPGO instrumentation. > clang -O2 -fprofile-use=pass1.profdata -fcs-profile-generate -o gen2 > ./gen2 // Merge two sets of profiles > llvm-profdata merge default.*profraw pass1.profdata -o profile.profdata // Use the combined profile. Pass manager will invoke two PGO use passes. > clang -O2 -fprofile-use=profile.profdata -o use This change touches many components in the compiler. The reviewed patch (D54175) will committed in phrases. Differential Revision: https://reviews.llvm.org/D54175 llvm-svn: 354930
* [InstrProf] Implement static profdata registrationReid Kleckner2019-02-081-32/+52
| | | | | | | | | | | | | | | | | | | | | | | | | | | | Summary: The motivating use case is eliminating duplicate profile data registered for the same inline function in two object files. Before this change, users would observe multiple symbol definition errors with VC link, but links with LLD would succeed. Users (Mozilla) have reported that PGO works well with clang-cl and LLD, but when using LLD without this static registration, we would get into a "relocation against a discarded section" situation. I'm not sure what happens in that situation, but I suspect that duplicate, unused profile information was retained. If so, this change will reduce the size of such binaries with LLD. Now, Windows uses static registration and is in line with all the other platforms. Reviewers: davidxl, wmi, inglorion, void, calixte Subscribers: mgorny, krytarowski, eraman, fedor.sergeev, hiraditya, #sanitizers, dmajor, llvm-commits Tags: #sanitizers, #llvm Differential Revision: https://reviews.llvm.org/D57929 llvm-svn: 353547
* [InstrProf] Avoid reconstructing Triple, NFCReid Kleckner2019-02-071-17/+15
| | | | llvm-svn: 353439
* [LICM/MSSA] Add promotion to scalars by building an AliasSetTracker with ↵Alina Sbirlea2019-02-061-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | MemorySSA. Summary: Experimentally we found that promotion to scalars carries less benefits than sinking and hoisting in LICM. When using MemorySSA, we build an AliasSetTracker on demand in order to reuse the current infrastructure. We only build it if less than AccessCapForMSSAPromotion exist in the loop, a cap that is by default set to 250. This value ensures there are no runtime regressions, and there are small compile time gains for pathological cases. A much lower value (20) was found to yield a single regression in the llvm-test-suite and much higher benefits for compile times. Conservatively we set the current cap to a high value, but we will explore lowering it when MemorySSA is enabled by default. Reviewers: sanjoy, chandlerc Subscribers: nemanjai, jlebar, Prazek, george.burgess.iv, jfb, jsji, llvm-commits Differential Revision: https://reviews.llvm.org/D56625 llvm-svn: 353339
* [PGO] Use a function for creating variable for profile file name. NFC.Rong Xu2019-02-051-16/+2
| | | | | | | Factored out the code for creating variable for profile file name to a function. llvm-svn: 353230
* [opaque pointer types] Pass value type to GetElementPtr creation.James Y Knight2019-02-011-1/+2
| | | | | | | | | This cleans up all GetElementPtr creation in LLVM to explicitly pass a value type rather than deriving it from the pointer's element-type. Differential Revision: https://reviews.llvm.org/D57173 llvm-svn: 352913
* [opaque pointer types] Pass value type to LoadInst creation.James Y Knight2019-02-011-3/+5
| | | | | | | | | This cleans up all LoadInst creation in LLVM to explicitly pass the value type rather than deriving it from the pointer's element-type. Differential Revision: https://reviews.llvm.org/D57172 llvm-svn: 352911
* [opaque pointer types] Pass function types to CallInst creation.James Y Knight2019-02-011-3/+2
| | | | | | | | | This cleans up all CallInst creation in LLVM to explicitly pass a function type rather than deriving it from the pointer's element-type. Differential Revision: https://reviews.llvm.org/D57170 llvm-svn: 352909
* [opaque pointer types] Add a FunctionCallee wrapper type, and use it.James Y Knight2019-02-011-14/+11
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Recommit r352791 after tweaking DerivedTypes.h slightly, so that gcc doesn't choke on it, hopefully. Original Message: The FunctionCallee type is effectively a {FunctionType*,Value*} pair, and is a useful convenience to enable code to continue passing the result of getOrInsertFunction() through to EmitCall, even once pointer types lose their pointee-type. Then: - update the CallInst/InvokeInst instruction creation functions to take a Callee, - modify getOrInsertFunction to return FunctionCallee, and - update all callers appropriately. One area of particular note is the change to the sanitizer code. Previously, they had been casting the result of `getOrInsertFunction` to a `Function*` via `checkSanitizerInterfaceFunction`, and storing that. That would report an error if someone had already inserted a function declaraction with a mismatching signature. However, in general, LLVM allows for such mismatches, as `getOrInsertFunction` will automatically insert a bitcast if needed. As part of this cleanup, cause the sanitizer code to do the same. (It will call its functions using the expected signature, however they may have been declared.) Finally, in a small number of locations, callers of `getOrInsertFunction` actually were expecting/requiring that a brand new function was being created. In such cases, I've switched them to Function::Create instead. Differential Revision: https://reviews.llvm.org/D57315 llvm-svn: 352827
* Revert "[opaque pointer types] Add a FunctionCallee wrapper type, and use it."James Y Knight2019-01-311-11/+14
| | | | | | | | | This reverts commit f47d6b38c7a61d50db4566b02719de05492dcef1 (r352791). Seems to run into compilation failures with GCC (but not clang, where I tested it). Reverting while I investigate. llvm-svn: 352800
* [opaque pointer types] Add a FunctionCallee wrapper type, and use it.James Y Knight2019-01-311-14/+11
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The FunctionCallee type is effectively a {FunctionType*,Value*} pair, and is a useful convenience to enable code to continue passing the result of getOrInsertFunction() through to EmitCall, even once pointer types lose their pointee-type. Then: - update the CallInst/InvokeInst instruction creation functions to take a Callee, - modify getOrInsertFunction to return FunctionCallee, and - update all callers appropriately. One area of particular note is the change to the sanitizer code. Previously, they had been casting the result of `getOrInsertFunction` to a `Function*` via `checkSanitizerInterfaceFunction`, and storing that. That would report an error if someone had already inserted a function declaraction with a mismatching signature. However, in general, LLVM allows for such mismatches, as `getOrInsertFunction` will automatically insert a bitcast if needed. As part of this cleanup, cause the sanitizer code to do the same. (It will call its functions using the expected signature, however they may have been declared.) Finally, in a small number of locations, callers of `getOrInsertFunction` actually were expecting/requiring that a brand new function was being created. In such cases, I've switched them to Function::Create instead. Differential Revision: https://reviews.llvm.org/D57315 llvm-svn: 352791
* Update the file headers across all of the LLVM projects in the monorepoChandler Carruth2019-01-191-4/+3
| | | | | | | | | | | | | | | | | to reflect the new license. We understand that people may be surprised that we're moving the header entirely to discuss the new license. We checked this carefully with the Foundation's lawyer and we believe this is the correct approach. Essentially, all code in the project is now made available by the LLVM project under our new license, so you will see that the license headers include that license only. Some of our contributors have contributed code under our old license, and accordingly, we have retained a copy of our old license notice in the top-level files in each project and repository. llvm-svn: 351636
* Add NetBSD support in needsRuntimeRegistrationOfSectionRange.Kamil Rytarowski2018-12-151-0/+1
| | | | | | Use linker script magic to get data/cnts/name start/end. llvm-svn: 349277
* [InstrProf] Use atomic profile counter updates for TSanVedant Kumar2018-08-161-6/+16
| | | | | | | | | | | | | | | | | | | | | | | | Thread sanitizer instrumentation fails to skip all loads and stores to profile counters. This can happen if profile counter updates are merged: %.sink = phi i64* ... %pgocount5 = load i64, i64* %.sink %27 = add i64 %pgocount5, 1 %28 = bitcast i64* %.sink to i8* call void @__tsan_write8(i8* %28) store i64 %27, i64* %.sink To suppress TSan diagnostics about racy counter updates, make the counter updates atomic when TSan is enabled. If there's general interest in this mode it can be surfaced as a clang/swift driver option. Testing: check-{llvm,clang,profile} rdar://40477803 Differential Revision: https://reviews.llvm.org/D50867 llvm-svn: 339955
* [InstrProf] Don't register __llvm_profile_runtime_userReid Kleckner2018-07-271-1/+1
| | | | | | | | Refactor some FileCheck prefixes while I'm at it. Fixes PR38340 llvm-svn: 338172
* [profile] Support profiling runtime on FuchsiaPetr Hosek2018-07-251-0/+1
| | | | | | | | | | | | | This ports the profiling runtime on Fuchsia and enables the instrumentation. Unlike on other platforms, Fuchsia doesn't use files to dump the instrumentation data since on Fuchsia, filesystem may not be accessible to the instrumented process. We instead use the data sink to pass the profiling data to the system the same sanitizer runtimes do. Differential Revision: https://reviews.llvm.org/D47208 llvm-svn: 337881
* Rename DEBUG macro to LLVM_DEBUG.Nicola Zaghen2018-05-141-2/+2
| | | | | | | | | | | | | | | | The DEBUG() macro is very generic so it might clash with other projects. The renaming was done as follows: - git grep -l 'DEBUG' | xargs sed -i 's/\bDEBUG\s\?(/LLVM_DEBUG(/g' - git diff -U0 master | ../clang/tools/clang-format/clang-format-diff.py -i -p1 -style LLVM - Manual change to APInt - Manually chage DOCS as regex doesn't match it. In the transition period the DEBUG() macro is still present and aliased to the LLVM_DEBUG() one. Differential Revision: https://reviews.llvm.org/D43624 llvm-svn: 332240
* Fix Layering, move instrumentation transform headers into Instrumentation ↵David Blaikie2018-03-231-1/+1
| | | | | | subdirectory llvm-svn: 328379
* [InstrProfiling] Emit the runtime hook when no counters are loweredVedant Kumar2018-02-281-12/+13
| | | | | | | | | | | | | | | | | | | | | | The API verification tool tapi has difficulty processing frameworks which enable code coverage, but which have no code. The profile lowering pass does not emit the runtime hook in this case because no counters are lowered. While the hook is not needed for program correctness (the profile runtime doesn't have to be linked in), it's needed to allow tapi to validate the exported symbol set of instrumented binaries. It was not possible to add a workaround in tapi for empty binaries due to an architectural issue: tapi generates its expected symbol set before it inspects a binary. Changing that model has a higher cost than simply forcing llvm to always emit the runtime hook. rdar://36076904 Differential Revision: https://reviews.llvm.org/D43794 llvm-svn: 326350
* [InstrProfiling] Don't exit early when an unused intrinsic is foundVedant Kumar2018-01-271-3/+6
| | | | | | This fixes a think-o in r323574. llvm-svn: 323576
* [InstrProfiling] Improve compile time when there is no workVedant Kumar2018-01-261-2/+21
| | | | | | | When there are no uses of profiling intrinsics in a module, and there's no coverage data to lower, InstrProfiling has no work to do. llvm-svn: 323574
* Remove redundant includes from lib/Transforms.Michael Zolotukhin2017-12-131-1/+0
| | | | llvm-svn: 320628
* [PGO] Skip counter promotion for infinite loopsXinliang David Li2017-11-301-0/+3
| | | | | | Differential Revision: http://reviews.llvm.org/D40662 llvm-svn: 319462
* [PGO] Enhance pgo counter promotionXinliang David Li2017-07-121-42/+115
| | | | | | | | | | | | | | | | | | | | | | | This is an incremental change to the promotion feature. There are two problems with the current behavior: 1) loops with multiple exiting blocks are totally disabled 2) a counter update can only be promoted one level up in the loop nest -- which does help much for short trip count inner loops inside a high trip-count outer loops. Due to this limitation, we still saw very large profile count fluctuations from run to run for the affected loops which are usually very hot. This patch adds the support for promotion counters iteratively across the loop nest. It also turns on the promotion for loops with multiple exiting blocks (with a limit). For single-threaded applications, the performance impact is flat on average. For instance, dealII improves, but povray regresses. llvm-svn: 307863
* [PGO] Implementate profile counter regiser promotionXinliang David Li2017-06-251-15/+215
| | | | | | Differential Revision: http://reviews.llvm.org/D34085 llvm-svn: 306231
* [InstrProf] Don't take the address of alwaysinline available_externally ↵Vedant Kumar2017-06-131-1/+11
| | | | | | | | | | | | | | | | | | | | | | | | functions Doing so breaks compilation of the following C program (under -fprofile-instr-generate): __attribute__((always_inline)) inline int foo() { return 0; } int main() { return foo(); } At link time, we fail because taking the address of an available_externally function creates an undefined external reference, which the TU cannot provide. Emitting the function definition into the object file at all appears to be a violation of the langref: "Globals with 'available_externally' linkage are never emitted into the object file corresponding to the LLVM module." Differential Revision: https://reviews.llvm.org/D34134 llvm-svn: 305327
* Sort the remaining #include lines in include/... and lib/....Chandler Carruth2017-06-061-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | I did this a long time ago with a janky python script, but now clang-format has built-in support for this. I fed clang-format every line with a #include and let it re-sort things according to the precise LLVM rules for include ordering baked into clang-format these days. I've reverted a number of files where the results of sorting includes isn't healthy. Either places where we have legacy code relying on particular include ordering (where possible, I'll fix these separately) or where we have particular formatting around #include lines that I didn't want to disturb in this patch. This patch is *entirely* mechanical. If you get merge conflicts or anything, just ignore the changes in this patch and run clang-format over your #include lines in the files. Sorry for any noise here, but it is important to keep these things stable. I was seeing an increasing number of patches with irrelevant re-ordering of #include lines because clang-format was used. This patch at least isolates that churn, makes it easy to skip when resolving conflicts, and gets us to a clean baseline (again). llvm-svn: 304787
* [IR] Abstract away ArgNo+1 attribute indexing as much as possibleReid Kleckner2017-05-031-2/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Summary: Do three things to help with that: - Add AttributeList::FirstArgIndex, which is an enumerator currently set to 1. It allows us to change the indexing scheme with fewer changes. - Add addParamAttr/removeParamAttr. This just shortens addAttribute call sites that would otherwise need to spell out FirstArgIndex. - Remove some attribute-specific getters and setters from Function that take attribute list indices. Most of these were only used from BuildLibCalls, and doesNotAlias was only used to test or set if the return value is malloc-like. I'm happy to split the patch, but I think they are probably easier to review when taken together. This patch should be NFC, but it sets the stage to change the indexing scheme to this, which is more convenient when indexing into an array: 0: func attrs 1: retattrs 2...: arg attrs Reviewers: chandlerc, pete, javed.absar Subscribers: david2050, llvm-commits Differential Revision: https://reviews.llvm.org/D32811 llvm-svn: 302060
* [ProfileData] Unify getInstrProf*SectionName helpersVedant Kumar2017-04-151-30/+10
| | | | | | | | | | | | | | | | | | | | | | This is a version of D32090 that unifies all of the `getInstrProf*SectionName` helper functions. (Note: the build failures which D32090 would have addressed were fixed with r300352.) We should unify these helper functions because they are hard to use in their current form. E.g we recently introduced more helpers to fix section naming for COFF files. This scheme doesn't totally succeed at hiding low-level details about section naming, so we should switch to an API that is easier to maintain. This is not an NFC commit because it fixes llvm-cov's testing support for COFF files (this falls out of the API change naturally). This is an area where we lack tests -- I will see about adding one as a follow up. Testing: check-clang, check-profile, check-llvm. Differential Revision: https://reviews.llvm.org/D32097 llvm-svn: 300381
* [Profile] PE binary coverage bug fixXinliang David Li2017-04-131-10/+10
| | | | | | | | PR/32584 Differential Revision: https://reviews.llvm.org/D32023 llvm-svn: 300277
* [PGO] Memory intrinsic calls optimization based on profiled sizeRong Xu2017-04-041-39/+24
| | | | | | | | | | | | | | | | | | | | | | | | This patch optimizes two memory intrinsic operations: memset and memcpy based on the profiled size of the operation. The high level transformation is like: mem_op(..., size) ==> switch (size) { case s1: mem_op(..., s1); goto merge_bb; case s2: mem_op(..., s2); goto merge_bb; ... default: mem_op(..., size); goto merge_bb; } merge_bb: Differential Revision: http://reviews.llvm.org/D28966 llvm-svn: 299446
* Resubmit r297897: [PGO] Value profile for size of memory intrinsic callsRong Xu2017-03-161-12/+75
| | | | | | | R297897 inadvertently enabled annotation for memop profiling. This new patch fixed it. llvm-svn: 297996
* Revert "[PGO] Value profile for size of memory intrinsic calls"Eric Liu2017-03-161-75/+12
| | | | | | This commit reverts r297897 and r297909. llvm-svn: 297951
* Fix build failure from r297897.Rong Xu2017-03-151-3/+3
| | | | llvm-svn: 297909
* [PGO] Value profile for size of memory intrinsic callsRong Xu2017-03-151-12/+75
| | | | | | | | | This patch adds the value profile support to profile the size parameter of memory intrinsic calls: memcpy, memcmp, and memmov. Differential Revision: http://reviews.llvm.org/D28965 llvm-svn: 297897
* Re-apply "[profiling] Remove dead profile name vars after emitting name data"Vedant Kumar2017-02-141-0/+5
| | | | | | | | | | | | | | | | | | | | | | | | This reverts 295092 (re-applies 295084), with a fix for dangling references from the array of coverage names passed down from frontends. I missed this in my initial testing because I only checked test/Profile, and not test/CoverageMapping as well. Original commit message: The profile name variables passed to counter increment intrinsics are dead after we emit the finalized name data in __llvm_prf_nm. However, we neglect to erase these name variables. This causes huge size increases in the __TEXT,__const section as well as slowdowns when linker dead stripping is disabled. Some affected projects are so massive that they fail to link on Darwin, because only the small code model is supported. Fix the issue by throwing away the name constants as soon as we're done with them. Differential Revision: https://reviews.llvm.org/D29921 llvm-svn: 295099
* Revert "[profiling] Remove dead profile name vars after emitting name data"Vedant Kumar2017-02-141-3/+0
| | | | | | | | This reverts commit r295084. There is a test failure on: http://lab.llvm.org:8011/builders/clang-atom-d525-fedora-rel/builds/2620/ llvm-svn: 295092
* [profiling] Remove dead profile name vars after emitting name dataVedant Kumar2017-02-141-0/+3
| | | | | | | | | | | | | | | | The profile name variables passed to counter increment intrinsics are dead after we emit the finalized name data in __llvm_prf_nm. However, we neglect to erase these name variables. This causes huge size increases in the __TEXT,__const section as well as slowdowns when linker dead stripping is disabled. Some affected projects are so massive that they fail to link on Darwin, because only the small code model is supported. Fix the issue by throwing away the name constants as soon as we're done with them. Differential Revision: https://reviews.llvm.org/D29921 llvm-svn: 295084
* [Target, Transforms] Fix some Clang-tidy modernize and Include What You Use ↵Eugene Zelenko2017-01-181-15/+38
| | | | | | warnings; other minor fixes (NFC). llvm-svn: 292320
* Resubmit "[PGO] Turn off comdat renaming in IR PGO by default"Rong Xu2017-01-111-1/+15
| | | | | | This patch resubmits the changes in r291588. llvm-svn: 291696
* Revert "[PGO] Turn off comdat renaming in IR PGO by default"Rong Xu2017-01-101-15/+1
| | | | | | | This patch reverts r291588: [PGO] Turn off comdat renaming in IR PGO by default, as we are seeing some hash mismatches in our internal tests. llvm-svn: 291621
* [PGO] Turn off comdat renaming in IR PGO by defaultRong Xu2017-01-101-1/+15
| | | | | | | | | | | | | | | | | | | | | | | Summary: In IR PGO we append the function hash to comdat functions to avoid the potential hash mismatch. This turns out not legal in some cases: if the comdat function is address-taken and used in comparison. Renaming changes the semantic. This patch turns off comdat renaming by default. To alleviate the hash mismatch issue, we now rename the profile variable for comdat functions. Profile allows co-existing multiple versions of profiles with different hash value. The inlined copy will always has the correct profile counter. The out-of-line copy might not have the correct count. But we will not have the bogus mismatch warning. Reviewers: davidxl Subscribers: llvm-commits, xur Differential Revision: https://reviews.llvm.org/D28416 llvm-svn: 291588
* [InstrProfiling] Mark __llvm_profile_instrument_target last parameter as i32 ↵Marcin Koscielnicki2016-11-211-11/+30
| | | | | | | | | | | | | | | | | zeroext if appropriate. On some architectures (s390x, ppc64, sparc64, mips), C-level int is passed as i32 signext instead of plain i32. Likewise, unsigned int may be passed as i32, i32 signext, or i32 zeroext depending on the platform. Mark __llvm_profile_instrument_target properly (its last parameter is unsigned int). This (together with the clang change) makes compiler-rt profile testsuite pass on s390x. Differential Revision: http://reviews.llvm.org/D21736 llvm-svn: 287534
* Utility functions for appending to llvm.used/llvm.compiler.used.Evgeniy Stepanov2016-10-251-25/+2
| | | | llvm-svn: 285143
* Use StringRef in Pass/PassManager APIs (NFC)Mehdi Amini2016-10-011-1/+1
| | | | llvm-svn: 283004
* [Profile] code refactoring: make getStep a method in base classXinliang David Li2016-09-201-9/+1
| | | | llvm-svn: 282002
* [Profile] Implement select instruction instrumentation in IR PGOXinliang David Li2016-09-181-2/+18
| | | | | | Differential Revision: http://reviews.llvm.org/D23727 llvm-svn: 281858
OpenPOWER on IntegriCloud