summaryrefslogtreecommitdiffstats
path: root/llvm/tools/opt
Commit message (Collapse)AuthorAgeFilesLines
...
* [NewPM] teach -passes= to emit meaningful error messagesFedor Sergeev2018-10-171-38/+52
| | | | | | | | | | | | | | All the PassBuilder::parse interfaces now return descriptive StringError instead of a plain bool. It allows to make -passes/aa-pipeline parsing errors context-specific and thus less confusing. TODO: ideally we should also make suggestions for misspelled pass names, but that requires some extensions to PassBuilder. Reviewed By: philip.pfaffe, chandlerc Differential Revision: https://reviews.llvm.org/D53246 llvm-svn: 344685
* Revert "[NewPM] teach -passes= to emit meaningful error messages"Fedor Sergeev2018-10-151-48/+36
| | | | | | This reverts r344519 due to failures in pipeline-parsing test. llvm-svn: 344524
* [NewPM] teach -passes= to emit meaningful error messagesFedor Sergeev2018-10-151-36/+48
| | | | | | | | | | | | | | | Summary: All the PassBuilder::parse interfaces now return descriptive StringError instead of a plain bool. It allows to make -passes/aa-pipeline parsing errors context-specific and thus less confusing. TODO: ideally we should also make suggestions for misspelled pass names, but that requires some extensions to PassBuilder. Reviewed By: philip.pfaffe, chandlerc Differential Revision: https://reviews.llvm.org/D53246 llvm-svn: 344519
* Add a flag to remap manglings when reading profile data information.Richard Smith2018-10-101-4/+8
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This can be used to preserve profiling information across codebase changes that have widespread impact on mangled names, but across which most profiling data should still be usable. For example, when switching from libstdc++ to libc++, or from the old libstdc++ ABI to the new ABI, or even from a 32-bit to a 64-bit build. The user can provide a remapping file specifying parts of mangled names that should be treated as equivalent (eg, std::__1 should be treated as equivalent to std::__cxx11), and profile data will be treated as applying to a particular function if its name is equivalent to the name of a function in the profile data under the provided equivalences. See the documentation change for a description of how this is configured. Remapping is supported for both sample-based profiling and instruction profiling. We do not support remapping indirect branch target information, but all other profile data should be remapped appropriately. Support is only added for the new pass manager. If someone wants to also add support for this for the old pass manager, doing so should be straightforward. This is the LLVM side of Clang r344199. Reviewers: davidxl, tejohnson, dlj, erik.pilkington Subscribers: mehdi_amini, steven_wu, dexonsmith, llvm-commits Differential Revision: https://reviews.llvm.org/D51249 llvm-svn: 344200
* [New PM][PassInstrumentation] IR printing support for New Pass ManagerFedor Sergeev2018-09-241-2/+7
| | | | | | | | | | | | | | | | Implementing -print-before-all/-print-after-all/-filter-print-func support through PassInstrumentation callbacks. - PrintIR routines implement printing callbacks. - StandardInstrumentations class provides a central place to manage all the "standard" in-tree pass instrumentations. Currently it registers PrintIR callbacks. Reviewers: chandlerc, paquette, philip.pfaffe Differential Revision: https://reviews.llvm.org/D50923 llvm-svn: 342896
* [Debugify] Export per-pass debug info loss statisticsVedant Kumar2018-07-243-16/+114
| | | | | | | | | | | | Add a -debugify-export option to opt. This exports per-pass `debugify` loss statistics to a file in CSV format. For some interesting numbers on debug value loss during an -O2 build of the sqlite3 amalgamation, see the review thread. Differential Revision: https://reviews.llvm.org/D49003 llvm-svn: 337787
* [Debugify] Move interface definitions to a header, NFCVedant Kumar2018-07-245-21/+42
| | | | | | | | This is a minor cleanup in preparation for a change to export DI statistics from -check-debugify. To do that, it would be cleaner to have a dedicated header for the debugify interface. llvm-svn: 337786
* [Debugify] Allow unsigned values narrower than their variablesVedant Kumar2018-07-061-2/+9
| | | | | | | | Suppress the diagnostic for mis-sized dbg.values when a value operand is narrower than the unsigned variable it describes. Assume that a debugger would implicitly zero-extend these values. llvm-svn: 336452
* [Debugify] Do not report line 0 locations as errorsVedant Kumar2018-06-281-5/+7
| | | | | | | | | | The checking logic should not treat artificial locations as being somehow problematic. Producing these locations can be the desired behavior of some passes. See llvm.org/PR37961. llvm-svn: 335897
* [Debugify] Handle failure to get fragment size when checking dbg.valuesVedant Kumar2018-06-271-4/+7
| | | | | | | | | | It's not possible to get the fragment size of some dbg.values. Teach the mis-sized dbg.value diagnostic to detect this scenario and bail out. Tested with: $ find test/Transforms -print -exec opt -debugify-each -instcombine {} \; llvm-svn: 335695
* [Debugify] Diagnose mis-sized dbg.valuesVedant Kumar2018-06-261-4/+40
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Report an error in -check-debugify when the size of a dbg.value operand doesn't match up with the size of the variable it describes. Eventually this check should be moved into the IR verifier. For the moment, it's useful to include the check in -check-debugify as a means of catching regressions and finding existing bugs. Here are some instances of bugs the new check finds in the -O2 pipeline (all in InstCombine): 1) A float is used where a double is expected: ERROR: dbg.value operand has size 32, but its variable has size 64: call void @llvm.dbg.value(metadata float %expf, metadata !12, metadata !DIExpression()), !dbg !15 2) An i8 is used where an i32 is expected: ERROR: dbg.value operand has size 8, but its variable has size 32: call void @llvm.dbg.value(metadata i8 %t4, metadata !14, metadata !DIExpression()), !dbg !24 3) A <4 x i32> is used where something twice as large is expected (perhaps a <4 x i64>, I haven't double-checked): ERROR: dbg.value operand has size 128, but its variable has size 256: call void @llvm.dbg.value(metadata <4 x i32> %4, metadata !40, metadata !DIExpression()), !dbg !95 Differential Revision: https://reviews.llvm.org/D48408 llvm-svn: 335682
* [Debugify] Don't treat missing dbg.values as an error (PR37942)Vedant Kumar2018-06-261-2/+1
| | | | | | | | | When checking the debug info in a module, don't treat a missing dbg.value as an error. The dbg.value may simply have been DCE'd, in which case the debugger has enough information to display the variable as <optimized out>. llvm-svn: 335647
* [Debugify] Move debug value intrinsics closer to their operand defsVedant Kumar2018-06-061-18/+24
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Before this patch, debugify would insert debug value intrinsics before the terminating instruction in a block. This had the advantage of being simple, but was a bit too simple/unrealistic. This patch teaches debugify to insert debug values immediately after their operand defs. This enables better testing of the compiler. For example, with this patch, `opt -debugify-each` is able to identify a vectorizer DI-invariance bug fixed in llvm.org/PR32761. In this bug, the vectorizer produced different output with/without debug info present. Reverting Davide's bugfix locally, I see: $ ~/scripts/opt-check-dbg-invar.sh ./bin/opt \ .../SLPVectorizer/AArch64/spillcost-di.ll -slp-vectorizer Comparing: -slp-vectorizer .../SLPVectorizer/AArch64/spillcost-di.ll Baseline: /var/folders/j8/t4w0bp8j6x1g6fpghkcb4sjm0000gp/T/tmp.iYYeL1kf With DI : /var/folders/j8/t4w0bp8j6x1g6fpghkcb4sjm0000gp/T/tmp.sQtQSeet 9,11c9,11 < %5 = getelementptr inbounds %0, %0* %2, i64 %0, i32 1 < %6 = bitcast i64* %4 to <2 x i64>* < %7 = load <2 x i64>, <2 x i64>* %6, align 8, !tbaa !0 --- > %5 = load i64, i64* %4, align 8, !tbaa !0 > %6 = getelementptr inbounds %0, %0* %2, i64 %0, i32 1 > %7 = load i64, i64* %6, align 8, !tbaa !5 12a13 > store i64 %5, i64* %8, align 8, !tbaa !0 14,15c15 < %10 = bitcast i64* %8 to <2 x i64>* < store <2 x i64> %7, <2 x i64>* %10, align 8, !tbaa !0 --- > store i64 %7, i64* %9, align 8, !tbaa !5 :: Found a test case ^ Running this over the *.ll files in tree, I found four additional examples which compile differently with/without DI present. I plan on filing bugs for these. llvm-svn: 334118
* [Debugify] Add a quiet mode to suppress warningsVedant Kumar2018-06-061-15/+16
| | | | | | | Suppressing warning output and module dumps significantly speeds up fuzzing with `opt -debugify-each`. llvm-svn: 334117
* [opt] Introduce -strip-named-metadataVedant Kumar2018-06-051-8/+12
| | | | | | | This renames and generalizes -strip-module-flags to erase all named metadata from a module. This makes it easier to diff IR. llvm-svn: 333977
* [Debugify] Don't insert debug values after terminating deoptsVedant Kumar2018-06-051-5/+14
| | | | | | | As is the case with musttail calls, the IR does not allow for instructions inserted after a terminating deopt. llvm-svn: 333976
* Apply clang-format on a file, NFCVedant Kumar2018-06-051-8/+5
| | | | llvm-svn: 333975
* [Debugify] Preserve analyses in -check-debugifyVedant Kumar2018-06-041-0/+4
| | | | | | | | | | | | | The -check-debugify pass should preserve all analyses. Otherwise, it may invalidate an optional analysis and inadvertently alter codegen. The test case is reduced from deopt-bundle.ll. The result of `opt -O1` on this file would differ when -debugify-each was toggled. That happened because CheckDebugify failed to preserve GlobalsAA. Thanks to Davide Italiano for his help chasing this down! llvm-svn: 333959
* [Debugify] Add debug intrinsics before terminating musttail callsVedant Kumar2018-06-041-9/+13
| | | | | | | | | | | After r333856, opt -debugify would just stop emitting debug value intrinsics after encountering a musttail call. This wasn't sufficient to avoid verifier failures. Debug value intrinicss for all instructions preceding a musttail call must also be emitted before the musttail call. llvm-svn: 333866
* [Debugify] Don't apply DI before the bitcode writer passVedant Kumar2018-06-041-2/+2
| | | | | | | | | | | | Applying synthetic debug info before the bitcode writer pass has no testing-related purpose. This commit prevents that from happening. It also adds tests which check that IR produced with/without -debugify-each enabled is identical after stripping. This makes it possible to check that individual passes (or full pipelines) are invariant to debug info. llvm-svn: 333861
* [opt] Add a -strip-module-flags optionVedant Kumar2018-06-041-0/+8
| | | | | | | | | | | | | | | The -strip-module-flags option strips llvm.module.flags metadata from a module at the beginning of the opt pipeline. This will be used to test whether the output of a pass is debug info (DI) invariant. E.g, after applying synthetic debug info to a test case, we'd like to strip out all DI-related metadata and check that the final IR is identical to a baseline file without any DI applied, to check that optimizations aren't inhibited by debug info. llvm-svn: 333860
* Reformat overflowing lines, NFCVedant Kumar2018-06-041-8/+10
| | | | llvm-svn: 333859
* [Debugify] Skip dbg.value placement for EH pads, musttailVedant Kumar2018-06-031-0/+9
| | | | | | | Placing meta-instructions into EH pads breaks certain IR invariants, as does placing instructions after a musttail call. llvm-svn: 333856
* [WebAssembly] Add Wasm exception handling prepare passHeejin Ahn2018-05-311-0/+1
| | | | | | | | | | | | | | | | Summary: This adds a pass that transforms a program to be prepared for Wasm exception handling. This is using Windows EH instructions and based on the previous Wasm EH proposal. (https://github.com/WebAssembly/exception-handling/blob/master/proposals/Exceptions.md) Reviewers: dschuff, majnemer Subscribers: jfb, mgorny, sbc100, jgravelle-google, JDevlieghere, sunfish, llvm-commits Differential Revision: https://reviews.llvm.org/D43746 llvm-svn: 333696
* [Debugify] Set a DI version module flag for llc compatibilityVedant Kumar2018-05-241-2/+10
| | | | | | | Setting the "Debug Info Version" module flag makes it possible to pipe synthetic debug info into llc, which is useful for testing backends. llvm-svn: 333237
* [Debugify] Avoid printing unnecessary square braces, NFCVedant Kumar2018-05-241-2/+4
| | | | llvm-svn: 333236
* [Debugify] Print the output to stderrAnastasis Grammenos2018-05-171-9/+9
| | | | | | | | | | Currently debugify prints it's output to stdout, with this patch all the output generated goes to stderr. This change lets us use debugify without taking away the ability to pipe the output to other llvm tools. llvm-svn: 332642
* [Debugfiy] Print the pass name next to the resultAnastasis Grammenos2018-05-153-15/+26
| | | | | | | | CheckDebugify now prints the pass name right next to the result of the check. Differential Revision: https://reviews.llvm.org/D46908 llvm-svn: 332416
* [Debugify] Add -debugify-each for testing each pass in a pipelineVedant Kumar2018-05-153-47/+171
| | | | | | | | | | | | | | | | This adds a -debugify-each mode to opt which, when enabled, wraps each {Module,Function}Pass in a pipeline with logic to add, check, and strip synthetic debug info for testing purposes. This mode can be used to test complex pipelines for debug info bugs, or to collect statistics about the number of debug values & locations lost throughout various stages of a pipeline. Patch by Son Tuan Vu! Differential Revision: https://reviews.llvm.org/D46525 llvm-svn: 332312
* Remove \brief commands from doxygen comments.Adrian Prantl2018-05-015-5/+5
| | | | | | | | | | | | | | | | We've been running doxygen with the autobrief option for a couple of years now. This makes the \brief markers into our comments redundant. Since they are a visual distraction and we don't want to encourage more \brief markers in new code either, this patch removes them all. Patch produced by for i in $(git grep -l '\\brief'); do perl -pi -e 's/\\brief //g' $i & done Differential Revision: https://reviews.llvm.org/D46290 llvm-svn: 331272
* IWYU for llvm-config.h in llvm, additions.Nico Weber2018-04-302-1/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | See r331124 for how I made a list of files missing the include. I then ran this Python script: for f in open('filelist.txt'): f = f.strip() fl = open(f).readlines() found = False for i in xrange(len(fl)): p = '#include "llvm/' if not fl[i].startswith(p): continue if fl[i][len(p):] > 'Config': fl.insert(i, '#include "llvm/Config/llvm-config.h"\n') found = True break if not found: print 'not found', f else: open(f, 'w').write(''.join(fl)) and then looked through everything with `svn diff | diffstat -l | xargs -n 1000 gvim -p` and tried to fix include ordering and whatnot. No intended behavior change. llvm-svn: 331184
* [AggressiveInstCombine] Add library initializer routine for ↵Craig Topper2018-04-241-1/+1
| | | | | | | | AggressiveInstCombine library. Use it in bugpoint and llvm-opt-fuzzer to match regular InstCombine. This should make aggressive instcombine usable with these tools. llvm-svn: 330663
* [DebugInfo][OPT] NFC follow-up on "Fixing a couple of DI duplication bugs of ↵Roman Tereshin2018-04-131-9/+9
| | | | | | CloneModule" llvm-svn: 330070
* [DebugInfo][OPT] Fixing a couple of DI duplication bugs of CloneModuleRoman Tereshin2018-04-131-9/+11
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | As demonstrated by the regression tests added in this patch, the following cases are valid cases: 1. A Function with no DISubprogram attached, but various debug info related to its instructions, coming, for instance, from an inlined function, also defined somewhere else in the same module; 2. ... or coming exclusively from the functions inlined and eliminated from the module entirely. The ValueMap shared between CloneFunctionInto calls within CloneModule needs to contain identity mappings for all of the DISubprogram's to prevent them from being duplicated by MapMetadata / RemapInstruction calls, this is achieved via DebugInfoFinder collecting all the DISubprogram's. However, CloneFunctionInto was missing calls into DebugInfoFinder for functions w/o DISubprogram's attached, but still referring DISubprogram's from within (case 1). This patch fixes that. The fix above, however, exposes another issue: if a module contains a DISubprogram referenced only indirectly from other debug info metadata, but not attached to any Function defined within the module (case 2), cloning such a module causes a DICompileUnit duplication: it will be moved in indirecty via a DISubprogram by DebugInfoFinder first (because of the first bug fix described above), without being self-mapped within the shared ValueMap, and then will be copied during named metadata cloning. So this patch makes sure DebugInfoFinder visits DICompileUnit's referenced from DISubprogram's as it goes w/o re-processing llvm.dbg.cu list over and over again for every function cloned, and makes sure that CloneFunctionInto self-maps DICompileUnit's referenced from the entire function, not just its own DISubprogram attached that may also be missing. The most convenient way of tesing CloneModule I found is to rely on CloneModule call from `opt -run-twice`, instead of writing tedious unit tests. That feature has a couple of properties that makes it hard to use for this purpose though: 1. CloneModule doesn't copy source filename, making `opt -run-twice` report it as a difference. 2. `opt -run-twice` does the second run on the original module, not its clone, making the result of cloning completely invisible in opt's actual output with and without `-run-twice` both, which directly contradicts `opt -run-twice`s own error message. This patch fixes this as well. Reviewed By: aprantl Reviewers: loladiro, GorNishanov, espindola, echristo, dexonsmith Subscribers: vsk, debug-info, JDevlieghere, llvm-commits Differential Revision: https://reviews.llvm.org/D45593 llvm-svn: 330069
* Define InitLLVM to do common initialization all at once.Rui Ueyama2018-04-131-6/+2
| | | | | | | | | | | We have a few functions that virtually all command wants to run on process startup/shutdown. This patch adds InitLLVM class to do that all at once, so that we don't need to copy-n-paste boilerplate code to each llvm command's main() function. Differential Revision: https://reviews.llvm.org/D45602 llvm-svn: 330046
* Rename *CommandFlags.def to *CommandFlags.incDavid Blaikie2018-04-111-1/+1
| | | | | | | | These aren't the .def style files used in LLVM that require a macro defined before their inclusion - they're just basic non-modular includes to stamp out command line flag variables. llvm-svn: 329840
* Re-land r329273: [Plugins] Add a slim plugin API to work together with the ↵Philip Pfaffe2018-04-051-0/+17
| | | | | | | | | new PM Fix unittest: Do not link LLVM into the test plugin. Additionally, remove an unrelated change that slipped in in r329273. llvm-svn: 329293
* Revert "[Plugins] Add a slim plugin API to work together with the new PM"Philip Pfaffe2018-04-051-17/+0
| | | | | | This reverts commit ecf3ba1ab45edb1b0fadce716a7facf50dca4fbb/r329273. llvm-svn: 329276
* [Plugins] Add a slim plugin API to work together with the new PMPhilip Pfaffe2018-04-051-0/+17
| | | | | | | | | | | | | | | | | | | | | | Summary: Add a new plugin API. This closes the gap between pass registration and out-of-tree passes for the new PassManager. Unlike with the existing API, interaction with a plugin is always initiated from the tools perspective. I.e., when a plugin is loaded, it resolves and calls a well-known symbol `llvmGetPassPluginInfo` to obtain details about the plugin. The fundamental motivation is to get rid of as many global constructors as possible. The API exposed by the plugin info is kept intentionally minimal. Reviewers: chandlerc Reviewed By: chandlerc Subscribers: bollu, grosser, lksbhm, mgorny, llvm-commits Differential Revision: https://reviews.llvm.org/D35258 llvm-svn: 329273
* [Debugify] Don't check functions which were skippedVedant Kumar2018-02-151-1/+11
| | | | | | | If no debug info was applied to a function, its debug info shouldn't be checked (it doesn't have any :). llvm-svn: 325297
* [opt] Port the debugify passes to the new pass managerVedant Kumar2018-02-155-7/+53
| | | | llvm-svn: 325294
* Pass a module reference to CloneModule.Rafael Espindola2018-02-141-4/+4
| | | | | | | It can never be null and most callers were already using references or std::unique_ptr. llvm-svn: 325160
* [Debugify] Avoid verifier failure on non-definition subprogramsVedant Kumar2018-02-131-2/+2
| | | | | | | | | | If a function doesn't have an exact definition, don't apply debugify metadata as it triggers a DIVerifier failure. The issue is that it's invalid to have DILocations inside a DISubprogram which isn't a definition ("scope points into the type hierarchy!"). llvm-svn: 325036
* LLParser: add an argument for overriding data layout and do not check alloca ↵Yaxun Liu2018-01-301-7/+5
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | addr space Sometimes users do not specify data layout in LLVM assembly and let llc set the data layout by target triple after loading the LLVM assembly. Currently the parser checks alloca address space no matter whether the LLVM assembly contains data layout definition, which causes false alarm since the default data layout does not contain the correct alloca address space. The parser also calls verifier to check debug info and updating invalid debug info. Currently there is no way to let the verifier to check debug info only. If the verifier finds non-debug-info issues the parser will fail. For llc, the fix is to remove the check of alloca addr space in the parser and disable updating debug info, and defer the updating of debug info and verification to be after setting data layout of the IR by target. For other llvm tools, since they do not override data layout by target but instead can override data layout by a command line option, an argument for overriding data layout is added to the parser. In cases where data layout overriding is necessary for the parser, the data layout can be provided by command line. Differential Revision: https://reviews.llvm.org/D41832 llvm-svn: 323826
* Another try to commit 323321 (aggressive instruction combine).Amjad Aboud2018-01-252-0/+2
| | | | llvm-svn: 323416
* Reverted 323321.Amjad Aboud2018-01-242-2/+0
| | | | llvm-svn: 323326
* [InstCombine] Introducing Aggressive Instruction Combine pass ↵Amjad Aboud2018-01-242-0/+2
| | | | | | | | | | | | | | | | | | (-aggressive-instcombine). Combine expression patterns to form expressions with fewer, simple instructions. This pass does not modify the CFG. For example, this pass reduce width of expressions post-dominated by TruncInst into smaller width when applicable. It differs from instcombine pass in that it contains pattern optimization that requires higher complexity than the O(1), thus, it should run fewer times than instcombine pass. Differential Revision: https://reviews.llvm.org/D38313 llvm-svn: 323321
* [Debugify] Add a mode to opt to enable faster testingVedant Kumar2018-01-232-0/+18
| | | | | | | | | | | | | | | | | | Opt's "-enable-debugify" mode adds an instance of Debugify at the beginning of the pass pipeline, and an instance of CheckDebugify at the end. You can enable this mode with lit using: -Dopt="opt -enable-debugify". Note that running test suites in this mode will result in many failures due to strict FileCheck commands, etc. It can be more useful to look for assertion failures which arise only when Debugify is enabled, e.g to prove that we have (or do not have) test coverage for some code path with debug info present. Differential Revision: https://reviews.llvm.org/D41793 llvm-svn: 323256
* NewPM: Add an extension point for the start of the pipeline.David Blaikie2018-01-231-0/+11
| | | | | | | | | | This applies to most pipelines except the LTO and ThinLTO backend actions - it is for use at the beginning of the overall pipeline. This extension point will be used to add the GCOV pass when enabled in Clang. llvm-svn: 323166
* Introduce the "retpoline" x86 mitigation technique for variant #2 of the ↵Chandler Carruth2018-01-221-0/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | speculative execution vulnerabilities disclosed today, specifically identified by CVE-2017-5715, "Branch Target Injection", and is one of the two halves to Spectre.. Summary: First, we need to explain the core of the vulnerability. Note that this is a very incomplete description, please see the Project Zero blog post for details: https://googleprojectzero.blogspot.com/2018/01/reading-privileged-memory-with-side.html The basis for branch target injection is to direct speculative execution of the processor to some "gadget" of executable code by poisoning the prediction of indirect branches with the address of that gadget. The gadget in turn contains an operation that provides a side channel for reading data. Most commonly, this will look like a load of secret data followed by a branch on the loaded value and then a load of some predictable cache line. The attacker then uses timing of the processors cache to determine which direction the branch took *in the speculative execution*, and in turn what one bit of the loaded value was. Due to the nature of these timing side channels and the branch predictor on Intel processors, this allows an attacker to leak data only accessible to a privileged domain (like the kernel) back into an unprivileged domain. The goal is simple: avoid generating code which contains an indirect branch that could have its prediction poisoned by an attacker. In many cases, the compiler can simply use directed conditional branches and a small search tree. LLVM already has support for lowering switches in this way and the first step of this patch is to disable jump-table lowering of switches and introduce a pass to rewrite explicit indirectbr sequences into a switch over integers. However, there is no fully general alternative to indirect calls. We introduce a new construct we call a "retpoline" to implement indirect calls in a non-speculatable way. It can be thought of loosely as a trampoline for indirect calls which uses the RET instruction on x86. Further, we arrange for a specific call->ret sequence which ensures the processor predicts the return to go to a controlled, known location. The retpoline then "smashes" the return address pushed onto the stack by the call with the desired target of the original indirect call. The result is a predicted return to the next instruction after a call (which can be used to trap speculative execution within an infinite loop) and an actual indirect branch to an arbitrary address. On 64-bit x86 ABIs, this is especially easily done in the compiler by using a guaranteed scratch register to pass the target into this device. For 32-bit ABIs there isn't a guaranteed scratch register and so several different retpoline variants are introduced to use a scratch register if one is available in the calling convention and to otherwise use direct stack push/pop sequences to pass the target address. This "retpoline" mitigation is fully described in the following blog post: https://support.google.com/faqs/answer/7625886 We also support a target feature that disables emission of the retpoline thunk by the compiler to allow for custom thunks if users want them. These are particularly useful in environments like kernels that routinely do hot-patching on boot and want to hot-patch their thunk to different code sequences. They can write this custom thunk and use `-mretpoline-external-thunk` *in addition* to `-mretpoline`. In this case, on x86-64 thu thunk names must be: ``` __llvm_external_retpoline_r11 ``` or on 32-bit: ``` __llvm_external_retpoline_eax __llvm_external_retpoline_ecx __llvm_external_retpoline_edx __llvm_external_retpoline_push ``` And the target of the retpoline is passed in the named register, or in the case of the `push` suffix on the top of the stack via a `pushl` instruction. There is one other important source of indirect branches in x86 ELF binaries: the PLT. These patches also include support for LLD to generate PLT entries that perform a retpoline-style indirection. The only other indirect branches remaining that we are aware of are from precompiled runtimes (such as crt0.o and similar). The ones we have found are not really attackable, and so we have not focused on them here, but eventually these runtimes should also be replicated for retpoline-ed configurations for completeness. For kernels or other freestanding or fully static executables, the compiler switch `-mretpoline` is sufficient to fully mitigate this particular attack. For dynamic executables, you must compile *all* libraries with `-mretpoline` and additionally link the dynamic executable and all shared libraries with LLD and pass `-z retpolineplt` (or use similar functionality from some other linker). We strongly recommend also using `-z now` as non-lazy binding allows the retpoline-mitigated PLT to be substantially smaller. When manually apply similar transformations to `-mretpoline` to the Linux kernel we observed very small performance hits to applications running typical workloads, and relatively minor hits (approximately 2%) even for extremely syscall-heavy applications. This is largely due to the small number of indirect branches that occur in performance sensitive paths of the kernel. When using these patches on statically linked applications, especially C++ applications, you should expect to see a much more dramatic performance hit. For microbenchmarks that are switch, indirect-, or virtual-call heavy we have seen overheads ranging from 10% to 50%. However, real-world workloads exhibit substantially lower performance impact. Notably, techniques such as PGO and ThinLTO dramatically reduce the impact of hot indirect calls (by speculatively promoting them to direct calls) and allow optimized search trees to be used to lower switches. If you need to deploy these techniques in C++ applications, we *strongly* recommend that you ensure all hot call targets are statically linked (avoiding PLT indirection) and use both PGO and ThinLTO. Well tuned servers using all of these techniques saw 5% - 10% overhead from the use of retpoline. We will add detailed documentation covering these components in subsequent patches, but wanted to make the core functionality available as soon as possible. Happy for more code review, but we'd really like to get these patches landed and backported ASAP for obvious reasons. We're planning to backport this to both 6.0 and 5.0 release streams and get a 5.0 release with just this cherry picked ASAP for distros and vendors. This patch is the work of a number of people over the past month: Eric, Reid, Rui, and myself. I'm mailing it out as a single commit due to the time sensitive nature of landing this and the need to backport it. Huge thanks to everyone who helped out here, and everyone at Intel who helped out in discussions about how to craft this. Also, credit goes to Paul Turner (at Google, but not an LLVM contributor) for much of the underlying retpoline design. Reviewers: echristo, rnk, ruiu, craig.topper, DavidKreitzer Subscribers: sanjoy, emaste, mcrosier, mgorny, mehdi_amini, hiraditya, llvm-commits Differential Revision: https://reviews.llvm.org/D41723 llvm-svn: 323155
OpenPOWER on IntegriCloud