summaryrefslogtreecommitdiffstats
path: root/clang/lib/Basic/Targets/X86.h
Commit message (Collapse)AuthorAgeFilesLines
* [Sema][X86] Consider target attribute into the checks in validateOutputSize ↵Craig Topper2019-12-231-6/+9
| | | | | | | | | | | | | and validateInputSize. The validateOutputSize and validateInputSize need to check whether AVX or AVX512 are enabled. But this can be affected by the target attribute so we need to factor that in. This patch moves some of the code from CodeGen to create an appropriate feature map that we can pass to the function. Differential Revision: https://reviews.llvm.org/D68627
* Add support for the MS qualifiers __ptr32, __ptr64, __sptr, __uptr.Amy Huang2019-12-181-0/+29
| | | | | | | | | | | | | | | | | | | | Summary: This adds parsing of the qualifiers __ptr32, __ptr64, __sptr, and __uptr and lowers them to the corresponding address space pointer for 32-bit and 64-bit pointers. (32/64-bit pointers added in https://reviews.llvm.org/D69639) A large part of this patch is making these pointers ignore the address space when doing things like overloading and casting. https://bugs.llvm.org/show_bug.cgi?id=42359 Reviewers: rnk, rsmith Subscribers: jholewinski, jvesely, nhaehnle, cfe-commits Tags: #clang Differential Revision: https://reviews.llvm.org/D71039
* Revert "[Sema][X86] Consider target attribute into the checks in ↵Reid Kleckner2019-12-061-9/+6
| | | | | | | | | validateOutputSize and validateInputSize." This reverts commit e1578fd2b79fe5af5f80c0c166a8abd0f816c022. It introduces a dependency on Attr.h which I am removing from ASTContext.h.
* [Sema][X86] Consider target attribute into the checks in validateOutputSize ↵Craig Topper2019-12-061-6/+9
| | | | | | | | | | | | | | | | and validateInputSize. The validateOutputSize and validateInputSize need to check whether AVX or AVX512 are enabled. But this can be affected by the target attribute so we need to factor that in. This patch copies some of the code from CodeGen to create an appropriate feature map that we can pass to the function. Probably need some refactoring here to share more code with Codegen. Is there a good place to do that? Also need to support the cpu_specific attribute as well. Differential Revision: https://reviews.llvm.org/D68627
* Reland "Change the X86 datalayout to add three address spacesAmy Huang2019-09-101-12/+20
| | | | | | | | | | for 32 bit signed, 32 bit unsigned, and 64 bit pointers." This reverts 57076d3199fc2b0af4a3736b7749dd5462cacda5. Original review at https://reviews.llvm.org/D64931. Review for added fix at https://reviews.llvm.org/D66843. llvm-svn: 371568
* [X86] Remove what little support we had for MPXCraig Topper2019-08-291-1/+0
| | | | | | | | | | | | | | | -Deprecate -mmpx and -mno-mpx command line options -Remove CPUID detection of mpx for -march=native -Remove MPX from all CPUs -Remove MPX preprocessor define I've left the "mpx" string in the backend so we don't fail on old IR, but its not connected to anything. gcc has also deprecated these command line options. https://www.phoronix.com/scan.php?page=news_item&px=GCC-Patch-To-Drop-MPX Differential Revision: https://reviews.llvm.org/D66669 llvm-svn: 370393
* Revert "Change the X86 datalayout to add three address spaces for 32 bit ↵Vlad Tsyrklevich2019-08-281-20/+12
| | | | | | | | | signed," This reverts commit r370083 because it caused check-lld failures on sanitizer-x86_64-linux-fast. llvm-svn: 370142
* Change the X86 datalayout to add three address spaces for 32 bit signed,Amy Huang2019-08-271-12/+20
| | | | | | 32 bit unsigned, and 64 bit pointers. llvm-svn: 370083
* Remove CallingConvMethodTypeErich Keane2019-07-251-3/+3
| | | | | | | | | This seems to be an old vestage of a previous implementation of getting the default calling convention, and everything is now using CXXABI/ASTContext's getDefaultCallingConvention. Remove it, since it isn't doing anything. llvm-svn: 367039
* [X86][PowerPC] Support -mlong-double-128Fangrui Song2019-07-121-2/+4
| | | | | | | | | | | | | | | This patch makes the driver option -mlong-double-128 available for X86 and PowerPC. The CC1 option -mlong-double-128 is available on all targets for users to test on unsupported targets. On PowerPC, -mlong-double-128 uses the IBM extended double format because we don't support -mabi=ieeelongdouble yet (D64283). Reviewed By: rnk Differential Revision: https://reviews.llvm.org/D64277 llvm-svn: 365866
* De-templatize non-dependent VS macro logic, NFCReid Kleckner2019-07-091-2/+0
| | | | | | | These macro definitions don't depend on the template parameter, so they don't need to be part of the template. Move them to a .cpp file. llvm-svn: 365556
* [ItaniumMangle] Refactor long double/__float128 mangling and fix the mangled ↵Fangrui Song2019-07-091-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | code In gcc PowerPC, long double has 3 mangling schemes: -mlong-double-64: `e` -mlong-double-128 -mabi=ibmlongdouble: `g` -mlong-double-128 -mabi=ieeelongdouble: `u9__ieee128` (gcc <= 8.1: `U10__float128`) The current useFloat128ManglingForLongDouble() bisection is not suitable when we support -mlong-double-128 in clang (D64277). Replace useFloat128ManglingForLongDouble() with getLongDoubleMangling() and getFloat128Mangling() to allow 3 mangling schemes. I also deleted the `getTriple().isOSBinFormatELF()` check (the Darwin support has gone: https://reviews.llvm.org/D50988). For x86, change the mangled code of __float128 from `U10__float128` to `g`. `U10__float128` was wrongly copied from PowerPC. The test will be added to `test/CodeGen/x86-long-double.cpp` in D64277. Reviewed By: erichkeane Differential Revision: https://reviews.llvm.org/D64276 llvm-svn: 365480
* [X86] Add ENQCMD instructionsPengfei Wang2019-06-061-0/+1
| | | | | | | | | | | | For more details about these instructions, please refer to the latest ISE document: https://software.intel.com/en-us/download/intel-architecture-instruction-set-extensions-programming-reference. Patch by Tianqing Wang (tianqing) Differential Revision: https://reviews.llvm.org/D62282 llvm-svn: 362685
* [X86] Add VP2INTERSECT instructionsPengfei Wang2019-05-311-0/+1
| | | | | | | | | | Support intel AVX512 VP2INTERSECT instructions in clang Patch by Xiang Zhang (xiangzhangllvm) Differential Revision: https://reviews.llvm.org/D62367 llvm-svn: 362196
* Enable intrinsics of AVX512_BF16, which are supported for BFLOAT16 in Cooper ↵Luo, Yuanke2019-05-061-0/+1
| | | | | | | | | | | | | | | | | | | | | | | Lake Summary: 1. Enable infrastructure of AVX512_BF16, which is supported for BFLOAT16 in Cooper Lake; 2. Enable intrinsics for VCVTNE2PS2BF16, VCVTNEPS2BF16 and DPBF16PS instructions, which are Vector Neural Network Instructions supporting BFLOAT16 inputs and conversion instructions from IEEE single precision. For more details about BF16 intrinsic, please refer to the latest ISE document: https://software.intel.com/en-us/download/intel-architecture-instruction-set-extensions-programming-reference Patch by LiuTianle Reviewers: craig.topper, smaslov, LuoYuanke, wxiao3, annita.zhang, spatel, RKSimon Reviewed By: craig.topper Subscribers: mgorny, cfe-commits Tags: #clang Differential Revision: https://reviews.llvm.org/D60552 llvm-svn: 360018
* [X86] Correct the value of MaxAtomicInlineWidth for pre-586 cpusCraig Topper2019-03-211-3/+8
| | | | | | | | | | Use the new cx8 feature flag that was added to the backend to represent support for cmpxchg8b. Use this flag to set the MaxAtomicInlineWidth. This also assumes all the cmpxchg instructions are enabled for CK_Generic which is what cc1 defaults to when nothing is specified. Differential Revision: https://reviews.llvm.org/D59566 llvm-svn: 356709
* [X86] Remove getCPUKindCanonicalName which is unused.Craig Topper2019-03-201-2/+0
| | | | | | Differential Revision: https://reviews.llvm.org/D59578 llvm-svn: 356580
* [X86] Prevent clang clobber checking for asm flag constraints.Nirav Dave2019-02-171-1/+1
| | | | | | | Update getConstraintRegister as X86 Asm flag output constraints are no longer fully alphanumeric, llvm-svn: 354211
* Update the file headers across all of the LLVM projects in the monorepoChandler Carruth2019-01-191-4/+3
| | | | | | | | | | | | | | | | | to reflect the new license. We understand that people may be surprised that we're moving the header entirely to discuss the new license. We checked this carefully with the Foundation's lawyer and we believe this is the correct approach. Essentially, all code in the project is now made available by the LLVM project under our new license, so you will see that the license headers include that license only. Some of our contributors have contributed code under our old license, and accordingly, we have retained a copy of our old license notice in the top-level files in each project and repository. llvm-svn: 351636
* Fix clang -Wimplicit-fallthrough warnings across llvm, NFCReid Kleckner2018-11-011-0/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | This patch should not introduce any behavior changes. It consists of mostly one of two changes: 1. Replacing fall through comments with the LLVM_FALLTHROUGH macro 2. Inserting 'break' before falling through into a case block consisting of only 'break'. We were already using this warning with GCC, but its warning behaves slightly differently. In this patch, the following differences are relevant: 1. GCC recognizes comments that say "fall through" as annotations, clang doesn't 2. GCC doesn't warn on "case N: foo(); default: break;", clang does 3. GCC doesn't warn when the case contains a switch, but falls through the outer case. I will enable the warning separately in a follow-up patch so that it can be cleanly reverted if necessary. Reviewers: alexfh, rsmith, lattner, rtrieu, EricWF, bollu Differential Revision: https://reviews.llvm.org/D53950 llvm-svn: 345882
* Implement Function Multiversioning for Non-ELF Systems.Erich Keane2018-10-251-3/+0
| | | | | | | | | | | | | | | | | | | | | | | | | | Similar to how ICC handles CPU-Dispatch on Windows, this patch uses the resolver function directly to forward the call to the proper function. This is not nearly as efficient as IFuncs of course, but is still quite useful for large functions specifically developed for certain processors. This is unfortunately still limited to x86, since it depends on __builtin_cpu_supports and __builtin_cpu_is, which are x86 builtins. The naming for the resolver/forwarding function for cpu-dispatch was taken from ICC's implementation, which uses the unmodified name for this (no mangling additions). This is possible, since cpu-dispatch uses '.A' for the 'default' version. In 'target' multiversioning, this function keeps the '.resolver' extension in order to keep the default function keeping the default mangling. Change-Id: I4731555a39be26c7ad59a2d8fda6fa1a50f73284 Differential Revision: https://reviews.llvm.org/D53586 llvm-svn: 345298
* Revert r323281 "Adjust MaxAtomicInlineWidth for i386/i486 targets."Hans Wennborg2018-08-241-5/+3
| | | | | | | | | | | | | | | | | | | | | | | | As reported on http://lists.llvm.org/pipermail/cfe-dev/2018-August/058760.html, this broke i386-freebsd11 due to its lack of atomic 64 bit primitives. While that's not really this commit's fault, let's revert back to the old behaviour until this can be fixed. This means generating cmpxchg8b etc for i386 and i486 which don't technically support those, but that's been the behaviour for a long time, so a little longer probably doesn't hurt that much. > Adjust MaxAtomicInlineWidth for i386/i486 targets. > > This is to fix the bug reported in https://bugs.llvm.org/show_bug.cgi?id=34347#c6. > Currently, all MaxAtomicInlineWidth of x86-32 targets are set to 64. However, > i386 doesn't support any cmpxchg related instructions. i486 only supports cmpxchg. > So in this patch MaxAtomicInlineWidth is reset as follows: > For i386, the MaxAtomicInlineWidth should be 0 because no cmpxchg is supported. > For i486, the MaxAtomicInlineWidth should be 32 because it supports cmpxchg. > For others 32 bits x86 cpu, the MaxAtomicInlineWidth should be 64 because of cmpxchg8b. > > Differential Revision: https://reviews.llvm.org/D42154 llvm-svn: 340666
* [x86/retpoline] Split the LLVM concept of retpolines into separateChandler Carruth2018-08-231-1/+0
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | subtarget features for indirect calls and indirect branches. This is in preparation for enabling *only* the call retpolines when using speculative load hardening. I've continued to use subtarget features for now as they continue to seem the best fit given the lack of other retpoline like constructs so far. The LLVM side is pretty simple. I'd like to eventually get rid of the old feature, but not sure what backwards compatibility issues that will cause. This does remove the "implies" from requesting an external thunk. This always seemed somewhat questionable and is now clearly not desirable -- you specify a thunk the same way no matter which set of things are getting retpolines. I really want to keep this nicely isolated from end users and just an LLVM implementation detail, so I've moved the `-mretpoline` flag in Clang to no longer rely on a specific subtarget feature by that name and instead to be directly handled. In some ways this is simpler, but in order to preserve existing behavior I've had to add some fallback code so that users who relied on merely passing -mretpoline-external-thunk continue to get the same behavior. We should eventually remove this I suspect (we have never tested that it works!) but I've not done that in this patch. Differential Revision: https://reviews.llvm.org/D51150 llvm-svn: 340515
* Remove trailing spaceFangrui Song2018-07-301-4/+4
| | | | | | sed -Ei 's/[[:space:]]+$//' include/**/*.{def,h,td} lib/**/*.{cpp,h} llvm-svn: 338291
* [ARM64] [Windows] Follow MS X86_64 C++ ABI when passing structsSanjin Sijaric2018-07-261-1/+1
| | | | | | | | | | | | | | | | | Summary: Microsoft's C++ object model for ARM64 is the same as that for X86_64. For example, small structs with non-trivial copy constructors or virtual function tables are passed indirectly. Currently, they are passed in registers when compiled with clang. Reviewers: rnk, mstorsjo, TomTan, haripul, javed.absar Reviewed By: rnk, mstorsjo Subscribers: kristof.beyls, chrib, llvm-commits, cfe-commits Differential Revision: https://reviews.llvm.org/D49770 llvm-svn: 338076
* Implement cpu_dispatch/cpu_specific MultiversioningErich Keane2018-07-201-0/+8
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | As documented here: https://software.intel.com/en-us/node/682969 and https://software.intel.com/en-us/node/523346. cpu_dispatch multiversioning is an ICC feature that provides for function multiversioning. This feature is implemented with two attributes: First, cpu_specific, which specifies the individual function versions. Second, cpu_dispatch, which specifies the location of the resolver function and the list of resolvable functions. This is valuable since it provides a mechanism where the resolver's TU can be specified in one location, and the individual implementions each in their own translation units. The goal of this patch is to be source-compatible with ICC, so this implementation diverges from the ICC implementation in a few ways: 1- Linux x86/64 only: This implementation uses ifuncs in order to properly dispatch functions. This is is a valuable performance benefit over the ICC implementation. A future patch will be provided to enable this feature on Windows, but it will obviously more closely fit ICC's implementation. 2- CPU Identification functions: ICC uses a set of custom functions to identify the feature list of the host processor. This patch uses the cpu_supports functionality in order to better align with 'target' multiversioning. 1- cpu_dispatch function def/decl: ICC's cpu_dispatch requires that the function marked cpu_dispatch be an empty definition. This patch supports that as well, however declarations are also permitted, since the linker will solve the issue of multiple emissions. Differential Revision: https://reviews.llvm.org/D47474 llvm-svn: 337552
* Revert r333791 "Cap "voluntary" vector alignment at 16 for all Darwin ↵Reid Kleckner2018-06-041-0/+7
| | | | | | | | | | | | | | platforms." Adding __attribute__((aligned(32))) to __m256 breaks the implementation of _mm256_loadu_ps on Windows. On Windows, alignment attributes have higher precedence than packing attributes. We also might want to carefully consider the consequences of changing our vector typedefs, since many users copy them and invent their own new, non-Intel specific vector type names. llvm-svn: 333958
* Cap "voluntary" vector alignment at 16 for all Darwin platforms.John McCall2018-06-011-7/+0
| | | | | | | | | | | | | | | | | | | | | This fixes two major problems: - We were not capping vector alignment as desired on 32-bit ARM. - We were using different alignments based on the AVX settings on Intel, so we did not have a consistent ABI. This is an ABI break, but we think we can get away with it because vectors tend to be used mostly in inline code (which is why not having a consistent ABI has not proven disastrous on Intel). Intel's AVX types are specified as having 32-byte / 64-byte alignment, so align them explicitly instead of relying on the base ABI rule. Note that this sort of attribute is stripped from template arguments in template substitution, so there's a possibility that code templated over vectors will produce inadequately-aligned objects. The right long-term solution for this is for alignment attributes to be interpreted as true qualifiers and thus preserved in the canonical type. llvm-svn: 333791
* [x86] invpcid intrinsicGabor Buella2018-05-251-0/+1
| | | | | | | | | | | | An intrinsic for an old instruction, as described in the Intel SDM. Reviewers: craig.topper, rnk Reviewed By: craig.topper, rnk Differential Revision: https://reviews.llvm.org/D47142 llvm-svn: 333256
* This patch aims to match the changes introducedAlexander Ivchenko2018-05-181-3/+7
| | | | | | | | | | | | | | | | | in gcc by https://gcc.gnu.org/ml/gcc-cvs/2018-04/msg00534.html. The -mibt feature flag is being removed, and the -fcf-protection option now also defines a CET macro and causes errors when used on non-X86 targets, while X86 targets no longer check for -mibt and -mshstk to determine if -fcf-protection is supported. -mshstk is now used only to determine availability of shadow stack intrinsics. Comes with an LLVM patch (D46882). Patch by mike.dvoretsky Differential Revision: https://reviews.llvm.org/D46881 llvm-svn: 332704
* [X86] ptwrite intrinsicGabor Buella2018-05-101-0/+1
| | | | | | | | | | Reviewers: craig.topper, RKSimon Reviewed By: craig.topper Differential Revision: https://reviews.llvm.org/D46540 llvm-svn: 331962
* Remove \brief commands from doxygen comments.Adrian Prantl2018-05-091-1/+1
| | | | | | | | | | | | | | | | | | | This is similar to the LLVM change https://reviews.llvm.org/D46290. We've been running doxygen with the autobrief option for a couple of years now. This makes the \brief markers into our comments redundant. Since they are a visual distraction and we don't want to encourage more \brief markers in new code either, this patch removes them all. Patch produced by for i in $(git grep -l '\@brief'); do perl -pi -e 's/\@brief //g' $i & done for i in $(git grep -l '\\brief'); do perl -pi -e 's/\\brief //g' $i & done Differential Revision: https://reviews.llvm.org/D46320 llvm-svn: 331834
* [x86] Introduce the pconfig intrinsicGabor Buella2018-05-081-0/+1
| | | | | | | | | | Reviewers: craig.topper, zvi Reviewed By: craig.topper Differential Revision: https://reviews.llvm.org/D46431 llvm-svn: 331740
* [X86] directstore and movdir64b intrinsicsGabor Buella2018-05-011-0/+2
| | | | | | | | | | Reviewers: spatel, craig.topper, RKSimon Reviewed By: craig.topper Differential Revision: https://reviews.llvm.org/D45984 llvm-svn: 331249
* [X86] WaitPKG intrinsicsGabor Buella2018-04-201-0/+1
| | | | | | | | | | Reviewers: craig.topper, zvi Reviewed By: craig.topper Differential Revision: https://reviews.llvm.org/D45254 llvm-svn: 330463
* [X86] Introduce cldemote intrinsicGabor Buella2018-04-131-0/+1
| | | | | | | | | | Reviewers: craig.topper, zvi Reviewed By: craig.topper Differential Revision: https://reviews.llvm.org/D45257 llvm-svn: 329993
* [x86] wbnoinvd intrinsicGabor Buella2018-04-111-0/+1
| | | | | | | | | | | | | | The WBNOINVD instruction writes back all modified cache lines in the processor’s internal cache to main memory but does not invalidate (flush) the internal caches. Reviewers: craig.topper, zvi, ashlykov Reviewed By: craig.topper Differential Revision: https://reviews.llvm.org/D43817 llvm-svn: 329848
* CodeGenCXX: support PreserveMostCC in MS ABISaleem Abdulrasool2018-04-021-0/+1
| | | | | | | | Microsoft has reserved 'U' for the PreserveMostCC which is used in the swift runtime. Add support for this. This allows the swift runtime to be built for Windows again. llvm-svn: 329025
* [ObjC++] Make parameter passing and function return compatible with ObjCAkira Hatanaka2018-03-281-0/+5
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | ObjC and ObjC++ pass non-trivial structs in a way that is incompatible with each other. For example: typedef struct { id f0; __weak id f1; } S; // this code is compiled in c++. extern "C" { void foo(S s); } void caller() { // the caller passes the parameter indirectly and destructs it. foo(S()); } // this function is compiled in c. // 'a' is passed directly and is destructed in the callee. void foo(S a) { } This patch fixes the incompatibility by passing and returning structs with __strong or weak fields using the C ABI in C++ mode. __strong and __weak fields in a struct do not cause the struct to be destructed in the caller and __strong fields do not cause the struct to be passed indirectly. Also, this patch fixes the microsoft ABI bug mentioned here: https://reviews.llvm.org/D41039?id=128767#inline-364710 rdar://problem/38887866 Differential Revision: https://reviews.llvm.org/D44908 llvm-svn: 328731
* [X86] Add 'sahf' CPU feature to frontendDimitry Andric2018-02-171-0/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Summary: Make clang accept `-msahf` (and `-mno-sahf`) flags to activate the `+sahf` feature for the backend, for bug 36028 (Incorrect use of pushf/popf enables/disables interrupts on amd64 kernels). This was originally submitted in bug 36037 by Jonathan Looney <jonlooney@gmail.com>. As described there, GCC also uses `-msahf` for this feature, and the backend already recognizes the `+sahf` feature. All that is needed is to teach clang to pass this on to the backend. The mapping of feature support onto CPUs may not be complete; rather, it was chosen to match LLVM's idea of which CPUs support this feature (see lib/Target/X86/X86.td). I also updated the affected test case (CodeGen/attr-target-x86.c) to match the emitted output. Reviewers: craig.topper, coby, efriedma, rsmith Reviewed By: craig.topper Subscribers: emaste, cfe-commits Differential Revision: https://reviews.llvm.org/D43394 llvm-svn: 325446
* Add X86 Support to ValidCPUList (enabling march notes)Erich Keane2018-02-081-0/+2
| | | | | | | | | | A followup to: https://reviews.llvm.org/D42978 This patch adds X86 and X86_64 support for enabling the march notes. Differential Revision: https://reviews.llvm.org/D43041 llvm-svn: 324674
* [NFCi] Replace a couple of usages of const StringRef& with StringRefErich Keane2018-02-071-2/+2
| | | | | | | No sense passing these by reference when a copy is about as free, and saves on potential indirection later. llvm-svn: 324540
* Adjust MaxAtomicInlineWidth for i386/i486 targets.Wei Mi2018-01-231-3/+6
| | | | | | | | | | | | | | This is to fix the bug reported in https://bugs.llvm.org/show_bug.cgi?id=34347#c6. Currently, all MaxAtomicInlineWidth of x86-32 targets are set to 64. However, i386 doesn't support any cmpxchg related instructions. i486 only supports cmpxchg. So in this patch MaxAtomicInlineWidth is reset as follows: For i386, the MaxAtomicInlineWidth should be 0 because no cmpxchg is supported. For i486, the MaxAtomicInlineWidth should be 32 because it supports cmpxchg. For others 32 bits x86 cpu, the MaxAtomicInlineWidth should be 64 because of cmpxchg8b. Differential Revision: https://reviews.llvm.org/D42154 llvm-svn: 323281
* Introduce the "retpoline" x86 mitigation technique for variant #2 of the ↵Chandler Carruth2018-01-221-0/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | speculative execution vulnerabilities disclosed today, specifically identified by CVE-2017-5715, "Branch Target Injection", and is one of the two halves to Spectre.. Summary: First, we need to explain the core of the vulnerability. Note that this is a very incomplete description, please see the Project Zero blog post for details: https://googleprojectzero.blogspot.com/2018/01/reading-privileged-memory-with-side.html The basis for branch target injection is to direct speculative execution of the processor to some "gadget" of executable code by poisoning the prediction of indirect branches with the address of that gadget. The gadget in turn contains an operation that provides a side channel for reading data. Most commonly, this will look like a load of secret data followed by a branch on the loaded value and then a load of some predictable cache line. The attacker then uses timing of the processors cache to determine which direction the branch took *in the speculative execution*, and in turn what one bit of the loaded value was. Due to the nature of these timing side channels and the branch predictor on Intel processors, this allows an attacker to leak data only accessible to a privileged domain (like the kernel) back into an unprivileged domain. The goal is simple: avoid generating code which contains an indirect branch that could have its prediction poisoned by an attacker. In many cases, the compiler can simply use directed conditional branches and a small search tree. LLVM already has support for lowering switches in this way and the first step of this patch is to disable jump-table lowering of switches and introduce a pass to rewrite explicit indirectbr sequences into a switch over integers. However, there is no fully general alternative to indirect calls. We introduce a new construct we call a "retpoline" to implement indirect calls in a non-speculatable way. It can be thought of loosely as a trampoline for indirect calls which uses the RET instruction on x86. Further, we arrange for a specific call->ret sequence which ensures the processor predicts the return to go to a controlled, known location. The retpoline then "smashes" the return address pushed onto the stack by the call with the desired target of the original indirect call. The result is a predicted return to the next instruction after a call (which can be used to trap speculative execution within an infinite loop) and an actual indirect branch to an arbitrary address. On 64-bit x86 ABIs, this is especially easily done in the compiler by using a guaranteed scratch register to pass the target into this device. For 32-bit ABIs there isn't a guaranteed scratch register and so several different retpoline variants are introduced to use a scratch register if one is available in the calling convention and to otherwise use direct stack push/pop sequences to pass the target address. This "retpoline" mitigation is fully described in the following blog post: https://support.google.com/faqs/answer/7625886 We also support a target feature that disables emission of the retpoline thunk by the compiler to allow for custom thunks if users want them. These are particularly useful in environments like kernels that routinely do hot-patching on boot and want to hot-patch their thunk to different code sequences. They can write this custom thunk and use `-mretpoline-external-thunk` *in addition* to `-mretpoline`. In this case, on x86-64 thu thunk names must be: ``` __llvm_external_retpoline_r11 ``` or on 32-bit: ``` __llvm_external_retpoline_eax __llvm_external_retpoline_ecx __llvm_external_retpoline_edx __llvm_external_retpoline_push ``` And the target of the retpoline is passed in the named register, or in the case of the `push` suffix on the top of the stack via a `pushl` instruction. There is one other important source of indirect branches in x86 ELF binaries: the PLT. These patches also include support for LLD to generate PLT entries that perform a retpoline-style indirection. The only other indirect branches remaining that we are aware of are from precompiled runtimes (such as crt0.o and similar). The ones we have found are not really attackable, and so we have not focused on them here, but eventually these runtimes should also be replicated for retpoline-ed configurations for completeness. For kernels or other freestanding or fully static executables, the compiler switch `-mretpoline` is sufficient to fully mitigate this particular attack. For dynamic executables, you must compile *all* libraries with `-mretpoline` and additionally link the dynamic executable and all shared libraries with LLD and pass `-z retpolineplt` (or use similar functionality from some other linker). We strongly recommend also using `-z now` as non-lazy binding allows the retpoline-mitigated PLT to be substantially smaller. When manually apply similar transformations to `-mretpoline` to the Linux kernel we observed very small performance hits to applications running typical workloads, and relatively minor hits (approximately 2%) even for extremely syscall-heavy applications. This is largely due to the small number of indirect branches that occur in performance sensitive paths of the kernel. When using these patches on statically linked applications, especially C++ applications, you should expect to see a much more dramatic performance hit. For microbenchmarks that are switch, indirect-, or virtual-call heavy we have seen overheads ranging from 10% to 50%. However, real-world workloads exhibit substantially lower performance impact. Notably, techniques such as PGO and ThinLTO dramatically reduce the impact of hot indirect calls (by speculatively promoting them to direct calls) and allow optimized search trees to be used to lower switches. If you need to deploy these techniques in C++ applications, we *strongly* recommend that you ensure all hot call targets are statically linked (avoiding PLT indirection) and use both PGO and ThinLTO. Well tuned servers using all of these techniques saw 5% - 10% overhead from the use of retpoline. We will add detailed documentation covering these components in subsequent patches, but wanted to make the core functionality available as soon as possible. Happy for more code review, but we'd really like to get these patches landed and backported ASAP for obvious reasons. We're planning to backport this to both 6.0 and 5.0 release streams and get a 5.0 release with just this cherry picked ASAP for distros and vendors. This patch is the work of a number of people over the past month: Eric, Reid, Rui, and myself. I'm mailing it out as a single commit due to the time sensitive nature of landing this and the need to backport it. Huge thanks to everyone who helped out here, and everyone at Intel who helped out in discussions about how to craft this. Also, credit goes to Paul Turner (at Google, but not an LLVM contributor) for much of the underlying retpoline design. Reviewers: echristo, rnk, ruiu, craig.topper, DavidKreitzer Subscribers: sanjoy, emaste, mcrosier, mgorny, mehdi_amini, hiraditya, llvm-commits Differential Revision: https://reviews.llvm.org/D41723 llvm-svn: 323155
* [X86] Add rdpid command line option and intrinsics.Craig Topper2018-01-201-0/+1
| | | | | | | | | | | | | | Summary: This patch adds -mrdpid/-mno-rdpid and the rdpid intrinsic. The corresponding LLVM commit has already been made. Reviewers: RKSimon, spatel, zvi, AndreiGrischenko Reviewed By: RKSimon Subscribers: cfe-commits Differential Revision: https://reviews.llvm.org/D42272 llvm-svn: 323047
* Added Control Flow Protection FlagOren Ben Simhon2018-01-091-0/+6
| | | | | | | | | | Cf-protection is a target independent flag that instructs the back-end to instrument control flow mechanisms like: Branch, Return, etc. For example in X86 this flag will be used to instrument Indirect Branch Tracking instructions. Differential Revision: https://reviews.llvm.org/D40478 Change-Id: I5126e766c0e6b84118cae0ee8a20fe78cc373dea llvm-svn: 322063
* Implement Attribute Target MultiVersioningErich Keane2018-01-081-0/+7
| | | | | | | | | | | | | | | | | | | | | | | | | | | | GCC's attribute 'target', in addition to being an optimization hint, also allows function multiversioning. We currently have the former implemented, this is the latter's implementation. This works by enabling functions with the same name/signature to coexist, so that they can all be emitted. Multiversion state is stored in the FunctionDecl itself, and SemaDecl manages the definitions. Note that it ends up having to permit redefinition of functions so that they can all be emitted. Additionally, all versions of the function must be emitted, so this also manages that. Note that this includes some additional rules that GCC does not, since defining something as a MultiVersion function after a usage has been made illegal. The only 'history rewriting' that happens is if a function is emitted before it has been converted to a multiversion'ed function, at which point its name needs to be changed. Function templates and virtual functions are NOT yet supported (not supported in GCC either). Additionally, constructors/destructors are disallowed, but the former is planned. llvm-svn: 322028
* [x86][icelake][vbmi2]Coby Tayree2017-12-271-0/+1
| | | | | | | | | | | | | | | added vbmi2 feature recognition added intrinsics support for vbmi2 instructions _mm[128,256,512]_mask[z]_compress_epi[16,32] _mm[128,256,512]_mask_compressstoreu_epi[16,32] _mm[128,256,512]_mask[z]_expand_epi[16,32] _mm[128,256,512]_mask[z]_expandloadu_epi[16,32] _mm[128,256,512]_mask[z]_sh[l,r]di_epi[16,32,64] _mm[128,256,512]_mask_sh[l,r]dv_epi[16,32,64] matching a similar work on the backend (D40206) Differential Revision: https://reviews.llvm.org/D41557 llvm-svn: 321487
* [x86][icelake][vnni]Coby Tayree2017-12-271-0/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | added vnni feature recognition added intrinsics support for VNNI instructions _mm256_mask_dpbusd_epi32 _mm256_maskz_dpbusd_epi32 _mm256_dpbusd_epi32 _mm256_mask_dpbusds_epi32 _mm256_maskz_dpbusds_epi32 _mm256_dpbusds_epi32 _mm256_mask_dpwssd_epi32 _mm256_maskz_dpwssd_epi32 _mm256_dpwssd_epi32 _mm256_mask_dpwssds_epi32 _mm256_maskz_dpwssds_epi32 _mm256_dpwssds_epi32 _mm128_mask_dpbusd_epi32 _mm128_maskz_dpbusd_epi32 _mm128_dpbusd_epi32 _mm128_mask_dpbusds_epi32 _mm128_maskz_dpbusds_epi32 _mm128_dpbusds_epi32 _mm128_mask_dpwssd_epi32 _mm128_maskz_dpwssd_epi32 _mm128_dpwssd_epi32 _mm128_mask_dpwssds_epi32 _mm128_maskz_dpwssds_epi32 _mm128_dpwssds_epi32 _mm512_mask_dpbusd_epi32 _mm512_maskz_dpbusd_epi32 _mm512_dpbusd_epi32 _mm512_mask_dpbusds_epi32 _mm512_maskz_dpbusds_epi32 _mm512_dpbusds_epi32 _mm512_mask_dpwssd_epi32 _mm512_maskz_dpwssd_epi32 _mm512_dpwssd_epi32 _mm512_mask_dpwssds_epi32 _mm512_maskz_dpwssds_epi32 _mm512_dpwssds_epi32 matching a similar work on the backend (D40208) Differential Revision: https://reviews.llvm.org/D41558 llvm-svn: 321484
* [x86][icelake][bitalg]Coby Tayree2017-12-271-0/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | added bitalg feature recognition added intrinsics support for bitalg instructions _mm512_popcnt_epi16 _mm512_mask_popcnt_epi16 _mm512_maskz_popcnt_epi16 _mm512_popcnt_epi8 _mm512_mask_popcnt_epi8 _mm512_maskz_popcnt_epi8 _mm512_mask_bitshuffle_epi64_mask _mm512_bitshuffle_epi64_mask _mm256_popcnt_epi16 _mm256_mask_popcnt_epi16 _mm256_maskz_popcnt_epi16 _mm128_popcnt_epi16 _mm128_mask_popcnt_epi16 _mm128_maskz_popcnt_epi16 _mm256_popcnt_epi8 _mm256_mask_popcnt_epi8 _mm256_maskz_popcnt_epi8 _mm128_popcnt_epi8 _mm128_mask_popcnt_epi8 _mm128_maskz_popcnt_epi8 _mm256_mask_bitshuffle_epi32_mask _mm256_bitshuffle_epi32_mask _mm128_mask_bitshuffle_epi16_mask _mm128_bitshuffle_epi16_mask matching a similar work on the backend (D40222) Differential Revision: https://reviews.llvm.org/D41564 llvm-svn: 321483
OpenPOWER on IntegriCloud