summaryrefslogtreecommitdiffstats
path: root/llvm/lib/Target/AArch64/AArch64ISelLowering.cpp
Commit message (Collapse)AuthorAgeFilesLines
* merge vector stores into wider vector stores and fix AArch64 misaligned ↵Sanjay Patel2015-09-251-3/+23
| | | | | | | | | | | | | | | | | | | | | | access TLI hook (PR21711) This is a redo of D7208 ( r227242 - http://llvm.org/viewvc/llvm-project?view=revision&revision=227242 ). The patch was reverted because an AArch64 target could infinite loop after the change in DAGCombiner to merge vector stores. That happened because AArch64's allowsMisalignedMemoryAccesses() wasn't telling the truth. It reported all unaligned memory accesses as fast, but then split some 128-bit unaligned accesses up in performSTORECombine() because they are slow. This patch attempts to fix the problem in AArch's allowsMisalignedMemoryAccesses() while preserving existing (perhaps questionable) lowering behavior. The x86 test shows that store merging is working as intended for a target with fast 32-byte unaligned stores. Differential Revision: http://reviews.llvm.org/D12635 llvm-svn: 248622
* [AArch64] Emit clrex in the expanded cmpxchg fail block.Ahmed Bougacha2015-09-221-0/+7
| | | | | | | | | | | | | | | | | In the comparison failure block of a cmpxchg expansion, the initial ldrex/ldxr will not be followed by a matching strex/stxr. On ARM/AArch64, this unnecessarily ties up the execution monitor, which might have a negative performance impact on some uarchs. Instead, release the monitor in the failure block. The clrex instruction was designed for this: use it. Also see ARMARM v8-A B2.10.2: "Exclusive access instructions and Shareable memory locations". Differential Revision: http://reviews.llvm.org/D13033 llvm-svn: 248291
* Improve ISel using across lane min/max reductionJun Bum Lim2015-09-141-53/+190
| | | | | | | | | | | | | | | | | | | | In vectorized integer min/max reduction code, the final "reduce" step is sub-optimal. In AArch64, this change wll combine : %svn0 = vector_shuffle %0, undef<2,3,u,u> %smax0 = smax %0, svn0 %svn3 = vector_shuffle %smax0, undef<1,u,u,u> %sc = setcc %smax0, %svn3, gt %n0 = extract_vector_elt %sc, #0 %n1 = extract_vector_elt %smax0, #0 %n2 = extract_vector_elt $smax0, #1 %result = select %n0, %n1, n2 becomes : %1 = smaxv %0 %result = extract_vector_elt %1, 0 This change extends r246790. llvm-svn: 247575
* [CodeGen] Refactor TLI/AtomicExpand interface to make LLSC explicit.Ahmed Bougacha2015-09-111-4/+6
| | | | | | | | | | | | | | | We used to have this magic "hasLoadLinkedStoreConditional()" callback, which really meant two things: - expand cmpxchg (to ll/sc). - expand atomic loads using ll/sc (rather than cmpxchg). Remove it, and, instead, introduce explicit callbacks: - bool shouldExpandAtomicCmpXchgInIR(inst) - AtomicExpansionKind shouldExpandAtomicLoadInIR(inst) Differential Revision: http://reviews.llvm.org/D12557 llvm-svn: 247429
* [CodeGen] Rename AtomicRMWExpansionKind to AtomicExpansionKind.Ahmed Bougacha2015-09-111-3/+2
| | | | | | This lets us generalize its usage to the other atomic instructions. llvm-svn: 247428
* Remove white space (test commit)Jun Bum Lim2015-09-081-1/+1
| | | | llvm-svn: 247021
* [AArch64] Improve ISel using across lane addition reduction.Chad Rosier2015-09-031-0/+99
| | | | | | | | | | | | | | | | | | | | In vectorized add reduction code, the final "reduce" step is sub-optimal. This change wll combine : ext v1.16b, v0.16b, v0.16b, #8 add v0.4s, v1.4s, v0.4s dup v1.4s, v0.s[1] add v0.4s, v1.4s, v0.4s into addv s0, v0.4s PR21371 http://reviews.llvm.org/D12325 Patch by Jun Bum Lim <junbuml@codeaurora.org>! llvm-svn: 246790
* [AArch64] Lower READCYCLECOUNTER using MRS PMCCTNR_EL0.Ahmed Bougacha2015-09-011-0/+5
| | | | | | | | | | This matches the ARM behavior. In both cases, the register is part of the optional Performance Monitors extension, so, add the feature, and enable it for the A-class processors we support. Differential Revision: http://reviews.llvm.org/D12425 llvm-svn: 246555
* AArch64: Fix loads to lower NEON vector lanes using GPR registersMatthias Braun2015-08-311-1/+4
| | | | | | | | | | | | | The ISelLowering code turned insertion turned the element for the lowest lane of a BUILD_VECTOR into an INSERT_SUBREG, this prohibited the patterns for SCALAR_TO_VECTOR(Load) to match later. Restrict this to cases without a load argument. Reported in rdar://22223823 Differential Revision: http://reviews.llvm.org/D12467 llvm-svn: 246462
* [AArch64] Unify the integer min/max vector selection patterns with the ↵Silviu Baranga2015-08-261-0/+12
| | | | | | | | | | | | | | | | | | | | intrinsic ones Summary: This change lowers the aarch64 integer vector min/max intrinsic nodes to generic min/max nodes and replaces the intrinsic selection patterns with the generic ones. There should already be testing in place for this, so no further tests were added. Reviewers: jmolloy Subscribers: aemerson, llvm-commits, rengolin Differential Revision: http://reviews.llvm.org/D12276 llvm-svn: 246030
* AArch64: Fix cmp;ccmp orderingMatthias Braun2015-08-201-3/+10
| | | | | | | | | | | | When producing conditional compare sequences for or operations we need to negate the operands and the finally tested flags. The thing is if we negate the finally tested flags this equals a logical negation of all previously emitted expressions. There was a case missing where we have to order OR expressions so they get emitted first. This fixes http://llvm.org/PR24459 llvm-svn: 245641
* AArch64: Do not create CCMP on multiple users.Matthias Braun2015-08-201-1/+1
| | | | | | | | | Create CMP;CCMP sequences from and/or trees does not gain us anything if the and/or tree is materialized to a GP register anyway. While most of the code already checked for hasOneUse() there was one important case missing. llvm-svn: 245640
* Remove hand-rolled matching for fmin and fmax.James Molloy2015-08-171-98/+2
| | | | | | SDAGBuilder now does this all for us. llvm-svn: 245198
* [AArch64] FMINNAN/FMAXNAN on f16 is not legal.James Molloy2015-08-141-2/+4
| | | | | | | | Spotted by Ahmed - in r244594 I inadvertently marked f16 min/max as legal. I've reverted it here, and marked min/max on scalar f16's as promote. I've also added a testcase. The test just checks that the compiler doesn't fall over - it doesn't create fmin nodes for f16 yet. llvm-svn: 245035
* [AArch64] Also custom-lowering mismatched vector/f16 FCOPYSIGN.Ahmed Bougacha2015-08-131-11/+5
| | | | | | | | | We can lower them using our cool tricks if we fpext/fptrunc the second input, like we do for f32/f64. Follow-up to r243924, r243926, and r244858. llvm-svn: 244860
* PseudoSourceValue: Replace global manager with a manager in a machine function.Alex Lorenz2015-08-111-18/+25
| | | | | | | | | | | | | | | | | | | | | | This commit removes the global manager variable which is responsible for storing and allocating pseudo source values and instead it introduces a new manager class named 'PseudoSourceValueManager'. Machine functions now own an instance of the pseudo source value manager class. This commit also modifies the 'get...' methods in the 'MachinePointerInfo' class to construct pseudo source values using the instance of the pseudo source value manager object from the machine function. This commit updates calls to the 'get...' methods from the 'MachinePointerInfo' class in a lot of different files because those calls now need to pass in a reference to a machine function to those methods. This change will make it easier to serialize pseudo source values as it will enable me to transform the mips specific MipsCallEntry PseudoSourceValue subclass into two target independent subclasses. Reviewers: Akira Hatanaka llvm-svn: 244693
* [AArch64] Match fminnum/fmaxnum for vector fminnm/fmaxnm instead of an ↵James Molloy2015-08-111-2/+11
| | | | | | | | | | | intrinsic. Lower Intrinsic::aarch64_neon_fmin/fmax to fminnum/fmannum and match that instead. Minimal functional change: - Extra tests added because coverage of scalar fminnm/fmaxnm instructions was nonexistant. - f16 test updated because now we actually generate scalar fminnm/fmaxnm we no longer need to bail out to a libcall! llvm-svn: 244595
* [AArch64] Replace the custom AArch64ISD::FMIN/MAX nodes with ISD::FMINNAN/MAXNANJames Molloy2015-08-111-6/+9
| | | | | | NFCI. This just removes custom ISDNodes that are no longer needed. llvm-svn: 244594
* Fix some comment typos.Benjamin Kramer2015-08-081-2/+2
| | | | llvm-svn: 244402
* wrap OptSize and MinSize attributes for easier and consistent access (NFCI)Sanjay Patel2015-08-041-4/+2
| | | | | | | | | | | | | | | | | Create wrapper methods in the Function class for the OptimizeForSize and MinSize attributes. We want to hide the logic of "or'ing" them together when optimizing just for size (-Os). Currently, we are not consistent about this and rely on a front-end to always set OptimizeForSize (-Os) if MinSize (-Oz) is on. Thus, there are 18 FIXME changes here that should be added as follow-on patches with regression tests. This patch is NFC-intended: it just replaces existing direct accesses of the attributes by the equivalent wrapper call. Differential Revision: http://reviews.llvm.org/D11734 llvm-svn: 243994
* [AArch64] Add isel support for f16 indexed LD/ST.Ahmed Bougacha2015-08-041-0/+2
| | | | llvm-svn: 243935
* [AArch64] Remove unnecessary "break". NFC.Ahmed Bougacha2015-08-041-1/+0
| | | | llvm-svn: 243931
* [AArch64] Use SDValue bool operator. NFC.Ahmed Bougacha2015-08-041-6/+3
| | | | llvm-svn: 243930
* [AArch64] Vector FCOPYSIGN supports Custom-lowering: mark it as such.Ahmed Bougacha2015-08-041-1/+4
| | | | | | | | | | | There's a bunch of code in LowerFCOPYSIGN that does smart lowering, and is actually already vector-aware; let's use it instead of scalarizing! The only interesting change is that for v2f32, we previously always used use v4i32 as the integer vector type. Use v2i32 instead, and mark FCOPYSIGN as Custom. llvm-svn: 243926
* Convert some AArch64 code to foreach loops. NFC.Pete Cooper2015-08-031-12/+11
| | | | | | | Also converted a cast<> to dyn_cast while i was working on the same line of code. llvm-svn: 243894
* [AArch64] Define subtarget feature strict-align.Akira Hatanaka2015-07-291-19/+12
| | | | | | | | | | This commit defines subtarget feature strict-align and uses it instead of cl::opt -aarch64-strict-align to decide whether strict alignment should be forced. rdar://problem/21529937 llvm-svn: 243516
* fix TLI's combineRepeatedFPDivisors interface to return the minimum user ↵Sanjay Patel2015-07-281-2/+2
| | | | | | | | | | | | | | | threshold This fix was suggested as part of D11345 and is part of fixing PR24141. With this change, we can avoid walking the uses of a divisor node if the target doesn't want the combineRepeatedFPDivisors transform in the first place. There is no NFC-intended other than that. Differential Revision: http://reviews.llvm.org/D11531 llvm-svn: 243498
* Implement target independent TLS compatible with glibc's emutls.c.Chih-Hung Hsieh2015-07-281-0/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The 'common' section TLS is not implemented. Current C/C++ TLS variables are not placed in common section. DWARF debug info to get the address of TLS variables is not generated yet. clang and driver changes in http://reviews.llvm.org/D10524 Added -femulated-tls flag to select the emulated TLS model, which will be used for old targets like Android that do not support ELF TLS models. Added TargetLowering::LowerToTLSEmulatedModel as a target-independent function to convert a SDNode of TLS variable address to a function call to __emutls_get_address. Added into lib/Target/*/*ISelLowering.cpp to call LowerToTLSEmulatedModel for TLSModel::Emulated. Although all targets supporting ELF TLS models are enhanced, emulated TLS model has been tested only for Android ELF targets. Modified AsmPrinter.cpp to print the emutls_v.* and emutls_t.* variables for emulated TLS variables. Modified DwarfCompileUnit.cpp to skip some DIE for emulated TLS variabls. TODO: Add proper DIE for emulated TLS variables. Added new unit tests with emulated TLS. Differential Revision: http://reviews.llvm.org/D10522 llvm-svn: 243438
* Implement __builtin_thread_pointerAdhemerval Zanella2015-07-281-0/+17
| | | | | | | This path add the aarch64 lowering of __builtin_thread_pointer. It uses the already implemented AArch64ISD::THREAD_POINTER used in TLS generation. llvm-svn: 243412
* When lowering vector shifts a check is performed to see if the value to shift byLuke Cheeseman2015-07-241-13/+6
| | | | | | | | | | | | is an immediate, in this check the value is negated and stored in and int64_t. The value can be -2^63 yet the result cannot be stored in an int64_t and this gives some undefined behaviour causing failures. The negation is only necessary when the values is within a certain range and so it should not need to negate -2^63, this patch introduces this and also a regression test. Differential Revision: http://reviews.llvm.org/D11408 llvm-svn: 243100
* Fix comment typo (test commit). NFCGeoff Berry2015-07-201-1/+1
| | | | llvm-svn: 242719
* [AArch64] Use [SU]ABSDIFF nodes instead of intrinsics for ABD/ABAJames Molloy2015-07-171-18/+29
| | | | | | | No functional change, but it preps codegen for the future when SABSDIFF will start getting generated in anger. llvm-svn: 242545
* AArch64: Implement conditional compare sequence matching.Matthias Braun2015-07-161-40/+270
| | | | | | | | | | | | | | | This is a new iteration of the reverted r238793 / http://reviews.llvm.org/D8232 which wrongly assumed that any and/or trees can be represented by conditional compare sequences, however there are some restrictions to that. This version fixes this and adds comments that explain exactly what types of and/or trees can actually be implemented as conditional compare sequences. Related to http://llvm.org/PR20927, rdar://18326194 Differential Revision: http://reviews.llvm.org/D10579 llvm-svn: 242436
* Allow {e,r}bp as the target of {read,write}_register.Pat Gavlin2015-07-091-2/+2
| | | | | | | | | | This patch allows the read_register and write_register intrinsics to read/write the RBP/EBP registers on X86 iff the targeted register is the frame pointer for the containing function. Differential Revision: http://reviews.llvm.org/D10977 llvm-svn: 241827
* Re-instate the EVT parameter to getScalarShiftAmountTy() for OOT userMehdi Amini2015-07-091-1/+2
| | | | | | | A documentation for this function would be nice by the way. From: Mehdi Amini <mehdi.amini@apple.com> llvm-svn: 241807
* Remove getDataLayout() from TargetLoweringMehdi Amini2015-07-091-12/+14
| | | | | | | | | | | | | | | | Summary: This change is part of a series of commits dedicated to have a single DataLayout during compilation by using always the one owned by the module. Reviewers: echristo Subscribers: yaron.keren, rafael, llvm-commits, jholewinski Differential Revision: http://reviews.llvm.org/D11042 From: Mehdi Amini <mehdi.amini@apple.com> llvm-svn: 241779
* Make isLegalAddressingMode() taking DataLayout as an argumentMehdi Amini2015-07-091-6/+6
| | | | | | | | | | | | | | | | Summary: This change is part of a series of commits dedicated to have a single DataLayout during compilation by using always the one owned by the module. Reviewers: echristo Subscribers: jholewinski, llvm-commits, rafael, yaron.keren Differential Revision: http://reviews.llvm.org/D11040 From: Mehdi Amini <mehdi.amini@apple.com> llvm-svn: 241778
* Make TargetLowering::getShiftAmountTy() taking DataLayout as an argumentMehdi Amini2015-07-091-1/+1
| | | | | | | | | | | | | | | | Summary: This change is part of a series of commits dedicated to have a single DataLayout during compilation by using always the one owned by the module. Reviewers: echristo Subscribers: jholewinski, llvm-commits, rafael, yaron.keren Differential Revision: http://reviews.llvm.org/D11037 From: Mehdi Amini <mehdi.amini@apple.com> llvm-svn: 241776
* Make TargetLowering::getPointerTy() taking DataLayout as an argumentMehdi Amini2015-07-091-62/+70
| | | | | | | | | | | | | | | | Summary: This change is part of a series of commits dedicated to have a single DataLayout during compilation by using always the one owned by the module. Reviewers: echristo Subscribers: jholewinski, ted, yaron.keren, rafael, llvm-commits Differential Revision: http://reviews.llvm.org/D11028 From: Mehdi Amini <mehdi.amini@apple.com> llvm-svn: 241775
* [TargetLowering] StringRefize asm constraint getters.Benjamin Kramer2015-07-051-7/+5
| | | | | | | | There is some functional change here because it changes target code from atoi(3) to StringRef::getAsInteger which has error checking. For valid constraints there should be no difference. llvm-svn: 241411
* Add op_values() to iterate over the SDValue operands of an SDNode.Pete Cooper2015-06-261-2/+1
| | | | | | | | | | SDNode already had ops() which would iterate over the operands and return SDUse*. This version instead gets the SDValue's out of the SDUse's so that we can use foreach in more places. Reviewed by David Blaikie. llvm-svn: 240805
* [AArch64] Lower interleaved memory accesses to ldN/stN intrinsics. This ↵Hao Liu2015-06-261-0/+154
| | | | | | | | | | | | | | | | | | | | | | | | | | patch also adds a function to calculate the cost of interleaved memory accesses. E.g. Lower an interleaved load: %wide.vec = load <8 x i32>, <8 x i32>* %ptr %v0 = shuffle %wide.vec, undef, <0, 2, 4, 6> %v1 = shuffle %wide.vec, undef, <1, 3, 5, 7> into: %ld2 = { <4 x i32>, <4 x i32> } call llvm.aarch64.neon.ld2(%ptr) %vec0 = extractelement { <4 x i32>, <4 x i32> } %ld2, i32 0 %vec1 = extractelement { <4 x i32>, <4 x i32> } %ld2, i32 1 E.g. Lower an interleaved store: %i.vec = shuffle <8 x i32> %v0, <8 x i32> %v1, <0, 4, 8, 1, 5, 9, 2, 6, 10, 3, 7, 11> store <12 x i32> %i.vec, <12 x i32>* %ptr into: %sub.v0 = shuffle <8 x i32> %v0, <8 x i32> v1, <0, 1, 2, 3> %sub.v1 = shuffle <8 x i32> %v0, <8 x i32> v1, <4, 5, 6, 7> %sub.v2 = shuffle <8 x i32> %v0, <8 x i32> v1, <8, 9, 10, 11> call void llvm.aarch64.neon.st3(%sub.v0, %sub.v1, %sub.v2, %ptr) Differential Revision: http://reviews.llvm.org/D10533 llvm-svn: 240754
* Fix "the the" in comments.Eric Christopher2015-06-191-2/+2
| | | | llvm-svn: 240112
* Revert "AArch64: Use CMP;CCMP sequences for and/or/setcc trees."Matthias Braun2015-06-171-178/+36
| | | | | | | | | The patch triggers a miscompile on SPEC 2006 403.gcc with the (ref) 200.i and scilab.i inputs. I opened PR23866 to track analysis of this. This reverts commit r238793. llvm-svn: 239880
* Clean up redundant copies of Triple objects. NFCDaniel Sanders2015-06-161-1/+1
| | | | | | | | | | | | | | Summary: Reviewers: rengolin Reviewed By: rengolin Subscribers: llvm-commits, rengolin, jholewinski Differential Revision: http://reviews.llvm.org/D10382 llvm-svn: 239823
* [AArch64] Generalize extract-high DUP extension to MOVI/MVNI.Ahmed Bougacha2015-06-161-15/+24
| | | | | | | | | | | | | | | | | | These are really immediate DUPs, and suffer from the same problem with long instructions with a high/2 variant (e.g. smull). By extending a MOVI (or DUP, before this patch), we can avoid an ext on the other operand of the long instruction, e.g. turning: ext.16b v0, v0, v0, #8 movi.4h v1, #0x53 smull.4s v0, v0, v1 into: movi.8h v1, #0x53 smull2.4s v0, v0, v1 While there, add a now-necessary combine to fold (VT NVCAST (VT x)). llvm-svn: 239799
* AArch64: Use CMP;CCMP sequences for and/or/setcc trees.Matthias Braun2015-06-011-36/+178
| | | | | | | | | | | | Previously CCMP/FCCMP instructions were only used by the AArch64ConditionalCompares pass for control flow. This patch uses them for SELECT like instructions as well by matching patterns in ISelLowering. PR20927, rdar://18326194 Differential Revision: http://reviews.llvm.org/D8232 llvm-svn: 238793
* Re-commit of r238201 with fix for building with shared libraries.Luke Cheeseman2015-06-011-1/+2
| | | | llvm-svn: 238739
* Add address space argument to isLegalAddressingModeMatt Arsenault2015-06-011-3/+5
| | | | | | | | | | This is important because of different addressing modes depending on the address space for GPU targets. This only adds the argument, and does not update any of the uses to provide the correct address space. llvm-svn: 238723
* Revert "Re-commit changes in r237579 with fix for bug breaking windows builds."Diego Novillo2015-05-261-2/+1
| | | | | | | This reverts commit r238201 to fix linking problems in x86 Linux http://lists.cs.uiuc.edu/pipermail/llvm-commits/Week-of-Mon-20150525/278413.html llvm-svn: 238223
OpenPOWER on IntegriCloud