summaryrefslogtreecommitdiffstats
path: root/llvm/lib/CodeGen/SelectionDAG/StatepointLowering.cpp
Commit message (Collapse)AuthorAgeFilesLines
* [statepoints][experimental] Add support for live-in semantics of values in ↵Philip Reames2016-08-311-5/+35
| | | | | | | | | | | | | | | | deopt bundles This is a first step towards supporting deopt value lowering and reporting entirely with the register allocator. I hope to build on this in the near future to support live-on-return semantics, but I have a use case which allows me to test and investigate code quality with just the live-in semantics so I've chosen to start there. For those curious, my use cases is our implementation of the "__llvm_deoptimize" function we bind to @llvm.deoptimize. I'm choosing not to hard code that fact in the patch and instead make it configurable via function attributes. The basic approach here is modelled on what is done for the "Live In" values on stackmaps and patchpoints. (A secondary goal here is to remove one of the last barriers to merging the pseudo instructions.) We start by adding the operands directly to the STATEPOINT SDNode. Once we've lowered to MI, we extend the remat logic used by the register allocator to fold virtual register uses into StackMap::Indirect entries as needed. This does rely on the fact that the register allocator rematerializes. If it didn't along some code path, we could end up with more vregs than physical registers and fail to allocate. Today, we *only* fold in the register allocator. This can create some weird effects when combined with arguments passed on the stack because we don't fold them appropriately. I have an idea how to fix that, but it needs this patch in place to work on that effectively. (There's some weird interaction with the scheduler as well, more investigation needed.) My near term plan is to land this patch off-by-default, experiment in my local tree to identify any correctness issues and then start fixing codegen problems one by one as I find them. Once I have the live-in lowering fully working (both correctness and code quality), I'm hoping to move on to the live-on-return semantics. Note: I don't have any *known* miscompiles with this patch enabled, but I'm pretty sure I'll find at least a couple. Thus, the "experimental" tag and the fact it's off by default. Differential Revision: https://reviews.llvm.org/D24000 llvm-svn: 280250
* MachineFunction: Return reference for getFrameInfo(); NFCMatthias Braun2016-07-281-5/+5
| | | | | | | getFrameInfo() never returns nullptr so we should use a reference instead of a pointer. llvm-svn: 277017
* [SelectionDAG] Get rid of bool parameters in SelectionDAG::getLoad, ↵Justin Lebar2016-07-151-4/+2
| | | | | | | | | | | | | | | | | | | | | | | getStore, and friends. Summary: Instead, we take a single flags arg (a bitset). Also add a default 0 alignment, and change the order of arguments so the alignment comes before the flags. This greatly simplifies many callsites, and fixes a bug in AMDGPUISelLowering, wherein the order of the args to getLoad was inverted. It also greatly simplifies the process of adding another flag to getLoad. Reviewers: chandlerc, tstellarAMD Subscribers: jholewinski, arsenm, jyknight, dsanders, nemanjai, llvm-commits Differential Revision: http://reviews.llvm.org/D22249 llvm-svn: 275592
* Introduce an GCRelocateInst class [NFC]Philip Reames2016-04-121-5/+4
| | | | | | Previously, we were using isGCRelocate predicates. Using a subclass of IntrinsicInst is far more idiomatic. The refactoring also enables a couple of minor simplifications and code sharing. llvm-svn: 266098
* Lower @llvm.experimental.deoptimize as a noreturn callSanjoy Das2016-04-061-6/+20
| | | | | | | | | | | | | | | | | | | | | | | While preserving the return value for @llvm.experimental.deoptimize at the IR level is useful during mid-level optimization, doing so at the machine instruction level requires generating some extra code and a return that is non-ideal. This change has LLVM lower ``` %val = call @llvm.experimental.deoptimize ret %val ``` to effectively ``` call @__llvm_deoptimize() unreachable ``` instead. llvm-svn: 265502
* Reduce code duplication by extracting out a helper function; NFCSanjoy Das2016-03-241-30/+17
| | | | llvm-svn: 264355
* Lower varargs correctly in deopt bundle loweringSanjoy Das2016-03-241-0/+1
| | | | | | | Earlier we were ignoring varargs in LowerCallSiteWithDeoptBundle because populateCallLoweringInfo does not set CallLoweringInfo::IsVarArg. llvm-svn: 264354
* Add lowering support for llvm.experimental.deoptimizeSanjoy Das2016-03-241-0/+34
| | | | | | | | | | | | | | | Summary: Only adds support for "naked" calls to llvm.experimental.deoptimize. Support for round-tripping through RewriteStatepointsForGC will come as a separate patch (should be simpler than this one). Reviewers: reames Subscribers: sanjoy, mcrosier, llvm-commits Differential Revision: http://reviews.llvm.org/D18429 llvm-svn: 264329
* [Statepoints] Fix yet another issue around gc pointer uniqueingSanjoy Das2016-03-241-18/+21
| | | | | | | | | | | | | Given that StatepointLowering now uniques derived pointers before putting them in the per-statepoint spill map, we may end up with missing entries for derived pointers when we visit a gc.relocate on a pointer that was de-duplicated away. Fix this by keeping two maps, one mapping gc pointers to their de-duplicated values, and one mapping a de-duplicated value to the slot it is spilled in. llvm-svn: 264320
* Minor cosmestic changes (NFC)Sanjoy Das2016-03-241-7/+7
| | | | | | | - Reflow comments - Rename function llvm-svn: 264319
* [StatepointLowering] Minor nfc refactoringSanjoy Das2016-03-231-29/+6
| | | | | | | | | | Now that StatepointLoweringInfo represents base pointers, derived pointers and gc relocates as SmallVectors and not ArrayRefs, we no longer need to allocate "backing storage" on stack in LowerStatepoint. So elide the backing storage, and inline the trivial body of getIncomingStatepointGCValues. llvm-svn: 264128
* [StatepointLowering] Schedule gc relocates before uniqueing themSanjoy Das2016-03-231-6/+10
| | | | | | Otherwise we can see an "unexpected" gc.relocate that we uniqued away. llvm-svn: 264127
* Allow lowering call sites with both funclets and deopt stateSanjoy Das2016-03-221-5/+1
| | | | | | | Lowering funclets is a no-op, so we can just go ahead and lower the deopt state. llvm-svn: 264078
* Add "first class" lowering for deopt operand bundlesSanjoy Das2016-03-221-19/+55
| | | | | | | | | | | | | | | | | Summary: After this change, deopt operand bundles can be lowered directly by SelectionDAG into STATEPOINT instructions (which are then lowered to a call or sequence of nop, with an associated __llvm_stackmaps entry0. This obviates the need to round-trip deoptimization state through gc.statepoint via RewriteStatepointsForGC. Reviewers: reames, atrick, majnemer, JosephTremoulet, pgavlin Subscribers: sanjoy, mcrosier, majnemer, llvm-commits Differential Revision: http://reviews.llvm.org/D18257 llvm-svn: 264015
* [SelectionDAG] Remove visitStatepoint; NFCSanjoy Das2016-03-171-8/+0
| | | | | | | This way we have a single entry point into StatepointLowering. The method was a direct dispatch to LowerStatepoint anyway. llvm-svn: 263682
* Fix indentation; NFCSanjoy Das2016-03-161-3/+2
| | | | llvm-svn: 263672
* Extract out a SelectionDAGBuilder::LowerAsStatepoint; NFCSanjoy Das2016-03-161-144/+149
| | | | | | | | | | | | | | | | | | | | | | | | | | | Summary: This is a step towards implementing "direct" lowering of calls and invokes with deopt operand bundles into STATEPOINT nodes (as opposed to having them mandatorily pass through RewriteStatepointsForGC, which is the case today). This change extracts out a `SelectionDAGBuilder::LowerAsStatepoint` helper function that is able to lower a "statepoint like thing", and uses it to lower `gc.statepoint` calls. This is an NFC now, but in a later change we will use `LowerAsStatepoint` to directly lower calls and invokes with operand bundles without going through an intermediate `gc.statepoint` IR representation. FYI: I expect `SelectionDAGBuilder::StatepointInfo` will evolve as I add support for lowering non gc.statepoints, right now it is fairly tightly coupled with an IR level `gc.statepoint`. Reviewers: reames, pgavlin, JosephTremoulet Subscribers: sanjoy, mcrosier, llvm-commits Differential Revision: http://reviews.llvm.org/D18106 llvm-svn: 263671
* [SelectionDAG] Extract out populateCallLoweringInfo; NFCSanjoy Das2016-03-161-4/+6
| | | | | | | | | SelectionDAGBuilder::populateCallLoweringInfo is now used instead of SelectionDAGBuilder::lowerCallOperands. The populateCallLoweringInfo interface is more composable in face of design changes like http://reviews.llvm.org/D18106 llvm-svn: 263663
* [StatepointLowering] Move an assertion; NFCISanjoy Das2016-03-151-6/+4
| | | | | | | | Instead of running an explicit loop over `gc.relocate` calls hanging off of a `gc.statepoint`, assert the validity of the type of the value being relocated in `visitRelocate`. llvm-svn: 263516
* Make gc relocates more strongly typed; NFCSanjoy Das2016-03-121-10/+13
| | | | | | | Don't use a `Value *` where we can use a stronger `GCRelocateInst *` type. llvm-svn: 263327
* [StatepointLowering] Minor non-semantic cleanupsSanjoy Das2016-02-191-23/+18
| | | | | | Use auto, bring file up to coding standards etc. llvm-svn: 261358
* [StatepointLowering] Update StatepointMaxSlotsRequired correctlySanjoy Das2016-02-191-3/+4
| | | | | | | | | | Now that we don't always add an element to AllocatedStackSlots if we don't find a pre-existing unallocated stack slot, bumping StatepointMaxSlotsRequired to `NumSlots + 1` is not correct. Instead bump the statistic near the push_back, to Builder.FuncInfo.StatepointStackSlots.size(). llvm-svn: 261348
* [StatepointLowering] Fix a mistake in rL261336Sanjoy Das2016-02-191-4/+5
| | | | | | | | The check on MFI->getObjectSize() has to be on the FrameIndex, not on the index of the FrameIndex in AllocatedStackSlots. Weirdly, the tests I added in rL261336 didn't catch this. llvm-svn: 261347
* [StatepointLowering] Change AllocatedStackSlots to use SmallBitVectorSanjoy Das2016-02-191-9/+10
| | | | | | | | | | | | | | | | | NFCI. They key motivation here is that I'd like to use SmallBitVector::all() in a later change. Also, using a bit vector here seemed better in general. The only interesting change here is that in the failure case of allocateStackSlot, we no longer (the equivalent of) push_back(true) to AllocatedStackSlots. As far as I can tell, this is fine, since we'd never re-use those slots in the same StatepointLoweringState instance. Technically there was no need to change the operator[] type accesses to set() and test(), but I thought it'd be nice to make it obvious that we're using something other than a std::vector like thing. llvm-svn: 261337
* [StatepointLowering] Fix bug in allocateStackSlotSanjoy Das2016-02-191-2/+19
| | | | | | | | | | | | | | | | | | | | | | | | | allocateStackSlot did not consider the size of the value to be spilled before deciding to re-use a spill slot. This was originally okay (since originally we'd only ever spill pointers), but it became not okay when we changed our scheme to directly spill vectors of pointers. While this change fixes the bug pointed out, it has two performance caveats: - It matches spill slot and spillee size exactly, while in theory we can spill, e.g., an 8 byte pointer into a 16 byte slot. This is slightly complicated to fix since in the stackmaps section, we report the size of the spill slot as the size of the "indirect value"; and if they're no longer equivalent, we'll have to keep track of the (indirect) value size separately from the stack slot size. - It will "spuriously run out" of reusable slots, since we now have an second check in the search loop in addition to the availablity check (e.g. you had two free scalar slots, and you first ask for a vector slot followed by a scalar slot). I'll fix this in a later commit. llvm-svn: 261336
* [StatepointLowering] Clean up allocateStackSlotSanjoy Das2016-02-191-35/+22
| | | | | | | | | This removes the unusual loop structure in allocateStackSlot in favor of something more straightforward. I've also removed the cautionary comment in the function, which I suspect is historical cruft now, and confuses more than it enlightens. llvm-svn: 261335
* [StatepointLower] Use None instead of Optional<int>()Sanjoy Das2016-02-051-5/+5
| | | | llvm-svn: 259956
* Avoid overly large SmallPtrSet/SmallSetMatthias Braun2016-01-301-1/+1
| | | | | | | These sets perform linear searching in small mode so it is never a good idea to use SmallSize/N bigger than 32. llvm-svn: 259283
* [Statepoints] Add test cases around vectors and stablize testPhilip Reames2016-01-071-1/+3
| | | | | | | | | | Unlike my comment in 257022 said, it turns out we do handle constant vectors in the statepoint lowering, but only because SelectionDAG doesn't actually produce constants for them. Add a couple of tests which show this working. Also, add a triple to the same test file to hopefully fix a failing bot. It turns out we do han llvm-svn: 257025
* [Statepoints] Initial support for relocating vectors of pointersPhilip Reames2016-01-071-7/+7
| | | | | | | | | | | | Currently, we try to split vectors of pointers back into their component pointer elements during rewrite-statepoints-for-gc. This is less than ideal since presumably the vectorizer chose to vectorize for a reason. :) It's also been a source of bugs - in particular, the relocation logic as currently implemented was recently discovered to be wrong. The alternate approach is to allow gc.relocates of vector-of-pointer type and update the backend to handle them. That's what this patch tries to do. This won't actually enable vector-of-pointers in practice - there are some RS4GC changes needed - but the lowering is standalone and testable so it makes sense to separate. Note that there are some known cases around vector constants which this patch does not handle. Once this is in, I'll send another patch with individual fixes and test cases. Differential Revision: http://reviews.llvm.org/D15632 llvm-svn: 257022
* [Statepoints] Refactor GCRelocateOperands into an intrinsic wrapper. NFC.Manuel Jacob2016-01-051-23/+18
| | | | | | | | | | | | | | | Summary: This commit renames GCRelocateOperands to GCRelocateInst and makes it an intrinsic wrapper, similar to e.g. MemCpyInst. Also, all users of GCRelocateOperands were changed to use the new intrinsic wrapper instead. Reviewers: sanjoy, reames Subscribers: reames, sanjoy, llvm-commits Differential Revision: http://reviews.llvm.org/D15762 llvm-svn: 256811
* [Statepoints] Use Indirect operands for spill slotsPhilip Reames2015-12-231-0/+4
| | | | | | | | | | Teach the statepoint lowering code to emit Indirect stackmap entries for spill inserted by StatepointLowering (i.e. SelectionDAG), but Direct stackmap entries for in-IR allocas which represent manual stack slots. This is what the docs call for (http://llvm.org/docs/StackMaps.html#stack-map-format), but we've been emitting both as Direct. This was pointed out recently on the mailing list as a bug. It also blocks http://reviews.llvm.org/D15632 which extends the lowering to handle vector-of-pointers since only Indirect references can encode a variable sized slot. To implement this, I introduced a new flag on the StackObject class used to maintian information about stack slots. I original considered (and prototyped in http://reviews.llvm.org/D15632), the idea of using the existing isSpillSlot flag, but end up deciding that was a bit too risky and that the cost of adding a new flag was low. Having the new flag will also allow us - in the future - to emit better comments in verbose assembly which indicate where a particular stack spill around a call comes from. (deopt, gc, regalloc). Differential Revision: http://reviews.llvm.org/D15759 llvm-svn: 256352
* [GC] Make GCStrategy::isGCManagedPointer a type predicate not a value ↵Philip Reames2015-12-231-3/+3
| | | | | | | | | | | predicate [NFC] Reasons: 1) The existing form was a form of false generality. None of the implemented GCStrategies use anything other than a type. Its becoming more and more clear we're going to need some type of strong GC pointer in the type system and we shouldn't pretend otherwise at this point. 2) The API was awkward when applied to vectors-of-pointers. The old one could have been made to work, but calling isGCManagedPointer(Ty->getScalarType()) is much cleaner than the Value alternatives. 3) The rewriting implementation effectively assumes the type based predicate as well. We should be consistent. llvm-svn: 256312
* Lower statepoints with multi-def targets.Pat Gavlin2015-11-171-7/+11
| | | | | | | | | | Statepoint lowering currently expects that the target method of a statepoint only defines a single value. This precludes using statepoints with ABIs that return values in multiple registers (e.g. the SysV AMD64 ABI). This change adds support for lowering statepoints with mutli-def targets. llvm-svn: 253339
* [StatepointLowering] Remove distinction between call and invoke safepointsIgor Laevsky2015-11-041-23/+29
| | | | | | | | | | | | There is no point in having invoke safepoints handled differently than the call safepoints. All relevant decisions could be made by looking at whether or not gc.result and gc.relocate lay in a same basic block. This change will allow to lower call safepoints with relocates and results in a different basic blocks. See test case for example. Differential Revision: http://reviews.llvm.org/D14158 llvm-svn: 252028
* [WinEH] Avoid creating MBBs for LLVM BBs that cannot contain codeReid Kleckner2015-09-081-4/+4
| | | | | | | | | | | | | | Typically these are catchpads, which hold data used to decide whether to catch the exception or continue unwinding. We also shouldn't create MBBs for catchendpads, cleanupendpads, or terminatepads, since no real code can live in them. This fixes a problem where MI passes (like the register allocator) would try to put code into catchpad blocks, which are not executed by the runtime. In the new world, blocks ending in invokes now have many possible successors. llvm-svn: 247102
* PseudoSourceValue: Replace global manager with a manager in a machine function.Alex Lorenz2015-08-111-4/+6
| | | | | | | | | | | | | | | | | | | | | | This commit removes the global manager variable which is responsible for storing and allocating pseudo source values and instead it introduces a new manager class named 'PseudoSourceValueManager'. Machine functions now own an instance of the pseudo source value manager class. This commit also modifies the 'get...' methods in the 'MachinePointerInfo' class to construct pseudo source values using the instance of the pseudo source value manager object from the machine function. This commit updates calls to the 'get...' methods from the 'MachinePointerInfo' class in a lot of different files because those calls now need to pass in a reference to a machine function to those methods. This change will make it easier to serialize pseudo source values as it will enable me to transform the mips specific MipsCallEntry PseudoSourceValue subclass into two target independent subclasses. Reviewers: Akira Hatanaka llvm-svn: 244693
* Fix some comment typos.Benjamin Kramer2015-08-081-12/+13
| | | | llvm-svn: 244402
* [Statepoints] Let patchable statepoints have a symbolic call target.Sanjoy Das2015-07-281-1/+17
| | | | | | | | | | | | | | | | | | | | Summary: As added initially, statepoints required their call targets to be a constant pointer null if ``numPatchBytes`` was non-zero. This turns out to be a problem ergonomically, since there is no way to mark patchable statepoints as calling a (readable) symbolic value. This change remove the restriction of requiring ``null`` call targets for patchable statepoints, and changes PlaceSafepoints to maintain the symbolic call target through its transformation. Reviewers: reames, swaroop.sridhar Subscribers: llvm-commits Differential Revision: http://reviews.llvm.org/D11550 llvm-svn: 243502
* Redirect DataLayout from TargetMachine to Module in ComputeValueVTs()Mehdi Amini2015-07-091-3/+3
| | | | | | | | | | | | | | | | | | | | Summary: Avoid using the TargetMachine owned DataLayout and use the Module owned one instead. This requires passing the DataLayout up the stack to ComputeValueVTs(). This change is part of a series of commits dedicated to have a single DataLayout during compilation by using always the one owned by the module. Reviewers: echristo Subscribers: jholewinski, yaron.keren, rafael, llvm-commits Differential Revision: http://reviews.llvm.org/D11019 From: Mehdi Amini <mehdi.amini@apple.com> llvm-svn: 241773
* [NFC] Make the Statepoint class more like CallSiteSanjoy Das2015-07-021-3/+3
| | | | | | | | | | Summary: Rename some methods to make Statepoint look more like CallSite. Subscribers: llvm-commits Differential Revision: http://reviews.llvm.org/D10756 llvm-svn: 241235
* [Statepoint] Remove unnecessary argument from Statepoint::getRelocatesSanjoy Das2015-06-201-4/+2
| | | | | | NFC. llvm-svn: 240198
* [StatepointLowering] Reuse stack slots across basic blocksIgor Laevsky2015-06-101-66/+115
| | | | | | | | | | | During statepoint lowering we can sometimes avoid spilling of the value if we know that it was already spilled for previous statepoint. We were doing this by checking if incoming statepoint value was lowered into load from stack slot. This was working only in boundaries of one basic block. But instead of looking at the lowered node we can look directly at the llvm-ir value and if it was gc.relocate (or some simple modification of it) look up stack slot for it's derived pointer and reuse stack slot from it. This allows us to look across basic block boundaries. Differential Revision: http://reviews.llvm.org/D10251 llvm-svn: 239472
* Statepoint: Fix handling of Far Immediate callsSwaroop Sridhar2015-06-041-11/+0
| | | | | | | | | | | | | | | | | | | | | | | | | | | gc.statepoint intrinsics with a far immediate call target were lowered incorrectly as pc-rel32 calls. This change fixes the problem, and generates an indirect call via a scratch register. For example: Intrinsic: %safepoint_token = call i32 (i64, i32, void ()*, i32, i32, ...) @llvm.experimental.gc.statepoint.p0f_isVoidf(i64 0, i32 0, void ()* inttoptr (i64 140727162896504 to void ()*), i32 0, i32 0, i32 0, i32 0) Old Incorrect Lowering: callq 140727162896504 New Correct Lowering: movabsq $140727162896504, %rax callq *%rax In lowerCallFromStatepoint(), the callee-target was modified and represented as a "TargetConstant" node, rather than a "Constant" node. Undoing this modification enabled LowerCall() to generate the correct CALL instruction. llvm-svn: 239114
* [StatepointLowering] Support of the gc.relocates for invoke statepoints.Igor Laevsky2015-05-201-32/+77
| | | | | | | | | | This change implements support for lowering of the gc.relocates tied to the invoke statepoint. This is acomplished by storing frame indices of the lowered values in "StatepointRelocatedValues" map inside FunctionLoweringInfo instead of storing them in per-basic block structure StatepointLowering. After this change StatepointLowering is used only during "LowerStatepoint" call and it is not necessary to store it as a field in SelectionDAGBuilder anymore. Differential Revision: http://reviews.llvm.org/D7798 llvm-svn: 237786
* Remove a stale commentPhilip Reames2015-05-191-3/+0
| | | | | | The todo was implemented a while ago; I just forgot to remove the comment. llvm-svn: 237736
* [Statepoints] Support for "patchable" statepoints.Sanjoy Das2015-05-121-6/+11
| | | | | | | | | | | | | | | | | | | | | | | | | | | | Summary: This change adds two new parameters to the statepoint intrinsic, `i64 id` and `i32 num_patch_bytes`. `id` gets propagated to the ID field in the generated StackMap section. If the `num_patch_bytes` is non-zero then the statepoint is lowered to `num_patch_bytes` bytes of nops instead of a call (the spill and reload code remains unchanged). A non-zero `num_patch_bytes` is useful in situations where a language runtime requires complete control over how a call is lowered. This change brings statepoints one step closer to patchpoints. With some additional work (that is not part of this patch) it should be possible to get rid of `TargetOpcode::STATEPOINT` altogether. PlaceSafepoints generates `statepoint` wrappers with `id` set to `0xABCDEF00` (the old default value for the ID reported in the stackmap) and `num_patch_bytes` set to `0`. This can be made more sophisticated later. Reviewers: reames, pgavlin, swaroop.sridhar, AndyAyers Subscribers: llvm-commits Differential Revision: http://reviews.llvm.org/D9546 llvm-svn: 237214
* [Statepoints] Clean up statepoint argument accessors.Pat Gavlin2015-05-121-22/+12
| | | | | | Differential Revision: http://reviews.llvm.org/D9622 llvm-svn: 237191
* [Statepoints] Split the calling convention and statepoint flags operand to ↵Pat Gavlin2015-05-121-22/+15
| | | | | | | | STATEPOINT into two separate operands. Differential Revision: http://reviews.llvm.org/D9623 llvm-svn: 237166
* Reverse ordering of base and derived pointer during safepoint lowering.Igor Laevsky2015-05-121-10/+12
| | | | | | | | According to the documentation in StackMap section for the safepoint we should have: "The first Location in each pair describes the base pointer for the object. The second is the derived pointer actually being relocated." But before this change we emitted them in reverse order - derived pointer first, base pointer second. llvm-svn: 237126
OpenPOWER on IntegriCloud