summaryrefslogtreecommitdiffstats
path: root/mlir/lib/Transforms/LoopFusion.cpp
Commit message (Collapse)AuthorAgeFilesLines
...
* Replace usages of Instruction with Operation in the /Analysis directory.River Riddle2019-03-291-2/+2
| | | | PiperOrigin-RevId: 240569775
* Introduce affine terminatorAlex Zinenko2019-03-291-3/+3
| | | | | | | | | | | | | | | | | | | | | | Due to legacy reasons (ML/CFG function separation), regions in affine control flow operations require contained blocks not to have terminators. This is inconsistent with the notion of the block and may complicate code motion between regions of affine control operations and other regions. Introduce `affine.terminator`, a special terminator operation that must be used to terminate blocks inside affine operations and transfers the control back to he region enclosing the affine operation. For brevity and readability reasons, allow `affine.for` and `affine.if` to omit the `affine.terminator` in their regions when using custom printing and parsing format. The custom parser injects the `affine.terminator` if it is missing so as to always have it present in constructed operations. Update transformations to account for the presence of terminator. In particular, most code motion transformation between loops should leave the terminator in place, and code motion between loops and non-affine blocks should drop the terminator. PiperOrigin-RevId: 240536998
* Replace usages of Instruction with Operation in the /IR directory.River Riddle2019-03-291-29/+29
| | | | | | This is step 2/N to renaming Instruction to Operation. PiperOrigin-RevId: 240459216
* Make FunctionPass::getFunction() return a reference to the function, instead ofChris Lattner2019-03-291-4/+4
| | | | | | | a pointer. This makes it consistent with all the other methods in FunctionPass, as well as with ModulePass::getModule(). NFC. PiperOrigin-RevId: 240257910
* Replace usages of "Op::operator->" with ".".River Riddle2019-03-291-19/+19
| | | | | | This is step 2/N of removing the temporary operator-> method as part of the de-const transition. PiperOrigin-RevId: 240200792
* Replace usages of "operator->" with "." for the AffineOps.River Riddle2019-03-291-40/+39
| | | | | Note: The "operator->" method is a temporary helper for the de-const transition and is gradually being phased out. PiperOrigin-RevId: 240179439
* NFC: Rename the 'for' operation in the AffineOps dialect to 'affine.for' and ↵River Riddle2019-03-291-1/+2
| | | | | | set the namespace of the AffineOps dialect to 'affine'. PiperOrigin-RevId: 240165792
* Remove OpPointer, cleaning up a ton of code. This also moves Ops to usingChris Lattner2019-03-291-14/+14
| | | | | | | | | inherited constructors, which is cleaner and means you can now use DimOp() to get a null op, instead of having to use Instruction::getNull<DimOp>(). This removes another 200 lines of code. PiperOrigin-RevId: 240068113
* Remove const from Value, Instruction, Argument, and the various methods on theChris Lattner2019-03-291-1/+1
| | | | | | *Op classes. This is a net reduction by almost 400LOC. PiperOrigin-RevId: 239972443
* Remove some statements that required >C++11, add includes and qualify names. ↵Jacques Pienaar2019-03-291-0/+1
| | | | | | NFC. PiperOrigin-RevId: 239197784
* Rename BlockList into RegionAlex Zinenko2019-03-291-2/+2
| | | | | | NFC. This is step 1/n to specifying regions as parts of any operation. PiperOrigin-RevId: 238472370
* Change getMemoryFootprintBytes emitError to a warningUday Bondhugula2019-03-291-1/+0
| | | | | | | | - this is really not a hard error; emit a warning instead (for inability to compute footprint due to the union failing due to unimplemented cases) - remove a misleading warning from LoopFusion.cpp PiperOrigin-RevId: 238118711
* Add a basic model to set tile sizes + some cleanupUday Bondhugula2019-03-291-16/+31
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | - compute tile sizes based on a simple model that looks at memory footprints (instead of using the hardcoded default value) - adjust tile sizes to make them factors of trip counts based on an option - update loop fusion CL options to allow setting maximal fusion at pass creation - change an emitError to emitWarning (since it's not a hard error unless the client treats it that way, in which case, it can emit one) $ mlir-opt -debug-only=loop-tile -loop-tile test/Transforms/loop-tiling.mlir test/Transforms/loop-tiling.mlir:81:3: note: using tile sizes [4 4 5 ] for %i = 0 to 256 { for %i0 = 0 to 256 step 4 { for %i1 = 0 to 256 step 4 { for %i2 = 0 to 250 step 5 { for %i3 = #map4(%i0) to #map11(%i0) { for %i4 = #map4(%i1) to #map11(%i1) { for %i5 = #map4(%i2) to #map12(%i2) { %0 = load %arg0[%i3, %i5] : memref<8x8xvector<64xf32>> %1 = load %arg1[%i5, %i4] : memref<8x8xvector<64xf32>> %2 = load %arg2[%i3, %i4] : memref<8x8xvector<64xf32>> %3 = mulf %0, %1 : vector<64xf32> %4 = addf %2, %3 : vector<64xf32> store %4, %arg2[%i3, %i4] : memref<8x8xvector<64xf32>> } } } } } } PiperOrigin-RevId: 237461836
* Convert ambiguous bool returns in /Analysis to use Status instead.River Riddle2019-03-291-13/+14
| | | | PiperOrigin-RevId: 237390240
* Use FlatAffineConstraints::unionBoundingBox to perform slice bounds union ↵MLIR Team2019-03-291-77/+106
| | | | | | | | | for loop fusion pass (WIP). Adds utility to convert slice bounds to a FlatAffineConstraints representation. Adds utility to FlatAffineConstraints to promote loop IV symbol identifiers to dim identifiers. PiperOrigin-RevId: 236973261
* Change Pass:getFunction() to return pointer instead of ref - NFCUday Bondhugula2019-03-291-1/+1
| | | | | | | | | - change this for consistency - everything else similar takes/returns a Function pointer - the FuncBuilder ctor, Block/Value/Instruction::getFunction(), etc. - saves a whole bunch of &s everywhere PiperOrigin-RevId: 236928761
* Handle MemRefRegion::compute return value in loop fusion pass (NFC).MLIR Team2019-03-291-8/+27
| | | | PiperOrigin-RevId: 236685849
* Remove hidden flag from fusion CL optionsUday Bondhugula2019-03-291-4/+4
| | | | PiperOrigin-RevId: 236409185
* NFC. Move all of the remaining operations left in BuiltinOps to StandardOps. ↵River Riddle2019-03-291-1/+0
| | | | | | The only thing left in BuiltinOps are the core MLIR types. The standard types can't be moved because they are referenced within the IR directory, e.g. in things like Builder. PiperOrigin-RevId: 236403665
* Use consistent names for dialect op source filesLei Zhang2019-03-291-1/+1
| | | | | | | | | | | | | | | This CL changes dialect op source files (.h, .cpp, .td) to follow the following convention: <full-dialect-name>/<dialect-namespace>Ops.{h|cpp|td} Builtin and standard dialects are specially treated, though. Both of them do not have dialect namespace; the former is still named as BuiltinOps.* and the latter is named as Ops.*. Purely mechanical. NFC. PiperOrigin-RevId: 236371358
* Loop fusion for input reuse.MLIR Team2019-03-291-43/+415
| | | | | | | | | | | | | *) Breaks fusion pass into multiple sub passes over nodes in data dependence graph: - first pass fuses single-use producers into their unique consumer. - second pass enables fusing for input-reuse by fusing sibling nodes which read from the same memref, but which do not share dependence edges. - third pass fuses remaining producers into their consumers (Note that the sibling fusion pass may have transformed a producer with multiple uses into a single-use producer). *) Fusion for input reuse is enabled by computing a sibling node slice using the load/load accesses to the same memref, and fusion safety is guaranteed by checking that the sibling node memref write region (to a different memref) is preserved. *) Enables output vector and output matrix computations from KFAC patches-second-moment operation to fuse into a single loop nest and reuse input from the image patches operation. *) Adds a generic loop utilitiy for finding all sequential loops in a loop nest. *) Adds and updates unit tests. PiperOrigin-RevId: 236350987
* Remove PassResult and have the runOnFunction/runOnModule functions return ↵River Riddle2019-03-291-3/+2
| | | | | | void instead. To signal a pass failure, passes should now invoke the 'signalPassFailure' method. This provides the equivalent functionality when needed, but isn't an intrusive part of the API like PassResult. PiperOrigin-RevId: 236202029
* Port all of the existing passes over to the new pass manager infrastructure. ↵River Riddle2019-03-291-9/+7
| | | | | | This is largely NFC. PiperOrigin-RevId: 235952357
* Temp change in FlatAffineConstraints::getSliceBounds() to deal with TODO inUday Bondhugula2019-03-291-2/+1
| | | | | | | | | | | LoopFusion - getConstDifference in LoopFusion is pending a refactoring to handle bounds with min's and max's; it currently asserts on some useful test cases that we want to experiment with. This CL changes getSliceBounds to be more conservative so as to not trigger the assertion. Filed b/126426796 to track this. PiperOrigin-RevId: 235826538
* Loop fusion comand line options cleanupUday Bondhugula2019-03-291-10/+25
| | | | | | | | - clean up loop fusion CL options for promoting local buffers to fast memory space - add parameters to loop fusion pass instantiation PiperOrigin-RevId: 235813419
* Refactor AffineExprFlattener and move FlatAffineConstraints out of IR intoUday Bondhugula2019-03-291-1/+1
| | | | | | | | | | | | | | | | | Analysis - NFC - refactor AffineExprFlattener (-> SimpleAffineExprFlattener) so that it doesn't depend on FlatAffineConstraints, and so that FlatAffineConstraints could be moved out of IR/; the simplification that the IR needs for AffineExpr's doesn't depend on FlatAffineConstraints - have AffineExprFlattener derive from SimpleAffineExprFlattener to use for all Analysis/Transforms purposes; override addLocalFloorDivId in the derived class - turn addAffineForOpDomain into a method on FlatAffineConstraints - turn AffineForOp::getAsValueMap into an AffineValueMap ctor PiperOrigin-RevId: 235283610
* Internal changeMLIR Team2019-03-291-1/+1
| | | | PiperOrigin-RevId: 235191129
* Define a PassID class to use when defining a pass. This allows for the type ↵River Riddle2019-03-291-3/+1
| | | | | | used for the ID field to be self documenting. It also allows for the compiler to know the set alignment of the ID object, which is useful for storing pointer identifiers within llvm data structures. PiperOrigin-RevId: 235107957
* Extend/improve getSliceBounds() / complete TODO + update unionBoundingBoxUday Bondhugula2019-03-291-11/+15
| | | | | | | | | - compute slices precisely where the destination iteration depends on multiple source iterations (instead of over-approximating to the whole source loop extent) - update unionBoundingBox to deal with input with non-matching symbols - reenable disabled backend test case PiperOrigin-RevId: 234714069
* NFC: Refactor the files related to passes.River Riddle2019-03-291-1/+1
| | | | | | | * PassRegistry is split into its own source file. * Pass related files are moved to a new library 'Pass'. PiperOrigin-RevId: 234705771
* Support fusing producer loop nests which write to a memref which is live ↵MLIR Team2019-03-291-21/+96
| | | | | | out, provided that the write region of the consumer loop nest to the same memref is a super set of the producer's write region. PiperOrigin-RevId: 234240958
* LoopFusion: perform a series of loop interchanges to increase the loop depth ↵MLIR Team2019-03-291-0/+141
| | | | | | | | | | | | | | at which slices of producer loop nests can be fused into constumer loop nests. *) Adds utility to LoopUtils to perform loop interchange of two AffineForOps. *) Adds utility to LoopUtils to sink a loop to a specified depth within a loop nest, using a series of loop interchanges. *) Computes dependences between all loads and stores in the loop nest, and classifies each loop as parallel or sequential. *) Computes loop interchange permutation required to sink sequential loops (and raise parallel loop nests) while preserving relative order among them. *) Checks each dependence against the permutation to make sure that dependences would not be violated by the loop interchange transformation. *) Calls loop interchange in LoopFusion pass on consumer loop nests before fusing in producers, sinking loops with loop carried dependences deeper into the consumer loop nest. *) Adds and updates related unit tests. PiperOrigin-RevId: 234158370
* Automated rollback of changelist 232717775.Uday Bondhugula2019-03-291-2/+1
| | | | PiperOrigin-RevId: 232807986
* NFC: Rename the 'for' operation in the AffineOps dialect to 'affine.for'. ↵River Riddle2019-03-291-1/+2
| | | | | | The is the second step to adding a namespace to the AffineOps dialect. PiperOrigin-RevId: 232717775
* Adds the ability to compute the MemRefRegion of a sliced loop nest. Utilizes ↵MLIR Team2019-03-291-11/+27
| | | | | | | | | this feature during loop fusion cost computation, to compute what the write region of a fusion candidate loop nest slice would be (without having to materialize the slice or change the IR). *) Adds parameter to public API of MemRefRegion::compute for passing in the slice loop bounds to compute the memref region of the loop nest slice. *) Exposes public method MemRefRegion::getRegionSize for computing the size of the memref region in bytes. PiperOrigin-RevId: 232706165
* Refactor the affine analysis by moving some functionality to IR and some to ↵River Riddle2019-03-291-1/+1
| | | | | | | | | | | | | | AffineOps. This is important for allowing the affine dialect to define canonicalizations directly on the operations instead of relying on transformation passes, e.g. ComposeAffineMaps. A summary of the refactoring: * AffineStructures has moved to IR. * simplifyAffineExpr/simplifyAffineMap/getFlattenedAffineExpr have moved to IR. * makeComposedAffineApply/fullyComposeAffineMapAndOperands have moved to AffineOps. * ComposeAffineMaps is replaced by AffineApplyOp::canonicalize and deleted. PiperOrigin-RevId: 232586468
* Loop fusion improvements:MLIR Team2019-03-291-7/+28
| | | | | | | *) After a private memref buffer is created for a fused loop nest, dependences on the old memref are reduced, which can open up fusion opportunities. In these cases, users of the old memref are added back to the worklist to be reconsidered for fusion. *) Fixed a bug in fusion insertion point dependence check where the memref being privatized was being skipped from the check. PiperOrigin-RevId: 232477853
* Remove InstWalker and move all instruction walking to the api facilities on ↵River Riddle2019-03-291-46/+44
| | | | | | Function/Block/Instruction. PiperOrigin-RevId: 232388113
* Refactor common code getting memref access in getMemRefRegion - NFCUday Bondhugula2019-03-291-3/+4
| | | | | | | | | | | | | | | - use getAccessMap() instead of repeating it - fold getMemRefRegion into MemRefRegion ctor (more natural, avoid heap allocation and unique_ptr where possible) - change extractForInductionVars - MutableArrayRef -> ArrayRef for the arguments. Since the method is just returning copies of 'Value *', the client can't mutate the pointers themselves; it's fine to mutate the 'Value''s themselves, but that doesn't mutate the pointers to those. - change the way extractForInductionVars returns (see b/123437690) PiperOrigin-RevId: 232359277
* Remove remaining usages of OperationInst in lib/Transforms.River Riddle2019-03-291-72/+62
| | | | PiperOrigin-RevId: 232323671
* Replace the walkOps/visitOperationInst variants from the InstWalkers with ↵River Riddle2019-03-291-2/+2
| | | | | | the Instruction variants. PiperOrigin-RevId: 232322030
* Update dma-generate pass to (1) work on blocks of instructions (instead of justUday Bondhugula2019-03-291-4/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | loops), (2) take into account fast memory space capacity and lower 'dmaDepth' to fit, (3) add location information for debug info / errors - change dma-generate pass to work on blocks of instructions (start/end iterators) instead of 'for' loops; complete TODOs - allows DMA generation for straightline blocks of operation instructions interspersed b/w loops - take into account fast memory capacity: check whether memory footprint fits in fastMemoryCapacity parameter, and recurse/lower the depth at which DMA generation is performed until it does fit in the provided memory - add location information to MemRefRegion; any insufficient fast memory capacity errors or debug info w.r.t dma generation shows location information - allow DMA generation pass to be instantiated with a fast memory capacity option (besides command line flag) - change getMemRefRegion to return unique_ptr's - change getMemRefFootprintBytes to work on a 'Block' instead of 'ForInst' - other helper methods; add postDomInstFilter option for replaceAllMemRefUsesWith; drop forInst->walkOps, add Block::walkOps methods Eg. output $ mlir-opt -dma-generate -dma-fast-mem-capacity=1 /tmp/single.mlir /tmp/single.mlir:9:13: error: Total size of all DMA buffers' for this block exceeds fast memory capacity for %i3 = (d0) -> (d0)(%i1) to (d0) -> (d0 + 32)(%i1) { ^ $ mlir-opt -debug-only=dma-generate -dma-generate -dma-fast-mem-capacity=400 /tmp/single.mlir /tmp/single.mlir:9:13: note: 8 KiB of DMA buffers in fast memory space for this block for %i3 = (d0) -> (d0)(%i1) to (d0) -> (d0 + 32)(%i1) { PiperOrigin-RevId: 232297044
* Promote local buffers created post fusion to higher memory spaceUday Bondhugula2019-03-291-8/+54
| | | | | | | | | | | | | - fusion already includes the necessary analysis to create small/local buffers post fusion; allocate these buffers in a higher memory space if the necessary pass parameters are provided (threshold size, memory space id) - although there will be a separate utility at some point to directly detect and promote small local buffers to higher memory spaces, doing it while fusion when possible is much less expensive, comes free with fusion analysis, and covers a key common case. PiperOrigin-RevId: 232063894
* Define the AffineForOp and replace ForInst with it. This patch is largely ↵River Riddle2019-03-291-94/+106
| | | | | | mechanical, i.e. changing usages of ForInst to OpPointer<AffineForOp>. An important difference is that upon construction an AffineForOp no longer automatically creates the body and induction variable. To generate the body/iv, 'createBody' can be called on an AffineForOp with no body. PiperOrigin-RevId: 232060516
* Fix ASAN issue: snapshot edge list before loop which can modify this list.MLIR Team2019-03-291-3/+15
| | | | PiperOrigin-RevId: 231686040
* LoopFusion: insert the source loop nest slice at a depth in the destination ↵MLIR Team2019-03-291-23/+90
| | | | | | loop nest which preserves dependences (above any loop carried or other dependences). This is accomplished by updating the maximum destination loop depth based on dependence checks between source loop nest loads and stores which access the memref on which the source loop nest has a store op. In addition, prevent fusing in source loop nests which write to memrefs which escape or are live out. PiperOrigin-RevId: 231684492
* Support fusing loop nests which require insertion into a new instruction ↵MLIR Team2019-03-291-89/+203
| | | | | | | | Block position while preserving dependences, opening up additional fusion opportunities. - Adds SSA Value edges to the data dependence graph used in the loop fusion pass. PiperOrigin-RevId: 231417649
* Recommit: Define a AffineOps dialect as well as an AffineIfOp operation. ↵River Riddle2019-03-291-12/+12
| | | | | | Replace all instances of IfInst with AffineIfOp and delete IfInst. PiperOrigin-RevId: 231342063
* Automated rollback of changelist 231318632.Nicolas Vasilache2019-03-291-12/+12
| | | | PiperOrigin-RevId: 231327161
* Define a AffineOps dialect as well as an AffineIfOp operation. Replace all ↵River Riddle2019-03-291-12/+12
| | | | | | instances of IfInst with AffineIfOp and delete IfInst. PiperOrigin-RevId: 231318632
OpenPOWER on IntegriCloud