Extend coroutines to support a "returned continuation" lowering.

A quick contrast of this ABI with the currently-implemented ABI:

- Allocation is implicitly managed by the lowering passes, which is fine
  for frontends that are fine with assuming that allocation cannot fail.
  This assumption is necessary to implement dynamic allocas anyway.

- The lowering attempts to fit the coroutine frame into an opaque,
  statically-sized buffer before falling back on allocation; the same
  buffer must be provided to every resume point.  A buffer must be at
  least pointer-sized.

- The resume and destroy functions have been combined; the continuation
  function takes a parameter indicating whether it has succeeded.

- Conversely, every suspend point begins its own continuation function.

- The continuation function pointer is directly returned to the caller
  instead of being stored in the frame.  The continuation can therefore
  directly destroy the frame when exiting the coroutine instead of having
  to leave it in a defunct state.

- Other values can be returned directly to the caller instead of going
  through a promise allocation.  The frontend provides a "prototype"
  function declaration from which the type, calling convention, and
  attributes of the continuation functions are taken.

- On the caller side, the frontend can generate natural IR that directly
  uses the continuation functions as long as it prevents IPO with the
  coroutine until lowering has happened.  In combination with the point
  above, the frontend is almost totally in charge of the ABI of the
  coroutine.

- Unique-yield coroutines are given some special treatment.

llvm-svn: 368788

1 files changed, 272 insertions, 31 deletions

diff --git a/llvm/docs/Coroutines.rst b/llvm/docs/Coroutines.rst
index f3667585c6c..c8b97cdb4ab 100644
--- a/llvm/docs/Coroutines.rst
+++ b/llvm/docs/Coroutines.rst
@@ -41,32 +41,144 @@ then destroy it:
 In addition to the function stack frame which exists when a coroutine is 
 executing, there is an additional region of storage that contains objects that 
 keep the coroutine state when a coroutine is suspended. This region of storage
-is called **coroutine frame**. It is created when a coroutine is called and 
-destroyed when a coroutine runs to completion or destroyed by a call to 
-the `coro.destroy`_ intrinsic. 
+is called the **coroutine frame**. It is created when a coroutine is called
+and destroyed when a coroutine either runs to completion or is destroyed
+while suspended.
+
+LLVM currently supports two styles of coroutine lowering. These styles
+support substantially different sets of features, have substantially
+different ABIs, and expect substantially different patterns of frontend
+code generation. However, the styles also have a great deal in common.
+
+In all cases, an LLVM coroutine is initially represented as an ordinary LLVM
+function that has calls to `coroutine intrinsics`_ defining the structure of
+the coroutine. The coroutine function is then, in the most general case,
+rewritten by the coroutine lowering passes to become the "ramp function",
+the initial entrypoint of the coroutine, which executes until a suspend point
+is first reached. The remainder of the original coroutine function is split
+out into some number of "resume functions". Any state which must persist
+across suspensions is stored in the coroutine frame. The resume functions
+must somehow be able to handle either a "normal" resumption, which continues
+the normal execution of the coroutine, or an "abnormal" resumption, which
+must unwind the coroutine without attempting to suspend it.
+
+Switched-Resume Lowering
+------------------------
+
+In LLVM's standard switched-resume lowering, signaled by the use of
+`llvm.coro.id`, the coroutine frame is stored as part of a "coroutine
+object" which represents a handle to a particular invocation of the
+coroutine.  All coroutine objects support a common ABI allowing certain
+features to be used without knowing anything about the coroutine's
+implementation:
+
+- A coroutine object can be queried to see if it has reached completion
+  with `llvm.coro.done`.
+
+- A coroutine object can be resumed normally if it has not already reached
+  completion with `llvm.coro.resume`.
+
+- A coroutine object can be destroyed, invalidating the coroutine object,
+  with `llvm.coro.destroy`.  This must be done separately even if the
+  coroutine has reached completion normally.
+
+- "Promise" storage, which is known to have a certain size and alignment,
+  can be projected out of the coroutine object with `llvm.coro.promise`.
+  The coroutine implementation must have been compiled to define a promise
+  of the same size and alignment.
+
+In general, interacting with a coroutine object in any of these ways while
+it is running has undefined behavior.
+
+The coroutine function is split into three functions, representing three
+different ways that control can enter the coroutine:
+
+1. the ramp function that is initially invoked, which takes arbitrary
+   arguments and returns a pointer to the coroutine object;
 
-An LLVM coroutine is represented as an LLVM function that has calls to
-`coroutine intrinsics`_ defining the structure of the coroutine.
-After lowering, a coroutine is split into several
-functions that represent three different ways of how control can enter the 
-coroutine: 
+2. a coroutine resume function that is invoked when the coroutine is resumed,
+   which takes a pointer to the coroutine object and returns `void`;
 
-1. a ramp function, which represents an initial invocation of the coroutine that
-   creates the coroutine frame and executes the coroutine code until it 
-   encounters a suspend point or reaches the end of the function;
+3. a coroutine destroy function that is invoked when the coroutine is
+   destroyed, which takes a pointer to the coroutine object and returns
+   `void`.
 
-2. a coroutine resume function that is invoked when the coroutine is resumed;
+Because the resume and destroy functions are shared across all suspend
+points, suspend points must store the index of the active suspend in
+the coroutine object, and the resume/destroy functions must switch over
+that index to get back to the correct point.  Hence the name of this
+lowering.
 
-3. a coroutine destroy function that is invoked when the coroutine is destroyed.
+Pointers to the resume and destroy functions are stored in the coroutine
+object at known offsets which are fixed for all coroutines.  A completed
+coroutine is represented with a null resume function.
 
-.. note:: Splitting out resume and destroy functions are just one of the 
-   possible ways of lowering the coroutine. We chose it for initial 
-   implementation as it matches closely the mental model and results in 
-   reasonably nice code.
+There is a somewhat complex protocol of intrinsics for allocating and
+deallocating the coroutine object.  It is complex in order to allow the
+allocation to be elided due to inlining.  This protocol is discussed
+in further detail below.
+
+The frontend may generate code to call the coroutine function directly;
+this will become a call to the ramp function and will return a pointer
+to the coroutine object.  The frontend should always resume or destroy
+the coroutine using the corresping intrinsics.
+
+Returned-Continuation Lowering
+------------------------------
+
+In returned-continuation lowering, signaled by the use of
+`llvm.coro.id.retcon` or `llvm.coro.id.retcon.once`, some aspects of
+the ABI must be handled more explicitly by the frontend.
+
+In this lowering, every suspend point takes a list of "yielded values"
+which are returned back to the caller along with a function pointer,
+called the continuation function.  The coroutine is resumed by simply
+calling this continuation function pointer.  The original coroutine
+is divided into the ramp function and then an arbitrary number of
+these continuation functions, one for each suspend point.
+
+LLVM actually supports two closely-related returned-continuation
+lowerings:
+
+- In normal returned-continuation lowering, the coroutine may suspend
+  itself multiple times. This means that a continuation function
+  itself returns another continuation pointer, as well as a list of
+  yielded values.
+
+  The coroutine indicates that it has run to completion by returning
+  a null continuation pointer. Any yielded values will be `undef`
+  should be ignored.
+
+- In yield-once returned-continuation lowering, the coroutine must
+  suspend itself exactly once (or throw an exception).  The ramp
+  function returns a continuation function pointer and yielded
+  values, but the continuation function simply returns `void`
+  when the coroutine has run to completion.
+
+The coroutine frame is maintained in a fixed-size buffer that is
+passed to the `coro.id` intrinsic, which guarantees a certain size
+and alignment statically. The same buffer must be passed to the
+continuation function(s). The coroutine will allocate memory if the
+buffer is insufficient, in which case it will need to store at
+least that pointer in the buffer; therefore the buffer must always
+be at least pointer-sized. How the coroutine uses the buffer may
+vary between suspend points.
+
+In addition to the buffer pointer, continuation functions take an
+argument indicating whether the coroutine is being resumed normally
+(zero) or abnormally (non-zero).
+
+LLVM is currently ineffective at statically eliminating allocations
+after fully inlining returned-continuation coroutines into a caller.
+This may be acceptable if LLVM's coroutine support is primarily being
+used for low-level lowering and inlining is expected to be applied
+earlier in the pipeline.
 
 Coroutines by Example
 =====================
 
+The examples below are all of switched-resume coroutines.
+
 Coroutine Representation
 ------------------------
 
@@ -554,6 +666,7 @@ and python iterator `__next__` would look like:
     return *(int*)coro.promise(hdl, 4, false);
   }
 
+
 Intrinsics
 ==========
 
@@ -580,7 +693,7 @@ Overview:
 """""""""
 
 The '``llvm.coro.destroy``' intrinsic destroys a suspended
-coroutine.
+switched-resume coroutine.
 
 Arguments:
 """"""""""
@@ -607,7 +720,7 @@ frame. Destroying a coroutine that is not suspended leads to undefined behavior.
 Overview:
 """""""""
 
-The '``llvm.coro.resume``' intrinsic resumes a suspended coroutine.
+The '``llvm.coro.resume``' intrinsic resumes a suspended switched-resume coroutine.
 
 Arguments:
 """"""""""
@@ -634,8 +747,8 @@ Resuming a coroutine that is not suspended leads to undefined behavior.
 Overview:
 """""""""
 
-The '``llvm.coro.done``' intrinsic checks whether a suspended coroutine is at 
-the final suspend point or not.
+The '``llvm.coro.done``' intrinsic checks whether a suspended
+switched-resume coroutine is at the final suspend point or not.
 
 Arguments:
 """"""""""
@@ -661,7 +774,7 @@ Overview:
 """""""""
 
 The '``llvm.coro.promise``' intrinsic obtains a pointer to a 
-`coroutine promise`_ given a coroutine handle and vice versa.
+`coroutine promise`_ given a switched-resume coroutine handle and vice versa.
 
 Arguments:
 """"""""""
@@ -739,7 +852,8 @@ Overview:
 """""""""
 
 The '``llvm.coro.size``' intrinsic returns the number of bytes
-required to store a `coroutine frame`_.
+required to store a `coroutine frame`_.  This is only supported for
+switched-resume coroutines.
 
 Arguments:
 """"""""""
@@ -772,7 +886,8 @@ The first argument is a token returned by a call to '``llvm.coro.id``'
 identifying the coroutine.
 
 The second argument is a pointer to a block of memory where coroutine frame
-will be stored if it is allocated dynamically.
+will be stored if it is allocated dynamically.  This pointer is ignored
+for returned-continuation coroutines.
 
 Semantics:
 """"""""""
@@ -798,7 +913,8 @@ Overview:
 
 The '``llvm.coro.free``' intrinsic returns a pointer to a block of memory where 
 coroutine frame is stored or `null` if this instance of a coroutine did not use
-dynamically allocated memory for its coroutine frame.
+dynamically allocated memory for its coroutine frame.  This intrinsic is not
+supported for returned-continuation coroutines.
 
 Arguments:
 """"""""""
@@ -847,6 +963,7 @@ Overview:
 
 The '``llvm.coro.alloc``' intrinsic returns `true` if dynamic allocation is
 required to obtain a memory for the coroutine frame and `false` otherwise.
+This is not supported for returned-continuation coroutines.
 
 Arguments:
 """"""""""
@@ -944,7 +1061,8 @@ coroutine frame.
 Overview:
 """""""""
 
-The '``llvm.coro.id``' intrinsic returns a token identifying a coroutine.
+The '``llvm.coro.id``' intrinsic returns a token identifying a
+switched-resume coroutine.
 
 Arguments:
 """"""""""
@@ -975,6 +1093,79 @@ duplicated.
 
 A frontend should emit exactly one `coro.id` intrinsic per coroutine.
 
+.. _coro.id.retcon:
+
+'llvm.coro.id.retcon' Intrinsic
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+::
+
+  declare token @llvm.coro.id.retcon(i32 <size>, i32 <align>, i8* <buffer>,
+                                     i8* <continuation prototype>,
+                                     i8* <alloc>, i8* <dealloc>)
+
+Overview:
+"""""""""
+
+The '``llvm.coro.id.retcon``' intrinsic returns a token identifying a
+multiple-suspend returned-continuation coroutine.
+
+The 'result-type sequence' of the coroutine is defined as follows:
+
+- if the return type of the coroutine function is ``void``, it is the
+  empty sequence;
+
+- if the return type of the coroutine function is a ``struct``, it is the
+  element types of that ``struct`` in order;
+
+- otherwise, it is just the return type of the coroutine function.
+
+The first element of the result-type sequence must be a pointer type;
+continuation functions will be coerced to this type.  The rest of
+the sequence are the 'yield types', and any suspends in the coroutine
+must take arguments of these types.
+
+Arguments:
+""""""""""
+
+The first and second arguments are the expected size and alignment of
+the buffer provided as the third argument.  They must be constant.
+
+The fourth argument must be a reference to a global function, called
+the 'continuation prototype function'.  The type, calling convention,
+and attributes of any continuation functions will be taken from this
+declaration.  The return type of the prototype function must match the
+return type of the current function.  The first parameter type must be
+a pointer type.  The second parameter type must be an integer type;
+it will be used only as a boolean flag.
+
+The fifth argument must be a reference to a global function that will
+be used to allocate memory.  It may not fail, either by returning null
+or throwing an exception.  It must take an integer and return a pointer.
+
+The sixth argument must be a reference to a global function that will
+be used to deallocate memory.  It must take a pointer and return ``void``.
+
+'llvm.coro.id.retcon.once' Intrinsic
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+::
+
+  declare token @llvm.coro.id.retcon.once(i32 <size>, i32 <align>, i8* <buffer>,
+                                          i8* <prototype>,
+                                          i8* <alloc>, i8* <dealloc>)
+
+Overview:
+"""""""""
+
+The '``llvm.coro.id.retcon.once``' intrinsic returns a token identifying a
+unique-suspend returned-continuation coroutine.
+
+Arguments:
+""""""""""
+
+As for ``llvm.core.id.retcon``, except that the return type of the
+continuation prototype must be `void` instead of matching the
+coroutine's return type.
+
 .. _coro.end:
 
 'llvm.coro.end' Intrinsic
@@ -1008,6 +1199,17 @@ The purpose of this intrinsic is to allow frontends to mark the cleanup and
 other code that is only relevant during the initial invocation of the coroutine
 and should not be present in resume and destroy parts. 
 
+In returned-continuation lowering, ``llvm.coro.end`` fully destroys the
+coroutine frame.  If the second argument is `false`, it also returns from
+the coroutine with a null continuation pointer, and the next instruction
+will be unreachable.  If the second argument is `true`, it falls through
+so that the following logic can resume unwinding.  In a yield-once
+coroutine, reaching a non-unwind ``llvm.coro.end`` without having first
+reached a ``llvm.coro.suspend.retcon`` has undefined behavior.
+
+The remainder of this section describes the behavior under switched-resume
+lowering.
+
 This intrinsic is lowered when a coroutine is split into
 the start, resume and destroy parts. In the start part, it is a no-op,
 in resume and destroy parts, it is replaced with `ret void` instruction and
@@ -1078,11 +1280,11 @@ The following table summarizes the handling of `coro.end`_ intrinsic.
 Overview:
 """""""""
 
-The '``llvm.coro.suspend``' marks the point where execution of the coroutine 
-need to get suspended and control returned back to the caller.
-Conditional branches consuming the result of this intrinsic lead to basic blocks
-where coroutine should proceed when suspended (-1), resumed (0) or destroyed 
-(1).
+The '``llvm.coro.suspend``' marks the point where execution of a
+switched-resume coroutine is suspended and control is returned back
+to the caller.  Conditional branches consuming the result of this
+intrinsic lead to basic blocks where coroutine should proceed when
+suspended (-1), resumed (0) or destroyed (1).
 
 Arguments:
 """"""""""
@@ -1178,6 +1380,45 @@ to the coroutine:
     switch i8 %suspend1, label %suspend [i8 0, label %resume1
                                          i8 1, label %cleanup]
 
+.. _coro.suspend.retcon:
+
+'llvm.coro.suspend.retcon' Intrinsic
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+::
+
+  declare i1 @llvm.coro.suspend.retcon(...)
+
+Overview:
+"""""""""
+
+The '``llvm.coro.suspend.retcon``' intrinsic marks the point where
+execution of a returned-continuation coroutine is suspended and control
+is returned back to the caller.
+
+`llvm.coro.suspend.retcon`` does not support separate save points;
+they are not useful when the continuation function is not locally
+accessible.  That would be a more appropriate feature for a ``passcon``
+lowering that is not yet implemented.
+
+Arguments:
+""""""""""
+
+The types of the arguments must exactly match the yielded-types sequence
+of the coroutine.  They will be turned into return values from the ramp
+and continuation functions, along with the next continuation function.
+
+Semantics:
+""""""""""
+
+The result of the intrinsic indicates whether the coroutine should resume
+abnormally (non-zero).
+
+In a normal coroutine, it is undefined behavior if the coroutine executes
+a call to ``llvm.coro.suspend.retcon`` after resuming abnormally.
+
+In a yield-once coroutine, it is undefined behavior if the coroutine
+executes a call to ``llvm.coro.suspend.retcon`` after resuming in any way.
+
 .. _coro.param:
 
 'llvm.coro.param' Intrinsic