SFrame for C++ coroutine stack traces

[Jens Remus]

On 1/22/2026 6:09 PM, Mehrdad Niknami via Binutils wrote:

> Briefly: we have been working on incorporating C++ coroutine frames into
> stack traces.
> The particular issue we have been trying to solve is that many common tools
> that obtain stack traces (e.g., perf) do not appear to have a mechanism for
> discovering or traversing C++ coroutine frames.
> It was suggested to us that SFrames could perhaps facilitate this across
> many tools, and so we're reaching out here to discuss whether/how that may
> be achievable, as we are not too familiar with SFrame details.
> 
> As C++ coroutines have a very high amount of complexity, I have attempted
> to distill down the relevant details of our current design below, along
> with a quick primer on how C++ coroutines work from the user side.
> 
> Overall, we're wondering if SFrames (either as they exist now, or via a
> practical extension) would be suitable for this "stitching" or
> "interleaving" of coroutine stack frames, and if so, what we could plan for
> and/or expect on that front.
> 
> Thank you.
> 
> ------------------------------
> A toy coroutine may look like this:
> 
> Task<int> factorial(int n) {
> 
>   printf("n = %d\n", n);
> 
>   if (n == 0) {
>      co_return 1;
>   }
> 
>   int subresult = co_await factorial(n - 1);
> 
>   co_return n * subresult;
> }
> 
> The Task type internally holds a std::coroutine_handle<> object, which is
> basically an opaque void* pointing to the various bookkeeping data of the
> coroutine.
> The Task type is specially marked to signify to the compiler that this
> function is a coroutine, and defines various behaviors, such as those of
> the keywords co_await and co_return (e.g., by storing the provided values
> in heap memory somewhere for later retrieval).
> 
> Ordinarily, if we have
> 
> int main() {
> 
>   factorial(10).Resume();  // Run the coroutine
> 
> }
> 
> and if we then stop the program to obtain a stack trace arbitrarily, we may
> obtain a typical CPU stack trace such as this:
> 
> printf()
> factorial(n=3)
> 
> *(Note that I'm including the n= values for illustrative purposes; they of
> course would not be apparent solely from a list of instruction addresses.)*
> 
> std::coroutine_handle<>::resume()
> 
> *(Note that the std::coroutine_handle<>::resume()
> <https://en.cppreference.com/w/cpp/coroutine/coroutine_handle/resume.html>
> function
> uses a compiler intrinsic such as __builtin_coro_resume()
> <https://clang.llvm.org/docs/LanguageExtensions.html#c-coroutines-support-builtins>
> to
> resume the coroutine underneath.)*
> 
> Task::Resume()
> main()
> 
> 
> However, what we *wish* to obtain in such a situation is the following:
> 
> printf()
> factorial(n=3)
> factorial(n=2) (.resume)
> factorial(n=1) (.resume)
> std::coroutine_handle<>::resume()
> 
> Task<>::Resume()
> main()
> 
> 
> The information required for observing factorial(2) and factorial(1) does
> exist in memory, but it is not part of the ordinary thread call stack.
> In our implementation, it is represented via a singly-linked list:
> 
>    1. The currently-running coroutine (in this case, factorial(3)), which
>    is the head of the list, is maintained in a known *thread-local*
>    variable.
>    2. Each coroutine also remembers the address of its caller (a.k.a.
>    waiter, a.k.a. continuation).
> 
> Typically, the address of the calling coroutine is stored inside a
> so-called "Promise" object on the heap associated with the Task, whose
> address is obtained via invoking std::coroutine_handle<T>::promise() on the
> coroutine handle embedded inside the Task.
> Each coroutine saves/restores the handle to its caller at various points
> via hooks provided through the language & compiler.
> When the compiler asks the coroutine "*Now that you are suspended, what
> should I resume next?*" (via complicated machinery
> <https://en.cppreference.com/w/cpp/language/coroutines.html#:~:text=if%20await_suspend%20returns%20a%20coroutine%20handle%20for%20some%20other%20coroutine%2C%20that%20handle%20is%20resumed>),
> the coroutine responds with the handle of its caller (the continuation).

Would it work if factorial() would get annotated as coroutine, e.g.
using a new CFI directive .cfi_coroutine[_frame] (similar to
.cfi_signal_frame) emitted by the compiler, so that e.g. a SFrame stack
tracer would then invoke a coroutine stack tracer before continuing
as usual?

The SFrame stack tracer would produce (only the IP's of course):
printf()
factorial(n=3)   // coroutine frame [*]

The then called coroutine stack tracer would produce:
factorial(n=2) (.resume)
factorial(n=1) (.resume)

Then the SFrame stack tracer would continue to produce:
std::coroutine_handle<>::resume()
Task<>::Resume()
main()

[*]: Actually this would only get detected while "unwinding" from this
     IP.  I annotated it here for easier understanding of the concept.

> The missing piece of the puzzle is: how would the interleaving process
> figure out *where* to insert the coroutine frames into the ordinary stack
> trace?
> Conceptually, we need some way to identify Task::Resume() in the stack
> trace, and insert everything beneath the std::coroutine_handle<>::resume()
> that it is eventually calling downstream.

If it is preferable to identify where to insert the missing pieces then
e.g. Task::Resume() could get annotated in a similar way.

> 
> Right now we accomplish this via some dynamic probing at program
> initialization:
> 
>    1. We block the inlining of Task::Resume()
>    2. We force the inlining of std::coroutine_handle<>::resume() (which
>    just reduces to an intrinsic, __builtin_coro_resume())
>    3. We call a dummy coroutine whose sole job is to return its own
>    __builtin_return_address(0).
> 
> This return address is in the body of Task<>::Resume, and this forces it to
> be the unique address in the entire program that all Task<T> coroutines
> eventually return to.
> Therefore, we perform a linear search for it starting from the leaf frame
> on the stack, and we insert all the asynchronous frames after that.
Regards,
Jens
-- 
Jens Remus
Linux on Z Development (D3303)
jremus@de.ibm.com / jremus@linux.ibm.com

IBM Deutschland Research & Development GmbH; Vorsitzender des Aufsichtsrats: Wolfgang Wendt; Geschäftsführung: David Faller; Sitz der Gesellschaft: Ehningen; Registergericht: Amtsgericht Stuttgart, HRB 243294
IBM Data Privacy Statement: https://www.ibm.com/privacy/