SFrame for C++ coroutine stack traces

[Mehrdad Niknami]

Hi,

(Apologies in advance for the long email. This is also my first time using
the mailing list, so sorry in advance for any missteps.)

Briefly: we have been working on incorporating C++ coroutine frames into
stack traces.
The particular issue we have been trying to solve is that many common tools
that obtain stack traces (e.g., perf) do not appear to have a mechanism for
discovering or traversing C++ coroutine frames.
It was suggested to us that SFrames could perhaps facilitate this across
many tools, and so we're reaching out here to discuss whether/how that may
be achievable, as we are not too familiar with SFrame details.

As C++ coroutines have a very high amount of complexity, I have attempted
to distill down the relevant details of our current design below, along
with a quick primer on how C++ coroutines work from the user side.

Overall, we're wondering if SFrames (either as they exist now, or via a
practical extension) would be suitable for this "stitching" or
"interleaving" of coroutine stack frames, and if so, what we could plan for
and/or expect on that front.

Thank you.

------------------------------
A toy coroutine may look like this:

Task<int> factorial(int n) {

  printf("n = %d\n", n);

  if (n == 0) {
     co_return 1;
  }

  int subresult = co_await factorial(n - 1);

  co_return n * subresult;
}

The Task type internally holds a std::coroutine_handle<> object, which is
basically an opaque void* pointing to the various bookkeeping data of the
coroutine.
The Task type is specially marked to signify to the compiler that this
function is a coroutine, and defines various behaviors, such as those of
the keywords co_await and co_return (e.g., by storing the provided values
in heap memory somewhere for later retrieval).

Ordinarily, if we have

int main() {

  factorial(10).Resume();  // Run the coroutine

}

and if we then stop the program to obtain a stack trace arbitrarily, we may
obtain a typical CPU stack trace such as this:

printf()
factorial(n=3)

*(Note that I'm including the n= values for illustrative purposes; they of
course would not be apparent solely from a list of instruction addresses.)*

std::coroutine_handle<>::resume()

*(Note that the std::coroutine_handle<>::resume()
<https://en.cppreference.com/w/cpp/coroutine/coroutine_handle/resume.html>
function
uses a compiler intrinsic such as __builtin_coro_resume()
<https://clang.llvm.org/docs/LanguageExtensions.html#c-coroutines-support-builtins>
to
resume the coroutine underneath.)*

Task::Resume()
main()


However, what we *wish* to obtain in such a situation is the following:

printf()
factorial(n=3)
factorial(n=2) (.resume)
factorial(n=1) (.resume)
std::coroutine_handle<>::resume()

Task<>::Resume()
main()


The information required for observing factorial(2) and factorial(1) does
exist in memory, but it is not part of the ordinary thread call stack.
In our implementation, it is represented via a singly-linked list:

   1. The currently-running coroutine (in this case, factorial(3)), which
   is the head of the list, is maintained in a known *thread-local*
   variable.
   2. Each coroutine also remembers the address of its caller (a.k.a.
   waiter, a.k.a. continuation).

Typically, the address of the calling coroutine is stored inside a
so-called "Promise" object on the heap associated with the Task, whose
address is obtained via invoking std::coroutine_handle<T>::promise() on the
coroutine handle embedded inside the Task.
Each coroutine saves/restores the handle to its caller at various points
via hooks provided through the language & compiler.
When the compiler asks the coroutine "*Now that you are suspended, what
should I resume next?*" (via complicated machinery
<https://en.cppreference.com/w/cpp/language/coroutines.html#:~:text=if%20await_suspend%20returns%20a%20coroutine%20handle%20for%20some%20other%20coroutine%2C%20that%20handle%20is%20resumed>),
the coroutine responds with the handle of its caller (the continuation).

The missing piece of the puzzle is: how would the interleaving process
figure out *where* to insert the coroutine frames into the ordinary stack
trace?
Conceptually, we need some way to identify Task::Resume() in the stack
trace, and insert everything beneath the std::coroutine_handle<>::resume()
that it is eventually calling downstream.

Right now we accomplish this via some dynamic probing at program
initialization:

   1. We block the inlining of Task::Resume()
   2. We force the inlining of std::coroutine_handle<>::resume() (which
   just reduces to an intrinsic, __builtin_coro_resume())
   3. We call a dummy coroutine whose sole job is to return its own
   __builtin_return_address(0).

This return address is in the body of Task<>::Resume, and this forces it to
be the unique address in the entire program that all Task<T> coroutines
eventually return to.
Therefore, we perform a linear search for it starting from the leaf frame
on the stack, and we insert all the asynchronous frames after that.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://sourceware.org/pipermail/binutils/attachments/20260122/ef6b9ac2/attachment.htm>