[RFC 2/5] elf: Add SFrame stack tracing

[Jens Remus]

On 18.03.2025 14:03, claudiu.zissulescu-ianculescu@oracle.com wrote:
> From: Claudiu Zissulescu <claudiu.zissulescu-ianculescu@oracle.com>
> 
> This patch adds the necessary bits to enable stack tracing using
> SFrame.  In the case the new SFrame stack tracing procedure doesn't
> find SFrame related info, the stack tracing falls back on default
> Dwarf implementation.
> 
> The new SFrame stack tracing procedure is added to bebug/backtrace.c
> file, the support functions are added in sysdeps folder, namely
> sframe.h, read-sframe.c and read-sfame.h.
> 
> Signed-off-by: Claudiu Zissulescu <claudiu.zissulescu-ianculescu@oracle.com>

> diff --git a/debug/backtrace.c b/debug/backtrace.c

> @@ -20,6 +20,7 @@
>   #include <stdlib.h>
>   #include <unwind.h>
>   #include <unwind-link.h>
> +#include <sframe.h>
>   
>   struct trace_arg
>   {
> @@ -30,6 +31,38 @@ struct trace_arg
>     int size;
>   };
>   
> +/* Initialize the SFrame backtrace routine, and try to backtrace the
> +   current stack using SFrame info.  The return value of SFrame stack
> +   tracer must be larger than one to consider the SFrame backtrace
> +   valid. Otherwise, there may be the case that glibc is compiled
> +   using SFrame but the application not.   */

This is because DO_SFRAME_BACKTRACE() then managed to find SFrame
information for __backtrace() as the topmost and only frame in Glibc
and at least one further function from the application, right?

> +
> +#define DO_SFRAME_BACKTRACE(ARRAY, SIZE)			\
> +  do								\
> +    {								\
> +      int cnt;							\
> +      frame frame;						\
> +      frame.pc = getPC();					\
> +      frame.sp = getSP();					\
> +      frame.fp = (_Unwind_Ptr) __builtin_frame_address (0);	\

Why doesn't this require to be wrapped in a noinline helper getFP()?  If
there is some clever trickery involved , could you maybe add a comment?

> +      cnt = stacktrace_sframe (ARRAY, SIZE, &frame);		\
> +      if (cnt > 1)						\
> +	return cnt;						\
> +    }								\
> +  while(0)
> +
> +static _Unwind_Ptr  __attribute__ ((noinline))
> +getPC (void)
> +{
> +  return (_Unwind_Ptr) __builtin_return_address (0);

Clever to wrap this in a noinline helper, to use this builtin to obtain
the PC in __backchain() (at invocation of the helper).

Although not required on x86-64, AArch64, and s390x: Would it make sense
to use __builtin_extract_return_addr() in addition?

> +}
> +
> +static _Unwind_Ptr  __attribute__ ((noinline))
> +getSP (void)
> +{
> +  return (_Unwind_Ptr) __builtin_dwarf_cfa();

This only works for architectures that define their CFA as SP at
call site.  On s390x the CFA is defined as SP at call site +160.
I will have to make either this helber architecture dependent or
introduce an architecture dependent SP value offset from CFA (e.g.
SFRAME_SP_VAL_OFFSET).

> +}
> +
>   static _Unwind_Reason_Code
>   backtrace_helper (struct _Unwind_Context *ctx, void *a)
>   {
> @@ -72,7 +105,13 @@ __backtrace (void **array, int size)
>        .cnt = -1
>       };
>   
> -  if (size <= 0 || arg.unwind_link == NULL)
> +  if (size <= 0)
> +    return 0;
> +

It's kind of obvious, but maybe add the following comment for those that
do not know SFrame:

   /* Thy the SFrame stack tracer.  */

> +  DO_SFRAME_BACKTRACE (array, size);
> +
> +  /* Try the dwarf unwinder.  */
> +  if (arg.unwind_link == NULL)
>       return 0;
>   
>     UNWIND_LINK_PTR (arg.unwind_link, _Unwind_Backtrace)

> diff --git a/sysdeps/generic/sframe.c b/sysdeps/generic/sframe.c

> @@ -0,0 +1,140 @@
> +/* Copyright (C) 2025 Free Software Foundation, Inc.
> +   This file is part of the GNU C Library.
> +
> +   The GNU C Library is free software; you can redistribute it and/or
> +   modify it under the terms of the GNU Lesser General Public License as
> +   published by the Free Software Foundation; either version 2.1 of the
> +   License, or (at your option) any later version.
> +
> +   The GNU C Library is distributed in the hope that it will be useful,
> +   but WITHOUT ANY WARRANTY; without even the implied warranty of
> +   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
> +   Lesser General Public License for more details.
> +
> +   You should have received a copy of the GNU General Public License
> +   along with this program.  If not, see <http://www.gnu.org/licenses/>.  */
> +
> +#include <sframe-read.h>
> +#include <stdlib.h>
> +#include <dlfcn.h>
> +#include <unwind.h>
> +#include <uw-sigframe.h>
> +
> +#if DLFO_STRUCT_HAS_SFRAME

IIUC this makes Glibc backtrace() use of SFrame for stack tracing solely
depend on whether an architecture has enabled DLFO_STRUCT_HAS_SFRAME.
Both x86-64 and AArch64 enable this unconditionally with your subsequent
patches.

Wouldn't it make sense to not enable it if Glibc is not compiled with
SFrame stack trace information?  Otherwise __backtrace() would always
attempt to use SFrame first with stacktrace_sframe() failing to locate
any SFrame information for __backtrace() itself.

In addition I wonder whether it would be good to allow to build Glibc
with SFrame stack trace information but disable the use in __backtrace().

> +
> +static inline uint64_t
> +read_stack_value (uint64_t loc)
> +{
> +  uint64_t value = *((uint64_t *) loc);
> +  return value;
> +}
> +
> +/* Backtrace the stack and collect the stacktrace given SFrame info.
> +   If successful, store the return addresses in RA_LST. The SIZE
> +   argument specifies the maximum number of return addresses that can
> +   be stored in RA_LST and contains the number of the addresses
> +   collected.  */
> +
> +int
> +stacktrace_sframe (void **ra_lst, int count, frame *frame)
> +{
> +  _Unwind_Ptr sframe_vma, cfa, return_addr, ra_stack_loc, rfp_stack_loc, pc;
> +  int cfa_offset, rfp_offset, ra_offset, i;

Why rfp_stack_loc and rfp_offset instead of fp_stack_loc and fp_offset?

> +  sframe_frame_row_entry fred, *frep = &fred;
> +
> +  if (!ra_lst || !count)
> +    return 0;
> +
> +  for (i = 0; i < count; i++)
> +    {
> +      _Unwind_Reason_Code err;
> +      struct dl_find_object data;
> +      sframe_decoder_ctx decoder_context, *dctx = &decoder_context;
> +
> +      /* Clean decoder context.  */
> +      memset (dctx, 0, sizeof (sframe_decoder_ctx));
> +
> +      /* Load and set up the SFrame stack trace info for pc.  */
> +      if (_dl_find_object ((void *) frame->pc, &data) < 0)
> +	return i;
> +
> +      sframe_vma = (_Unwind_Ptr) data.dlfo_sframe;
> +      if (!sframe_vma)
> +	{
> +#ifdef MD_DECODE_SIGNAL_FRAME
> +	  /* I cannot find a valid SFrame section. Check if it is a
> +	     signal frame.  */
> +	  if (MD_DECODE_SIGNAL_FRAME (frame) == _URC_NO_REASON)
> +	    {
> +	      ra_lst[i] = (void *) frame->pc;
> +	      continue;
> +	    }
> +#endif
> +	  return i;
> +	}
> +
> +      /* Decode the specified SFrame buffer populate sframe's decoder
> +	 context.  */
> +      if (sframe_decode (dctx, (char *) data.dlfo_sframe) == _URC_END_OF_STACK)
> +	return i;
> +
> +      pc = frame->pc - sframe_vma;
> +      /* Find the SFrame Row Entry which contains the PC.  */
> +      if (sframe_find_fre (dctx, pc, frep) == _URC_END_OF_STACK)
> +	{
> +#ifdef MD_DECODE_SIGNAL_FRAME
> +	  /* I cannot find any FREs, try to see if it is a signal
> +	     frame, and if so decode it.  */
> +	  if (MD_DECODE_SIGNAL_FRAME (frame) == _URC_NO_REASON)
> +	    {
> +	      ra_lst[i] = (void *) frame->pc;
> +	      continue;
> +	    }
> +#endif
> +	  return i;
> +	}
> +
> +      /* Get the CFA offset from the FRE.  */
> +      cfa_offset = sframe_fre_get_cfa_offset (dctx, frep);
> +
> +      /* Get the base reg id from the FRE info.  */

The comment is misleading, as the computed value is the CFA.  Maybe:

       /* Get CFA using base reg id from the FRE info.  */

> +      cfa = ((sframe_fre_get_base_reg_id (frep)
> +	      == SFRAME_BASE_REG_SP) ? frame->sp : frame->fp) + cfa_offset;
> +
> +      /* Get the RA offset from the FRE.  */
> +      ra_offset = sframe_fre_get_ra_offset (dctx, frep);
> +
> +      ra_stack_loc = cfa + ra_offset;
> +      return_addr = read_stack_value (ra_stack_loc);

On s390x the RA (and FP) may be saved in registers in the topost frame,
e.g. when in a leaf function.  In that case s390x will encode the DWARF
register number in the SFrame RA/FP offset.  The good thing is that
neither the caller of backtrace() nor __backtrace() itself are leaf
functions, so that this special case may not occur in the this Glibc
use case.  I would need to add a check though, that the returned offset
is not an encoded DWARF register number on s390x.

> +
> +      ra_lst[i] = (void *)return_addr;
> +
> +      /* Set up for the next frame.  */
> +      /* Get the FP offset from the FRE.  If the offset is invalid,
> +	 sets errp.  */
> +      rfp_offset = sframe_fre_get_fp_offset (dctx, frep, &err);
> +      if (err == _URC_NO_REASON)
> +	{
> +	  /* Frame is valid, get the value stored in the stack
> +	     location.  */
> +	  rfp_stack_loc = cfa + rfp_offset;
> +	  frame->fp = read_stack_value (rfp_stack_loc);

Same as for RA above.

> +	}
> +
> +      frame->sp = cfa;

Same as for GetSP() above.  I guess introducing an architecture
dependent SP value offset from CFA (e.g. SFRAME_SP_VAL_OFFSET) would be
the easiest solution?  On architectures, such as x86-64 and AArch64,
where it would be defined to 0 the compiler would optimize the offset
away.  What do you think?

> +      frame->pc = return_addr;
> +    }
> +  return i;
> +}
> +#else

#else /* !DLFO_STRUCT_HAS_SFRAME  */

> +/* Dummy function called when SFrame is not available for a target.  */
> +int
> +stacktrace_sframe (void **ra_lst __attribute__ ((__unused__)),
> +		   int count __attribute__ ((__unused__)),
> +		   frame *frame __attribute__ ((__unused__)))
> +{
> +  return 0;
> +}
> +#endif /* DLFO_STRUCT_HAS_SFRAME  */
> +
> +libc_hidden_def (stacktrace_sframe)

> diff --git a/sysdeps/generic/sframe.h b/sysdeps/generic/sframe.h

> @@ -0,0 +1,359 @@
> +/* SFrame format description.
> +   Copyright (C) 2022-2025 Free Software Foundation, Inc.
> +   This file is part of the GNU C Library.
> +
> +   The GNU C Library is free software; you can redistribute it and/or
> +   modify it under the terms of the GNU Lesser General Public
> +   License as published by the Free Software Foundation; either
> +   version 2.1 of the License, or (at your option) any later version.
> +
> +   The GNU C Library is distributed in the hope that it will be useful,
> +   but WITHOUT ANY WARRANTY; without even the implied warranty of
> +   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
> +   Lesser General Public License for more details.
> +
> +   You should have received a copy of the GNU General Public License
> +   along with this program; see the file COPYING.  If not see
> +   <http://www.gnu.org/licenses/>.  */

Would it make sense to add a note that this is a copy of GNU Binutils
include/sframe.h with only minor adaptions and additions and therefore
should be kept in sync?

> +
> +#ifndef	_SFRAME_H
> +#define	_SFRAME_H

Thanks and regards,
Jens
-- 
Jens Remus
Linux on Z Development (D3303)
+49-7031-16-1128 Office
jremus@de.ibm.com

IBM

IBM Deutschland Research & Development GmbH; Vorsitzender des Aufsichtsrats: Wolfgang Wendt; Geschäftsführung: David Faller; Sitz der Gesellschaft: Böblingen; Registergericht: Amtsgericht Stuttgart, HRB 243294
IBM Data Privacy Statement: https://www.ibm.com/privacy/