GNU Tools Cauldron SFrame talk notes

[Indu Bhagat]

On 10/25/25 5:46 AM, Jose E. Marchesi wrote:
> 
>>> On Oct 9, 2025, at 8:14, Indu Bhagat <indu.bhagat@oracle.com> wrote:
>>>
>>> Hi,
>>>
>>> A brief summary of two SFrame-related items discussed at GNU Tools Cauldron
>>> 2025 (talk on "SFrame for effective userspace stack tracing"
>>> https://conf.gnu-tools-cauldron.org/opo25/talk/ADVMH7/). More details on SFrame
>>> stack trace format can be found on the wiki [1].  Thanks for your inputs during
>>> the session and all the hallway chats.
>>>
>>> * Handling in SFrame unaware linkers
>>>
>>>   For the case of distro-wide enablement where the assembler generates an
>>>   SFrame section, but the used linker does not support merging of SFrame
>>>   sections, following link-time failures will be seen:
>>>
>>>     error: relocation refers to a discarded section: XYZ
>>>
>>>   The error is in accordance with the ELF specification (handling of unknown
>>>   section type), and the eventual resolution is to support merging of SFrame
>>>   sections.
>>>
>>>   As per the ELF specification
>>> (https://gabi.xinuos.com/elf/03-sheader.html#rules-for-linking-unrecognized-sections):
>>>    - "Unrecognized sections that do not have the SHF_OS_NONCONFORMING attribute,
>>>      are combined in a two-phase process."
>>>    - "In the first phase, input sections that match in name, type and attribute
>>>      flags should be concatenated into single sections."
>>>
>>>   We discussed how this default behaviour proves to be a limitation to allow
>>>   new section types (SFrame being one example), where
>>>    - the new section type's layout is such that concatenated output is not
>>>      meaningful/useful.
>>>    - not all link-editors may want to deal with the new section type.
>>>
>>>   It may be worthwhile to explore the possibility of adjusting the ELF
>>>   specification to have SFrame sections be marked with a new flag, such that
>>>   the default linker behavior of concatenating the sections can be skipped.
>>>
>>>   The above issue is a blocker for SFrame enablement on some distros. Further,
>>>   some distros reportedly have as many as 8 linkers.  If ELF can be adjusted, the
>>>   currently SFrame-unaware linkers can discard the SFrame section.
>>>
>>>   After the discussion at Cauldron, I think the following addition to ELF may
>>>   work:
>>>
>>>   SHF_OS_NONCONFORMING_DISCARD (new flag)
>>>   This section requires special OS-specific processing (beyond the standard
>>>   linking rules) to avoid incorrect behavior. If this section has either an
>>>   sh_type value or contains sh_flags bits in the OS-specific ranges for those
>>>   fields, and a link editor processing this section does not recognize those
>>>   values, then the link editor should discard the section without an error.
>>>
>>>   WRT relocations, I think the following may work (does this need to be
>>>   specified?):
>>>   Any section linked via sh_link must also be discarded.  Sections with flag
>>>   SHF_OS_NONCONFORMING_DISCARD cannot have any relocations referring to these
>>>   sections from outside of the section.
>>>
>>>   WDYT ?
>>>
>>> * Linker-friendly SFrame section layout
>>>
>>>   We also discussed the question of whether it is possible to make SFrame
>>>   handling linker-friendly.
>>>
>>>   One of the possible options was to split SFrame section so that link-editors
>>>   can assemble final section without needing to understand the contents of the
>>>   section.
>>>
>>>   Separate SFrame data into three sections
>>>   - SFrame header (.sframe_header)
>>>     flags = SHF_ALLOC
>>>   - SFrame function index (FDEs sorted on start PC) (say, .sframe_idx),
>>>     flags = SHF_LINK_ORDER | SHF_ALLOC
>>>   - SFrame FRE data (concatenable, say .sframe_info)
>>>     flags = SHF_ALLOC
>>>
>>>   Use linker script to place the header first, followed by .sframe_idx, and
>>>   finally .sframe_info.  For linkers which work without linker script, they
>>>   usually assume sections are placed in alphabetical order.  But since we need
>>>   the linker script to emit some vital section (header) data like number of FDEs,
>>>   using linker scripts has a major drawback that SFrame will simply be
>>>   unsupported on some linkers.  Its also not clear how/where to emit one header
>>>   with all the necessary bits like number of FDEs and other offsets. Getting rid
>>>   of the header or duplicating the header SFrame FRE data is a no-go:
>>>    - SFrame header holds vital information like version, flags and other
>>>      provisions necessary for format evolution
>>>    - It is wasteful to duplicate it for each set of function FREs (in
>>>      .sframe_info).  It will bloat up the format considerably.
>>>
>>>   So, it seems the general advise was to not rely on linker script mechanism.
>>>   Above all, making the layout linker-friendly with this approach adversely
>>>   affects the efficacy of the format.
>>
>> In light of recent discussions, I was re-reading the thread and thought this
>> might be worth revisiting.
>>
>> The advantage of using existing linker constructs is that you have a clear
>> contract between SFrame and the linker. The linker is not in charge of
>> providing compat; if compat works through that linker construct, then you have
>> forward and backward compat.
> 
> That is clearly the best possible outcome.
> 
> I think we all agree in that it would be great if we could come with a
> concatenable format for SFrame, _provided_ it stays simple, compact and
> expressive enough for the requirements of the kernel userland unwinder
> and similar clients.
> 
>> Merging the header is indeed harder. I imagine that one would take the maximum
>> version across all object files and union of the flags, but AFAIK there are no
>> ELF constructs for that (closest is IMAGE_COMDAT_SELECT_LARGEST in PE). So it
>> seems reasonable to have custom merging for the header only. I strongly suggest
>> specifying the linker contract (i.e. merging rules) in the spec, e.g. how each
>> field should be merged, to avoid the slippery slope of “linkers DTRT” (which is
>> not possible if you don’t specify DTRT before upgrading the spec).
> 
> I think it would be possible to have a concatenable header if it would
> be defined in terms of a variable-length sequence of tags.  Something
> like:
> 
>        .sframe_hdr
>     +------+---...--+
>     | tag  |  value |
>     +------+--...---+
>     | tag  |  value |
>     +------+--...---+
>           ...
> 
> With tags like:
> 
>    version (length 1), flags (length 4), abi_arch (length 1),
>    fixed_fp_offset (length 1)
> 
> i.e. the stuff we currently have in struct sframe_header.  This way the
> concatenation of several headers (sets of tags) would be a valid header
> (the union set of tags).  It is a flexible layout when it comes to add
> new tags in future versions of the format.
> 
> This would obviously complicate the client side to some extent... the
> clients would need to use little loops in order to fetch tag values from
> headers, checking for integrity on the way: all version tags in the
> header shall be the same, the different flags shall be coherent, etc.
> It would be important to determine whether this extra complexity would
> be assumible by the existing clients wanting to use the format.  The
> presence of variable-length structures in the format may make it
> difficult (or even impossible) to process SFrame with verifiable BPF for
> example.  Also, along with a pointer to the header you would need its
> size.
> 
> But lets assume we indeed have concatenable headers like sketched above
> or by some other mean.  Then we would need to split the sframe
> generation in the assembler into two sections:
> 
>     .sframe_hdr   -> concatenable header
>     .sframe       -> set of FDEs followed by a set of FREs
> 
> Like .eh_frame, the single .sframe section contains data for all text
> sections in the object.  Then AFAIK at least two issues would remain:
> 
> 1. An explicit index generated at link-time will be necessary.  In the
>     current SFrame design, the index is implicit since the FDEs are well
>     demarcated (by the header) and ordered, so clients can just binary
>     search there.
> 
>     This will still require linker awareness.
> 
> 2. Input text sections covered by the unique .sframe may be discarded,
>     leading to an error.
> 
>     Roland just suggested a possible solution for this based on section
>     groups:
>     https://groups.google.com/g/generic-abi/c/3ZMVJDF79g8/m/udrN4dc3AAAJ
> 
>     Alternatively we could have the Sframe to somehow refer to the
>     covered sections and make the linker using that info.  This would
>     require linker awareness.
> 
> Any idea on how to handle these issues, while keeping the format
> suitable for its intended usage?
> 

Concatenating header will increase the size, especially because the 
approach also needs one .sframe per .text* section.

What we need de-duped SFrame headers.  In .sframe.hdr, we will still 
want to keep the two offsets (offset to SFrame FDE index and offset of 
SFrame FREs).

Encoding de-dup for .sframe.hdr means using a flag of SHF_MERGE, but I 
think the two offsets will come in the way.  As per ELF specification: 
"Note that any relocatable values, including values that would result in 
run-time relocations, must be analyzed to determine whether the run-time 
values would actually be identical."  We will need to manually edit 
these values I think.  Most importantly, the index creation is being 
deferred to the stack tracer (see next).

For ensuring index can be built using existing ELF mechanisms, we will 
actually need three sections:
   - .sframe.hdr SHF_ALLOC
   - .sframe.fdes SHF_ALLOC | SHF_LINK_ORDER
   - .sframe.fres SHF_ALLOC
emitted in a section group with the corresponding .text.*

Consider the case when SFrame V3, and V4 sections are to be merged.  The 
set of SFrame V3 FDEs will need to be kept separate from SFrame V4 FDEs 
(with each .sframe.hdr pointing to the start addr of the respective set 
of SFrame FDE index).  A consumer of this section needs to recreate the 
final index for binary search; this is undesirable.

>> Tatsuyuki
>>
>>>
>>> [1] SFrame wiki:https://sourceware.org/binutils/wiki/sframe
>>>
>>>