GNU Tools Cauldron SFrame talk notes

[Indu Bhagat]

On 10/10/25 9:50 AM, Fangrui Song wrote:
> On Fri, Oct 10, 2025 at 1:09 AM Indu Bhagat <indu.bhagat@oracle.com> wrote:
>>
>> On 10/8/25 11:20 PM, Fangrui Song wrote:
>>> Hi Indu,
>>>
>>> Thanks for the excellent summary and for tagging me.
>>>
>>> On Wed, Oct 8, 2025 at 4:15 PM Indu Bhagat <indu.bhagat@oracle.com> wrote:
>>>>
>>>> Hi,
>>>>
>>>> A brief summary of two SFrame-related items discussed at GNU Tools Cauldron
>>>> 2025 (talk on "SFrame for effective userspace stack tracing"
>>>> https://conf.gnu-tools-cauldron.org/opo25/talk/ADVMH7/). More details on
>>>> SFrame
>>>> stack trace format can be found on the wiki [1].  Thanks for your inputs
>>>> during
>>>> the session and all the hallway chats.
>>>>
>>>> * Handling in SFrame unaware linkers
>>>>
>>>>      For the case of distro-wide enablement where the assembler generates an
>>>>      SFrame section, but the used linker does not support merging of SFrame
>>>>      sections, following link-time failures will be seen:
>>>>
>>>>        error: relocation refers to a discarded section: XYZ
>>>>
>>>>      The error is in accordance with the ELF specification (handling of
>>>> unknown
>>>>      section type), and the eventual resolution is to support merging of
>>>> SFrame
>>>>      sections.
>>>>
>>>>      As per the ELF specification
>>>>
>>>> (https://gabi.xinuos.com/elf/03-sheader.html#rules-for-linking-unrecognized-sections):
>>>>       - "Unrecognized sections that do not have the SHF_OS_NONCONFORMING
>>>> attribute,
>>>>         are combined in a two-phase process."
>>>>       - "In the first phase, input sections that match in name, type and
>>>> attribute
>>>>         flags should be concatenated into single sections."
>>>>
>>>>      We discussed how this default behaviour proves to be a limitation to
>>>> allow
>>>>      new section types (SFrame being one example), where
>>>>       - the new section type's layout is such that concatenated output is not
>>>>         meaningful/useful.
>>>>       - not all link-editors may want to deal with the new section type.
>>>
>>> We should not treat concatenated output as not meaningful/useful.
>>>
>>> Currently, Binutils enforces a single-element structure within each
>>> .sframe section, regardless of whether it resides in a relocatable
>>> object or final executable. This approach differs from DWARF sections,
>>> which support multiple concatenated elements, each with its own header
>>> and body.
>>>
>>> This design choice stems from Linux kernel requirements, where kernel
>>> modules are relocatable files created with ld -r. The kernel's SFrame
>>> support expects each module to contain a single indexed format for
>>> efficient runtime processing. Consequently, GNU ld merges all input
>>> .sframe sections into a single indexed element, even when producing
>>> relocatable files. This behavior deviates from standard relocatable
>>> linking conventions that suppress synthetic section finalization.
>>>
>>
>> Hi Fangrui,
>>
>> Kernel relocatable files is not the sole reason of the design choice of
>> having an index.  Without an index, stacktracers will have a run-time
>> hit of parsing the section to create an index or resorting to linear
>> scan; neither of these is acceptable for performance reasons.
> 
> But kernel relocatable files is the sole reason for building an index
> even for ld -r,
> deviating from the convention that relocatable linking should skip
> section finalization.
> 
> Binutils’ SFrame v3 TODO page
> https://sourceware.org/binutils/wiki/sframe/sframev3todo currently
> specifies
> 
>> The updated linker will not, however, continue to link SFrame V2 sections.
>> The linker will not automatically upgrade input SFrame V2 sections to output a SFrame V3 section.
> 
> While folks might reconsider supporting mix-and-match v2/v3 in GNU ld,
> that would impose a very significant burden on other linker
> implementations.
> 
> If input files do have .sframe of different versions, should the
> linker build an index for v3 sections while concatenating v2 sections?
> Neither reporting an error nor ignoring v2 sections is ideal, since
> users may have a toolchain with v3 support but some prebuilt v2 object
> files.
> 

The intent to start with V3 for userspace packages.

The above behavior (linker not automatically upgrading to a more recent 
version) will need to stop once the V3 version is picked up for 
userspace.  As you suggest, neither ignoring nor reporting error with V3 
(or higher) will be acceptable.

> Considering that linkers might not support automatic upgrade and
> mix-and-match, they should concatenate sections by default rather than
> building an index unless explicitly requested, even for executable and
> shared object linking. This aligns with the --gdb-index and
> --debug-names behavior.
> 

I would like to remind that SFrame is a very simple format.

FWIW, in hindsight, the format changes from V1 to V2 _were_ simple 
enough that we could have offered silent upgrade, but for ease of 
implementation[*], V1 was obsoleted.  Similarly SFrame V2 errata 1 
release, I'd argue that the fix itself was simple (addition of 
SFRAME_F_FDE_FUNC_START_PCREL).

[*] The interfaces in libsframe were too tied to the on-disk 
representation of the format; something that a library like libsframe 
should have avoided in hindsight.  But this does not translate to the 
format changes being complicated.

>>> The fundamental design issue lies in making linker merging mandatory.
>>> For optimal portability, unwinders should support multiple-element
>>> structures within a .sframe section. When a linker builds an index for
>>> .sframe, it should be viewed as an optimization that relieves the
>>> unwinder from constructing its own index at runtime. This index
>>> construction should remain optional rather than required. While the
>>> SFRAME_F_FDE_SORTED flag can be cleared to permit unsorted FDEs,
>>> current unwinder implementations do not seem to support multiple
>>> elements in a single section.
>>>
>>
>> Constructing index at runtime will hurt.  Kernel's stacktracing of
>> userspace stacks will be done on the return to user path, and cannot
>> take that sort of performance hit.
>>
>> Its not just the kernel, building the index at run-time will hurt _all_
>> other profiling and debugging use-cases where fast stack traces are
>> desirable.
> 
> Sure, but a slow patch supporting multiple-element structure within a
> .sframe section
> should be supported for mix-and-match scenarios, or linkers that do
> not intend to support the build-an-index feature.
> 
> This aligns with how gdb loads debug info and .gdb_index
> https://sourceware.org/gdb/current/onlinedocs/gdb.html/Index-Files.html
> 

With gdb-index, its already a realization that creating these indexes at 
runtime can be a huge slow down.  Isnt it? Do we really want to go down 
that route again for stack tracing ?  Can Stack tracing, when not for 
offline debugging, afford to suffer such slow downs ?

So you see, we are already converging to say that for this format to be 
useful, implementing --sframe-index (or merging the sections in some 
form) is _necessary_.

>> Building the index with GNU gold
>>
>> The GNU gold linker can write the index at link time into the resulting ELF file, by passing the --gdb-index flag to gold.
>>
>> This is especially helpful if you intend to build a program and immediately run it under GDB. You may find that it is faster overall for the linker to write the index while it has the relevant information in memory anyways, rather than re-reloading the data from disk with gdb-add-index, or debugging the program without an index at all, especially for large programs.
> 
> 
>>> A future version should distinguish between linking and execution views:
>>>
>>> - Linking view: Assemblers produce a simpler format, omitting
>>> index-specific metadata fields
>>> - Linkers concatenate .sframe input sections by default, consistent
>>> with DWARF and other metadata sections
>>> - A new --sframe-index option enables linkers to synthesize a
>>> .sframe_idx section containing the indexed format, analogous to
>>> --gdb-index and --debug-names. The linker builds .sframe_idx from
>>> input .sframe sections. To support the Linux kernel workflow (ld -r
>>> for kernel modules), ld -r --sframe-index must also generate the
>>> indexed format.
>>> - Linker scripts control placement using: .sframe_idx : {
>>> *(.sframe_idx) }. From the linker perspective, .sframe input sections
>>> have been replaced by the linker-synthesized .sframe_idx. This output
>>> section description places the .sframe_idx into the .sframe_idx output
>>> section.
>>>
>>> The linking view could omit index-specific metadata fields such as
>>> sfh_num_fdes, sfh_num_fres, sfh_fdeoff, and sfh_freoff.
>>>
>>> I've also posted the analysis here
>>> https://maskray.me/blog/2025-09-28-remarks-on-sframe#linking-and-execution-views
>>> (draft)
>>>
>>>>      It may be worthwhile to explore the possibility of adjusting the ELF
>>>>      specification to have SFrame sections be marked with a new flag, such
>>>> that
>>>>      the default linker behavior of concatenating the sections can be skipped.
>>>>
>>>>      The above issue is a blocker for SFrame enablement on some distros.
>>>> Further,
>>>>      some distros reportedly have as many as 8 linkers.  If ELF can be
>>>> adjusted, the
>>>>      currently SFrame-unaware linkers can discard the SFrame section.
>>>>
>>>>      After the discussion at Cauldron, I think the following addition to
>>>> ELF may
>>>>      work:
>>>>
>>>>      SHF_OS_NONCONFORMING_DISCARD (new flag)
>>>>      This section requires special OS-specific processing (beyond the standard
>>>>      linking rules) to avoid incorrect behavior. If this section has either an
>>>>      sh_type value or contains sh_flags bits in the OS-specific ranges for
>>>> those
>>>>      fields, and a link editor processing this section does not recognize
>>>> those
>>>>      values, then the link editor should discard the section without an error.
>>>>
>>>>      WRT relocations, I think the following may work (does this need to be
>>>>      specified?):
>>>>      Any section linked via sh_link must also be discarded.  Sections with
>>>> flag
>>>>      SHF_OS_NONCONFORMING_DISCARD cannot have any relocations referring to
>>>> these
>>>>      sections from outside of the section.
>>>>
>>>>      WDYT ?
>>>
>>> If .sframe consumers are made to support multiple concatenated
>>> elements - consistent with DWARF - we won't need a new ELF section
>>> flag.
>>>
>>>> * Linker-friendly SFrame section layout
>>>>
>>>>      We also discussed the question of whether it is possible to make SFrame
>>>>      handling linker-friendly.
>>>>
>>>>      One of the possible options was to split SFrame section so that
>>>> link-editors
>>>>      can assemble final section without needing to understand the contents
>>>> of the
>>>>      section.
>>>>
>>>>      Separate SFrame data into three sections
>>>>      - SFrame header (.sframe_header)
>>>>        flags = SHF_ALLOC
>>>>      - SFrame function index (FDEs sorted on start PC) (say, .sframe_idx),
>>>>        flags = SHF_LINK_ORDER | SHF_ALLOC
>>>>      - SFrame FRE data (concatenable, say .sframe_info)
>>>>        flags = SHF_ALLOC
>>>>
>>>>      Use linker script to place the header first, followed by .sframe_idx, and
>>>>      finally .sframe_info.  For linkers which work without linker script, they
>>>>      usually assume sections are placed in alphabetical order.  But since
>>>> we need
>>>>      the linker script to emit some vital section (header) data like
>>>> number of FDEs,
>>>>      using linker scripts has a major drawback that SFrame will simply be
>>>>      unsupported on some linkers.  Its also not clear how/where to emit
>>>> one header
>>>>      with all the necessary bits like number of FDEs and other offsets.
>>>> Getting rid
>>>>      of the header or duplicating the header SFrame FRE data is a no-go:
>>>>       - SFrame header holds vital information like version, flags and other
>>>>         provisions necessary for format evolution
>>>>       - It is wasteful to duplicate it for each set of function FREs (in
>>>>         .sframe_info).  It will bloat up the format considerably.
>>>
>>> If multiple concatenated elements are supported, having more sections
>>> is not necessary.
>>> It would introduce more complexity due to unclear dependencies among
>>> these sections, complexity requiring linker to synthesize multiple,
>>> inter-related sections.
>>>
>>>>      So, it seems the general advise was to not rely on linker script
>>>> mechanism.
>>>>      Above all, making the layout linker-friendly with this approach adversely
>>>>      affects the efficacy of the format.
>>>>
>>>> [1] SFrame wiki:https://sourceware.org/binutils/wiki/sframe
>>>>
>>>
>>> Adopting the .sframe linking view and .sframe_idx execution view
>>> doesn't require any special semantics to linker scripts.
>>>
>>> Linker scripts can be used to control placement using: .sframe_idx : {
>>> *(.sframe_idx) }.
>>>   From the linker perspective, .sframe input sections have been replaced
>>> by the linker-synthesized `.sframe_idx`, meaning the linker never
>>> actually see `.sframe` in input section descriptions.
>>> This is similar to .got : { *(.got) } where .got is linker-synthesized.
>>