GNU Tools Cauldron SFrame talk notes

[Indu Bhagat]

On 10/12/25 1:25 PM, Fangrui Song wrote:
> On Sun, Oct 12, 2025 at 12:42 PM Indu Bhagat <indu.bhagat@oracle.com> wrote:
>>
>> On 10/10/25 9:50 AM, Fangrui Song wrote:
>>> On Fri, Oct 10, 2025 at 1:09 AM Indu Bhagat <indu.bhagat@oracle.com> wrote:
>>>>
>>>> On 10/8/25 11:20 PM, Fangrui Song wrote:
>>>>> Hi Indu,
>>>>>
>>>>> Thanks for the excellent summary and for tagging me.
>>>>>
>>>>> On Wed, Oct 8, 2025 at 4:15 PM Indu Bhagat <indu.bhagat@oracle.com> wrote:
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> A brief summary of two SFrame-related items discussed at GNU Tools Cauldron
>>>>>> 2025 (talk on "SFrame for effective userspace stack tracing"
>>>>>> https://conf.gnu-tools-cauldron.org/opo25/talk/ADVMH7/). More details on
>>>>>> SFrame
>>>>>> stack trace format can be found on the wiki [1].  Thanks for your inputs
>>>>>> during
>>>>>> the session and all the hallway chats.
>>>>>>
>>>>>> * Handling in SFrame unaware linkers
>>>>>>
>>>>>>       For the case of distro-wide enablement where the assembler generates an
>>>>>>       SFrame section, but the used linker does not support merging of SFrame
>>>>>>       sections, following link-time failures will be seen:
>>>>>>
>>>>>>         error: relocation refers to a discarded section: XYZ
>>>>>>
>>>>>>       The error is in accordance with the ELF specification (handling of
>>>>>> unknown
>>>>>>       section type), and the eventual resolution is to support merging of
>>>>>> SFrame
>>>>>>       sections.
>>>>>>
>>>>>>       As per the ELF specification
>>>>>>
>>>>>> (https://gabi.xinuos.com/elf/03-sheader.html#rules-for-linking-unrecognized-sections):
>>>>>>        - "Unrecognized sections that do not have the SHF_OS_NONCONFORMING
>>>>>> attribute,
>>>>>>          are combined in a two-phase process."
>>>>>>        - "In the first phase, input sections that match in name, type and
>>>>>> attribute
>>>>>>          flags should be concatenated into single sections."
>>>>>>
>>>>>>       We discussed how this default behaviour proves to be a limitation to
>>>>>> allow
>>>>>>       new section types (SFrame being one example), where
>>>>>>        - the new section type's layout is such that concatenated output is not
>>>>>>          meaningful/useful.
>>>>>>        - not all link-editors may want to deal with the new section type.
>>>>>
>>>>> We should not treat concatenated output as not meaningful/useful.
>>>>>
>>>>> Currently, Binutils enforces a single-element structure within each
>>>>> .sframe section, regardless of whether it resides in a relocatable
>>>>> object or final executable. This approach differs from DWARF sections,
>>>>> which support multiple concatenated elements, each with its own header
>>>>> and body.
>>>>>
>>>>> This design choice stems from Linux kernel requirements, where kernel
>>>>> modules are relocatable files created with ld -r. The kernel's SFrame
>>>>> support expects each module to contain a single indexed format for
>>>>> efficient runtime processing. Consequently, GNU ld merges all input
>>>>> .sframe sections into a single indexed element, even when producing
>>>>> relocatable files. This behavior deviates from standard relocatable
>>>>> linking conventions that suppress synthetic section finalization.
>>>>>
>>>>
>>>> Hi Fangrui,
>>>>
>>>> Kernel relocatable files is not the sole reason of the design choice of
>>>> having an index.  Without an index, stacktracers will have a run-time
>>>> hit of parsing the section to create an index or resorting to linear
>>>> scan; neither of these is acceptable for performance reasons.
>>>
>>> But kernel relocatable files is the sole reason for building an index
>>> even for ld -r,
>>> deviating from the convention that relocatable linking should skip
>>> section finalization.
>>>
>>> Binutils’ SFrame v3 TODO page
>>> https://sourceware.org/binutils/wiki/sframe/sframev3todo currently
>>> specifies
>>>
>>>> The updated linker will not, however, continue to link SFrame V2 sections.
>>>> The linker will not automatically upgrade input SFrame V2 sections to output a SFrame V3 section.
>>>
>>> While folks might reconsider supporting mix-and-match v2/v3 in GNU ld,
>>> that would impose a very significant burden on other linker
>>> implementations.
>>>
>>> If input files do have .sframe of different versions, should the
>>> linker build an index for v3 sections while concatenating v2 sections?
>>> Neither reporting an error nor ignoring v2 sections is ideal, since
>>> users may have a toolchain with v3 support but some prebuilt v2 object
>>> files.
>>>
>>
>> The intent to start with V3 for userspace packages.
>>
>> The above behavior (linker not automatically upgrading to a more recent
>> version) will need to stop once the V3 version is picked up for
>> userspace.  As you suggest, neither ignoring nor reporting error with V3
>> (or higher) will be acceptable.
> 
> The only feasible approach for handling version mismatches is to concatenate
> .sframe sections by default. Consumers that require an index should request it
> explicitly via --sframe-index. When --sframe-index is used, the linker can
> report a warning for unrecognized versions (e.g., when the linker supports v3
> index building but doesn't want to retain support for v2).
> 
>>> Considering that linkers might not support automatic upgrade and
>>> mix-and-match, they should concatenate sections by default rather than
>>> building an index unless explicitly requested, even for executable and
>>> shared object linking. This aligns with the --gdb-index and
>>> --debug-names behavior.
>>>
>>
>> I would like to remind that SFrame is a very simple format.
>>
>> FWIW, in hindsight, the format changes from V1 to V2 _were_ simple
>> enough that we could have offered silent upgrade, but for ease of
>> implementation[*], V1 was obsoleted.  Similarly SFrame V2 errata 1
>> release, I'd argue that the fix itself was simple (addition of
>> SFRAME_F_FDE_FUNC_START_PCREL).
>>
>> [*] The interfaces in libsframe were too tied to the on-disk
>> representation of the format; something that a library like libsframe
>> should have avoided in hindsight.  But this does not translate to the
>> format changes being complicated.
> 
> While SFrame is simpler than compression or multimedia codecs, it's
> significantly more complex than ELF control structures and any other LLVM
> metadata section. The magnitude of changes from v2 to v3 is expected to be
> substantial, causing toolchain support churn if a toolchain decides to support
> v2 at this point.
> 
>>>>> The fundamental design issue lies in making linker merging mandatory.
>>>>> For optimal portability, unwinders should support multiple-element
>>>>> structures within a .sframe section. When a linker builds an index for
>>>>> .sframe, it should be viewed as an optimization that relieves the
>>>>> unwinder from constructing its own index at runtime. This index
>>>>> construction should remain optional rather than required. While the
>>>>> SFRAME_F_FDE_SORTED flag can be cleared to permit unsorted FDEs,
>>>>> current unwinder implementations do not seem to support multiple
>>>>> elements in a single section.
>>>>>
>>>>
>>>> Constructing index at runtime will hurt.  Kernel's stacktracing of
>>>> userspace stacks will be done on the return to user path, and cannot
>>>> take that sort of performance hit.
>>>>
>>>> Its not just the kernel, building the index at run-time will hurt _all_
>>>> other profiling and debugging use-cases where fast stack traces are
>>>> desirable.
>>>
>>> Sure, but a slow patch supporting multiple-element structure within a
>>> .sframe section
>>> should be supported for mix-and-match scenarios, or linkers that do
>>> not intend to support the build-an-index feature.
>>>
>>> This aligns with how gdb loads debug info and .gdb_index
>>> https://sourceware.org/gdb/current/onlinedocs/gdb.html/Index-Files.html
>>>
>>
>> With gdb-index, its already a realization that creating these indexes at
>> runtime can be a huge slow down.  Isnt it? Do we really want to go down
>> that route again for stack tracing ?  Can Stack tracing, when not for
>> offline debugging, afford to suffer such slow downs ?
> 
> I agree that performance is crucial for stack tracing. However, we
> need to distinguish between optimal performance and baseline functionality.
> 
> For optimal portability, we must support object files from diverse origins—not
> just those built from a single toolchain. 

Sure. Thats not forsaken with the current design.

> In environments where almost
> everything is built from source with a single toolchain offering strong SFrame
> support, forcing default-on index building may be acceptable. However, we must
> also accommodate environments with prebuilt object files using older SFrame
> versions, or toolchains that don't support old formats.
> 

RE: accommodate environments with prebuilt object files using older 
SFrame versions
Yes, surely. An SFrame aware linker will be able to do that without much 
difficulty.

RE: or toolchains that don't support old formats.
If an older version of the format is out in the wild, a well-intentioned 
tool provider will want to include support for older version of the 
format, yes.  Is that a problem ? Its subjective. That argument applies 
to _all_ old and new ELF features and philosophically, any forward 
looking innovation: supporting old(er) specification is a choice that 
needs to be made keeping users in mind.

I would like to underline that reading SFrame is simple enough: Being 
able to read _a_ version of SFrame (V1, V2, or V3 for matter) is not 
going to be wildly difficult if a previous or newer version is already 
intended to be supported. (a.k.a. implementation complexity is low).

> Forward compatibility is another important consideration. Many metadata sections
> can be used with very old linkers, which is a significant advantage
> for adoption.
> > The solution is to make the indexed format an optimization, not a 
requirement:
> 
> 1. Default behavior: Concatenate .sframe sections (baseline functionality)
> 2. Optimized behavior: Use --sframe-index to build indices (best performance)
> 3. Runtime fallback: Unwinders can handle concatenated sections when no index
>     is available
> 

As I mentioned previously, IMO having stack tracers to handle 
concatenated SFrame sections to perform stack tracing will not the right 
thing to do. It will be a performance hit; the whole point of SFrame is 
enable fast stack tracing.

> LLDB already supports this model for .debug_names, which can either be
> concatenated or an optimized index built by ld.lld --debug-names.
> 
> 
>> So you see, we are already converging to say that for this format to be
>> useful, implementing --sframe-index (or merging the sections in some
>> form) is _necessary_.
> 
> Thanks. I'm glad we agree that ld --sframe-index is necessary.
> 
> However, I must emphasize that this should be opt-in, not default behavior.
> Making index building opt-in provides several benefits:
> 

I am not convinced that --sframe-index as opt-in is the right choice.

Stack traces needing to create index by reading in the concatenated 
sections will be
  1. a performance hit; not recommended for the use case of fast stack 
tracing.
  2. duplicated functionality in Linux kernel, glibc and other usecases.

> 1. Linkers can support basic .sframe handling (concatenation) without
> implementing the full index-building logic
> 2. Mixed-version scenarios degrade gracefully to concatenation rather
> than failing
> 3. Consistent with precedent .debug_names
> 4. Distros can roll out SFrame support incrementally without requiring
> all linkers to support index building immediately
> 

While this looks pleasing and sits well with the status quo (linker's 
default behaviour of concatenating unknown sections etc.),  concatenated 
SFrame sections are effectively useless because of the performance hit 
of reading in the input sections and creating the index at the time of 
stack tracing.

Taking a step back though, I think we are mixing up a lot of different 
things here (backward/forward compatibility, mixing artifacts from two 
different toolchains etc).  The fundamental concern is: SFrame is a 
format that asks for merging to be as the default link-editor behavior. 
This does not align well with the existing ELF provisions.

I think revisiting ELF and its default behavior for unknown sections is 
sensible in this scenario. An unaware linker should then discard these 
sections until it gains the capability to merge SFrame sections, and not 
pass the buck to stack tracers to do the right thing.

>>>> Building the index with GNU gold
>>>>
>>>> The GNU gold linker can write the index at link time into the resulting ELF file, by passing the --gdb-index flag to gold.
>>>>
>>>> This is especially helpful if you intend to build a program and immediately run it under GDB. You may find that it is faster overall for the linker to write the index while it has the relevant information in memory anyways, rather than re-reloading the data from disk with gdb-add-index, or debugging the program without an index at all, especially for large programs.
>>>
>>>
>>>>> A future version should distinguish between linking and execution views:
>>>>>
>>>>> - Linking view: Assemblers produce a simpler format, omitting
>>>>> index-specific metadata fields
>>>>> - Linkers concatenate .sframe input sections by default, consistent
>>>>> with DWARF and other metadata sections
>>>>> - A new --sframe-index option enables linkers to synthesize a
>>>>> .sframe_idx section containing the indexed format, analogous to
>>>>> --gdb-index and --debug-names. The linker builds .sframe_idx from
>>>>> input .sframe sections. To support the Linux kernel workflow (ld -r
>>>>> for kernel modules), ld -r --sframe-index must also generate the
>>>>> indexed format.
>>>>> - Linker scripts control placement using: .sframe_idx : {
>>>>> *(.sframe_idx) }. From the linker perspective, .sframe input sections
>>>>> have been replaced by the linker-synthesized .sframe_idx. This output
>>>>> section description places the .sframe_idx into the .sframe_idx output
>>>>> section.
>>>>>
>>>>> The linking view could omit index-specific metadata fields such as
>>>>> sfh_num_fdes, sfh_num_fres, sfh_fdeoff, and sfh_freoff.
>>>>>
>>>>> I've also posted the analysis here
>>>>> https://maskray.me/blog/2025-09-28-remarks-on-sframe#linking-and-execution-views
>>>>> (draft)
>>>>>
>>>>>>       It may be worthwhile to explore the possibility of adjusting the ELF
>>>>>>       specification to have SFrame sections be marked with a new flag, such
>>>>>> that
>>>>>>       the default linker behavior of concatenating the sections can be skipped.
>>>>>>
>>>>>>       The above issue is a blocker for SFrame enablement on some distros.
>>>>>> Further,
>>>>>>       some distros reportedly have as many as 8 linkers.  If ELF can be
>>>>>> adjusted, the
>>>>>>       currently SFrame-unaware linkers can discard the SFrame section.
>>>>>>
>>>>>>       After the discussion at Cauldron, I think the following addition to
>>>>>> ELF may
>>>>>>       work:
>>>>>>
>>>>>>       SHF_OS_NONCONFORMING_DISCARD (new flag)
>>>>>>       This section requires special OS-specific processing (beyond the standard
>>>>>>       linking rules) to avoid incorrect behavior. If this section has either an
>>>>>>       sh_type value or contains sh_flags bits in the OS-specific ranges for
>>>>>> those
>>>>>>       fields, and a link editor processing this section does not recognize
>>>>>> those
>>>>>>       values, then the link editor should discard the section without an error.
>>>>>>
>>>>>>       WRT relocations, I think the following may work (does this need to be
>>>>>>       specified?):
>>>>>>       Any section linked via sh_link must also be discarded.  Sections with
>>>>>> flag
>>>>>>       SHF_OS_NONCONFORMING_DISCARD cannot have any relocations referring to
>>>>>> these
>>>>>>       sections from outside of the section.
>>>>>>
>>>>>>       WDYT ?
>>>>>
>>>>> If .sframe consumers are made to support multiple concatenated
>>>>> elements - consistent with DWARF - we won't need a new ELF section
>>>>> flag.
>>>>>
>>>>>> * Linker-friendly SFrame section layout
>>>>>>
>>>>>>       We also discussed the question of whether it is possible to make SFrame
>>>>>>       handling linker-friendly.
>>>>>>
>>>>>>       One of the possible options was to split SFrame section so that
>>>>>> link-editors
>>>>>>       can assemble final section without needing to understand the contents
>>>>>> of the
>>>>>>       section.
>>>>>>
>>>>>>       Separate SFrame data into three sections
>>>>>>       - SFrame header (.sframe_header)
>>>>>>         flags = SHF_ALLOC
>>>>>>       - SFrame function index (FDEs sorted on start PC) (say, .sframe_idx),
>>>>>>         flags = SHF_LINK_ORDER | SHF_ALLOC
>>>>>>       - SFrame FRE data (concatenable, say .sframe_info)
>>>>>>         flags = SHF_ALLOC
>>>>>>
>>>>>>       Use linker script to place the header first, followed by .sframe_idx, and
>>>>>>       finally .sframe_info.  For linkers which work without linker script, they
>>>>>>       usually assume sections are placed in alphabetical order.  But since
>>>>>> we need
>>>>>>       the linker script to emit some vital section (header) data like
>>>>>> number of FDEs,
>>>>>>       using linker scripts has a major drawback that SFrame will simply be
>>>>>>       unsupported on some linkers.  Its also not clear how/where to emit
>>>>>> one header
>>>>>>       with all the necessary bits like number of FDEs and other offsets.
>>>>>> Getting rid
>>>>>>       of the header or duplicating the header SFrame FRE data is a no-go:
>>>>>>        - SFrame header holds vital information like version, flags and other
>>>>>>          provisions necessary for format evolution
>>>>>>        - It is wasteful to duplicate it for each set of function FREs (in
>>>>>>          .sframe_info).  It will bloat up the format considerably.
>>>>>
>>>>> If multiple concatenated elements are supported, having more sections
>>>>> is not necessary.
>>>>> It would introduce more complexity due to unclear dependencies among
>>>>> these sections, complexity requiring linker to synthesize multiple,
>>>>> inter-related sections.
>>>>>
>>>>>>       So, it seems the general advise was to not rely on linker script
>>>>>> mechanism.
>>>>>>       Above all, making the layout linker-friendly with this approach adversely
>>>>>>       affects the efficacy of the format.
>>>>>>
>>>>>> [1] SFrame wiki:https://sourceware.org/binutils/wiki/sframe
>>>>>>
>>>>>
>>>>> Adopting the .sframe linking view and .sframe_idx execution view
>>>>> doesn't require any special semantics to linker scripts.
>>>>>
>>>>> Linker scripts can be used to control placement using: .sframe_idx : {
>>>>> *(.sframe_idx) }.
>>>>>    From the linker perspective, .sframe input sections have been replaced
>>>>> by the linker-synthesized `.sframe_idx`, meaning the linker never
>>>>> actually see `.sframe` in input section descriptions.
>>>>> This is similar to .got : { *(.got) } where .got is linker-synthesized.
>>>>