GNU Tools Cauldron SFrame talk notes

[Indu Bhagat]

On 10/16/25 1:03 AM, Fangrui Song wrote:
>>> I agree that performance is crucial for stack tracing. However, we
>>> need to distinguish between optimal performance and baseline functionality.
>>>
>>> For optimal portability, we must support object files from diverse origins—not
>>> just those built from a single toolchain.
>>
>> Sure. Thats not forsaken with the current design.
>>
>>> In environments where almost
>>> everything is built from source with a single toolchain offering strong SFrame
>>> support, forcing default-on index building may be acceptable. However, we must
>>> also accommodate environments with prebuilt object files using older SFrame
>>> versions, or toolchains that don't support old formats.
>>>
>>
>> RE: accommodate environments with prebuilt object files using older
>> SFrame versions
>> Yes, surely. An SFrame aware linker will be able to do that without much
>> difficulty.
>>
>> RE: or toolchains that don't support old formats.
>> If an older version of the format is out in the wild, a well-intentioned
>> tool provider will want to include support for older version of the
>> format, yes.  Is that a problem ? Its subjective. That argument applies
>> to _all_ old and new ELF features and philosophically, any forward
>> looking innovation: supporting old(er) specification is a choice that
>> needs to be made keeping users in mind.
>>
>> I would like to underline that reading SFrame is simple enough: Being
>> able to read _a_ version of SFrame (V1, V2, or V3 for matter) is not
>> going to be wildly difficult if a previous or newer version is already
>> intended to be supported. (a.k.a. implementation complexity is low).
>>
>>> Forward compatibility is another important consideration. Many metadata sections
>>> can be used with very old linkers, which is a significant advantage
>>> for adoption.
>>>> The solution is to make the indexed format an optimization, not a
>> requirement:
>>>
>>> 1. Default behavior: Concatenate .sframe sections (baseline functionality)
>>> 2. Optimized behavior: Use --sframe-index to build indices (best performance)
>>> 3. Runtime fallback: Unwinders can handle concatenated sections when no index
>>>      is available
>>>
> 
> The version mix-and-match problem requires a concrete solution—we cannot
> defer it. Concatenation with opt-in indexing is the only viable approach
> that handles all scenarios.
> 
> Supporting v2, v3, and v4 simultaneously in the linker is a significant
> burden. No other metadata format has ever demanded this much from linkers.
> 
> Consider the LLVM-supported metadata sections: __asan_globals, .stack_sizes,
> __patchable_function_entries, __llvm_prf_cnts, __sancov_bools, __sancov_pcs,
> __llvm_covmap, __llvm_gcov_ctr_section, .llvmcmd, and llvm_offload_entries.
> They are all well-behaved ELF citizens that do not require special linker
> support. These can all be concatenated without issues. Why should SFrame be
> treated differently?
> 
>> As I mentioned previously, IMO having stack tracers to handle
>> concatenated SFrame sections to perform stack tracing will not the right
>> thing to do. It will be a performance hit; the whole point of SFrame is
>> enable fast stack tracing.
> 
> A stack unwinder already needs to support .sframe sections across the main
> executable and all shared objects. Why would supporting multiple concatenated
> elements within a single .sframe section be problematic?
> 
>>> LLDB already supports this model for .debug_names, which can either be
>>> concatenated or an optimized index built by ld.lld --debug-names.
>>>
>>>
>>>> So you see, we are already converging to say that for this format to be
>>>> useful, implementing --sframe-index (or merging the sections in some
>>>> form) is _necessary_.
>>>
>>> Thanks. I'm glad we agree that ld --sframe-index is necessary.
>>>
>>> However, I must emphasize that this should be opt-in, not default behavior.
>>> Making index building opt-in provides several benefits:
>>>
>>
>> I am not convinced that --sframe-index as opt-in is the right choice.
> 
> .sframe is far from an established technology. Why is opt-in --sframe-index
> not the right choice? If .sframe ever becomes ubiquitous like RELRO, you could
> make an argument for making indexing the default—but even then, you could
> achieve this through compiler drivers rather than changing linker defaults.
> 
>> Stack traces needing to create index by reading in the concatenated
>> sections will be
>>    1. a performance hit; not recommended for the use case of fast stack
>> tracing.
>>    2. duplicated functionality in Linux kernel, glibc and other usecases.
>>
>>> 1. Linkers can support basic .sframe handling (concatenation) without
>>> implementing the full index-building logic
>>> 2. Mixed-version scenarios degrade gracefully to concatenation rather
>>> than failing
>>> 3. Consistent with precedent .debug_names
>>> 4. Distros can roll out SFrame support incrementally without requiring
>>> all linkers to support index building immediately
>>>
>>
>> While this looks pleasing and sits well with the status quo (linker's
>> default behaviour of concatenating unknown sections etc.),  concatenated
>> SFrame sections are effectively useless because of the performance hit
>> of reading in the input sections and creating the index at the time of
>> stack tracing.
>>
>> Taking a step back though, I think we are mixing up a lot of different
>> things here (backward/forward compatibility, mixing artifacts from two
>> different toolchains etc).  The fundamental concern is: SFrame is a
>> format that asks for merging to be as the default link-editor behavior.
>> This does not align well with the existing ELF provisions.
>>
> 
> I don't think we're mixing things up. The version mix-and-match problem is
> central to SFrame's real-world viability, not a peripheral concern.
> 
> Discarding .sframe sections sidesteps the version mismatch issue rather than
> solving it. The critical question remains: What happens when a linker only
> supports v3/v4 but encounters object files with v2 .sframe sections? Given
> the drop of v1 and the planned major changes for v3, I don't think linker
> maintainers should be forced to play this "live at head, implement everything
> we throw at you" game.
> 

We are aiming to use SFrame V3 for any distro wide builds.  We are in 
touch with the some of the distros to co-ordinate this better. glibc 
backtrace () has experimental support for V2, but the Linux kernel will 
support V3 and above, so there is little incentive to create SFrame V2 
binaries distro wide.

> Consider this realistic scenario:
> 
> - A distribution upgrades its toolchain to support v4 and drop legacy v2.
> - Users still have prebuilt libraries or object files containing v2 sections
> - The v3/v4-aware linker must decide: discard v2? error? concatenate?
> 

If the new version of the format is simply an alternate layout, linker 
upgrading to the newer version should be easily doable.

If the new version of the format is because it is to support a new 
architecture feature, and if this is required to be an arch-wide 
enablement, distros will need a full rebuild anyway.

For SFrame V3 and above, much of this depends on what changes are 
introduced from one version to the next, but the linker will need to 
DTRT: Either generate the latest version Vn as output (by upgrading on 
the fly) or generate a previous version if so desired by the user..

> Again, supporting all versions a significant burden. No other metadata format
> has ever demanded this much from linkers.
> 

The term "significant burden" is contestable, as previously argued. 
SFrame is a simple format to read, once you have a library to read/write 
one version of the format, to support iterations of it should be easily 
doable (both minimal increase in implementation complexity and runtime 
cost to linkers).

RE:"No other metadata format..."
I will refrain to pick on one format or another to prove/disprove the 
need for SFrame.  A few formats out there use index or perform 
de-duplication of some sort already at link-time with varying degree of 
support from the various linkers.  They are each driven by user-needs 
and serve their own specific usecase.

That said, to some of us the need for SFrame is clear: fast, precise 
stack tracing. Such a usecase needs the data to be indexed, and hence 
some of the design choices.

>> I think revisiting ELF and its default behavior for unknown sections is
>> sensible in this scenario. An unaware linker should then discard these
>> sections until it gains the capability to merge SFrame sections, and not
>> pass the buck to stack tracers to do the right thing.
> 
> I believe "revisiting ELF default behavior" is not realistic. We need a
> solution that works within existing ELF conventions, not one that requires
> fundamental changes to them. I don't think the .sframe use case justifies
> introducing a new section flag.
> 
> However, if you insist on proposing SHF_OS_NONCONFORMING_DISCARD, you can
> bring it to https://groups.google.com/g/generic-abi
> 
> ---
> 
>> re: --sframe-index, there are two issues that make it an unacceptable
>> solution. Both boil down to the fact that debug-info is not a good
>> model for this problem.
>>
>> The first is that sframes (and the index itself, in whatever form),
>> needs to be loadable, and indeed, *loaded* at the point the stack
>> trace is taken, possibly by the kernel itself. Adding a section like
>> .gdb_index post-link is simply a matter of objcopy --add-section=....
>> It doesn't need to be loaded, or have any special handling by anything
>> other than the debugger. Adding a loadable segment post-link is
>> substantially more complicated.
>>
>> The second is that even though gdb_index is easy to add post-link,
>> users thought it was sufficiently annoying that we added linker
>> support in three linkers: gnu-ld, gnu-gold, and lld itself. These
>> additions were completely uncontroversial. That's a pretty big
>> testament to the value of having the linker do it. It's easy to add a
>> linker flag, but much harder to add a post-link build step. Especially
>> with things like build-ids, read-only file systems, and whatever else.
>> Adding sframes would mean every package in every distro would need to
>> add a post-link build step, and a distro would have a very hard time
>> enabling it by default without hundreds of packages updating
>> themselves.
>>
>> I have only anecdata here, but I suspect somewhere around 90% of
>> gdb_index users do it at the link step, rather than post-link.
> 
> I agree that SHF_ALLOC makes post-link tooling more difficult than for
> .gdb_index. However, this argues for *having* linker support, not for making
> it default behavior. The two issues can be addressed separately:
> 
> .gdb_index had years of post-link usage proving its value before
> linkers added support.
> The three linkers adding support was "uncontroversial" precisely because the
> feature was mature and well-understood.
> 
> .sframe is in a different position: v1 dropped, v3 upcoming with major changes,
> limited deployment, and unresolved version compatibility issues. Making an
> immature format with known version churn the default behavior is premature.
> 
> Regarding distro enablement: if --sframe-index is opt-in, distros can enable
> it through compiler drivers (like -gz or -gsplit-dwarf) once the format
> stabilizes.
> 
> For experimentation with this format, concatenation provides a viable path:
> 
> - Design .sframe with minimal per-element overhead
> - Concatenated sections remain reasonably sized
> - A post-link tool can optimize and overwrite the section in-place

As I mentioned before, this is a no-go.  We need support for SFrame 
sections in the toolchain proper and not outside of it.

If there is a solution to merge SFrame sections, to create stack trace 
information along with an index, using existing/new ELF provisions, I 
would like to hear about it.

> - For cases where optimization significantly shrinks the section, place
>    .sframe at the end of the file (similar to BOLT moving .rodata)
> 
> This approach lets the format mature and prove its value before committing
> to complex linker behavior.
> 
> ---
> 
> On a separate note regarding format direction: It appears that .sframe
> won't fully replace .eh_frame.
> In some cases, such as this sqlite3 amalgamation, .sframe is actually
> larger than .eh_frame.
> This raises the question of whether the marginal performance advantage over
> frame pointer unwinding justifies the additional section overhead.
> 
>    [16] .eh_frame         PROGBITS        0000000000000000 0494c8
> 0090c8 00   A  0   0  8
>    [17] .rela.eh_frame    RELA            0000000000000000 079740
> 004620 18   I 20  16  8
>    [18] .sframe           LOOS+0xffffff4  0000000000000000 052590
> 00a909 00   A  0   0  8
>    [19] .rela.sframe      RELA            0000000000000000 07dd60
> 004620 18   I 20  18  8