GNU Tools Cauldron SFrame talk notes

[Jose E. Marchesi]

> On Mon, Oct 20, 2025 at 12:57 PM Indu Bhagat <indu.bhagat@oracle.com> wrote:
>>
>> On 10/16/25 1:03 AM, Fangrui Song wrote:
>> >>> I agree that performance is crucial for stack tracing. However, we
>> >>> need to distinguish between optimal performance and baseline functionality.
>> >>>
>> >>> For optimal portability, we must support object files from diverse origins—not
>> >>> just those built from a single toolchain.
>> >>
>> >> Sure. Thats not forsaken with the current design.
>> >>
>> >>> In environments where almost
>> >>> everything is built from source with a single toolchain offering strong SFrame
>> >>> support, forcing default-on index building may be acceptable. However, we must
>> >>> also accommodate environments with prebuilt object files using older SFrame
>> >>> versions, or toolchains that don't support old formats.
>> >>>
>> >>
>> >> RE: accommodate environments with prebuilt object files using older
>> >> SFrame versions
>> >> Yes, surely. An SFrame aware linker will be able to do that without much
>> >> difficulty.
>> >>
>> >> RE: or toolchains that don't support old formats.
>> >> If an older version of the format is out in the wild, a well-intentioned
>> >> tool provider will want to include support for older version of the
>> >> format, yes.  Is that a problem ? Its subjective. That argument applies
>> >> to _all_ old and new ELF features and philosophically, any forward
>> >> looking innovation: supporting old(er) specification is a choice that
>> >> needs to be made keeping users in mind.
>> >>
>> >> I would like to underline that reading SFrame is simple enough: Being
>> >> able to read _a_ version of SFrame (V1, V2, or V3 for matter) is not
>> >> going to be wildly difficult if a previous or newer version is already
>> >> intended to be supported. (a.k.a. implementation complexity is low).
>> >>
>> >>> Forward compatibility is another important consideration. Many metadata sections
>> >>> can be used with very old linkers, which is a significant advantage
>> >>> for adoption.
>> >>>> The solution is to make the indexed format an optimization, not a
>> >> requirement:
>> >>>
>> >>> 1. Default behavior: Concatenate .sframe sections (baseline functionality)
>> >>> 2. Optimized behavior: Use --sframe-index to build indices (best performance)
>> >>> 3. Runtime fallback: Unwinders can handle concatenated sections when no index
>> >>>      is available
>> >>>
>> >
>> > The version mix-and-match problem requires a concrete solution—we cannot
>> > defer it. Concatenation with opt-in indexing is the only viable approach
>> > that handles all scenarios.
>> >
>> > Supporting v2, v3, and v4 simultaneously in the linker is a significant
>> > burden. No other metadata format has ever demanded this much from linkers.
>> >
>> > Consider the LLVM-supported metadata sections: __asan_globals, .stack_sizes,
>> > __patchable_function_entries, __llvm_prf_cnts, __sancov_bools, __sancov_pcs,
>> > __llvm_covmap, __llvm_gcov_ctr_section, .llvmcmd, and llvm_offload_entries.
>> > They are all well-behaved ELF citizens that do not require special linker
>> > support. These can all be concatenated without issues. Why should SFrame be
>> > treated differently?
>> >
>> >> As I mentioned previously, IMO having stack tracers to handle
>> >> concatenated SFrame sections to perform stack tracing will not the right
>> >> thing to do. It will be a performance hit; the whole point of SFrame is
>> >> enable fast stack tracing.
>> >
>> > A stack unwinder already needs to support .sframe sections across the main
>> > executable and all shared objects. Why would supporting multiple concatenated
>> > elements within a single .sframe section be problematic?
>> >
>> >>> LLDB already supports this model for .debug_names, which can either be
>> >>> concatenated or an optimized index built by ld.lld --debug-names.
>> >>>
>> >>>
>> >>>> So you see, we are already converging to say that for this format to be
>> >>>> useful, implementing --sframe-index (or merging the sections in some
>> >>>> form) is _necessary_.
>> >>>
>> >>> Thanks. I'm glad we agree that ld --sframe-index is necessary.
>> >>>
>> >>> However, I must emphasize that this should be opt-in, not default behavior.
>> >>> Making index building opt-in provides several benefits:
>> >>>
>> >>
>> >> I am not convinced that --sframe-index as opt-in is the right choice.
>> >
>> > .sframe is far from an established technology. Why is opt-in --sframe-index
>> > not the right choice? If .sframe ever becomes ubiquitous like RELRO, you could
>> > make an argument for making indexing the default—but even then, you could
>> > achieve this through compiler drivers rather than changing linker defaults.
>> >
>> >> Stack traces needing to create index by reading in the concatenated
>> >> sections will be
>> >>    1. a performance hit; not recommended for the use case of fast stack
>> >> tracing.
>> >>    2. duplicated functionality in Linux kernel, glibc and other usecases.
>> >>
>> >>> 1. Linkers can support basic .sframe handling (concatenation) without
>> >>> implementing the full index-building logic
>> >>> 2. Mixed-version scenarios degrade gracefully to concatenation rather
>> >>> than failing
>> >>> 3. Consistent with precedent .debug_names
>> >>> 4. Distros can roll out SFrame support incrementally without requiring
>> >>> all linkers to support index building immediately
>> >>>
>> >>
>> >> While this looks pleasing and sits well with the status quo (linker's
>> >> default behaviour of concatenating unknown sections etc.),  concatenated
>> >> SFrame sections are effectively useless because of the performance hit
>> >> of reading in the input sections and creating the index at the time of
>> >> stack tracing.
>> >>
>> >> Taking a step back though, I think we are mixing up a lot of different
>> >> things here (backward/forward compatibility, mixing artifacts from two
>> >> different toolchains etc).  The fundamental concern is: SFrame is a
>> >> format that asks for merging to be as the default link-editor behavior.
>> >> This does not align well with the existing ELF provisions.
>> >>
>> >
>> > I don't think we're mixing things up. The version mix-and-match problem is
>> > central to SFrame's real-world viability, not a peripheral concern.
>> >
>> > Discarding .sframe sections sidesteps the version mismatch issue rather than
>> > solving it. The critical question remains: What happens when a linker only
>> > supports v3/v4 but encounters object files with v2 .sframe sections? Given
>> > the drop of v1 and the planned major changes for v3, I don't think linker
>> > maintainers should be forced to play this "live at head, implement everything
>> > we throw at you" game.
>> >
>>
>> We are aiming to use SFrame V3 for any distro wide builds.  We are in
>> touch with the some of the distros to co-ordinate this better. glibc
>> backtrace () has experimental support for V2, but the Linux kernel will
>> support V3 and above, so there is little incentive to create SFrame V2
>> binaries distro wide.
>
> I appreciate the coordination with distros for V3 rollout. However, this
> plan addresses only clean distro-wide rebuilds while leaving realistic
> mixing scenarios unresolved:
>
> - Third-party vendor libraries built with older toolchains
> - Users linking against prebuilt libraries from different sources
> - Users who don't need SFrame but must handle prebuilt libraries with
> older SFrame versions
> - Users updating their linker to a newer version that drops legacy
> SFrame support
>
> The distro plan doesn't eliminate these cases—it defers them to linker
> implementers. Having SFrame in prebuilt libraries doesn't mean users need it,
> yet they must handle any issues arising from merging different versions.
>
>> > Consider this realistic scenario:
>> >
>> > - A distribution upgrades its toolchain to support v4 and drop legacy v2.
>> > - Users still have prebuilt libraries or object files containing v2 sections
>> > - The v3/v4-aware linker must decide: discard v2? error? concatenate?
>> >
>>
>> If the new version of the format is simply an alternate layout, linker
>> upgrading to the newer version should be easily doable.
>>
>> If the new version of the format is because it is to support a new
>> architecture feature, and if this is required to be an arch-wide
>> enablement, distros will need a full rebuild anyway.
>>
>> For SFrame V3 and above, much of this depends on what changes are
>> introduced from one version to the next, but the linker will need to
>> DTRT: Either generate the latest version Vn as output (by upgrading on
>> the fly) or generate a previous version if so desired by the user..
>
> This "linker will DTRT" assertion glosses over significant implementation
> complexity. Each version needs not just a reader but version-specific *merging*
> logic in every linker—fundamentally different from simply reading a format.
>
>
>> > Again, supporting all versions a significant burden. No other metadata format
>> > has ever demanded this much from linkers.
>> >
>>
>> The term "significant burden" is contestable, as previously argued.
>> SFrame is a simple format to read, once you have a library to read/write
>> one version of the format, to support iterations of it should be easily
>> doable (both minimal increase in implementation complexity and runtime
>> cost to linkers).
>
> I must disagree. Reading is simple; merging multiple versions with
> potentially different layouts is not. Consider:
>
> - V2 uses layout A, V3 uses layout B, V4 uses layout C
> - Linker receives objects with all three versions
> - Must produce coherent output with proper indexing
> - Requires maintaining version-specific merge logic for each version
>
> Can you name another loadable metadata format that requires this level of
> version-specific processing in linkers? The LLVM metadata sections I listed
> (__asan_globals, .stack_sizes, etc.) all concatenate trivially—they don't
> need version-aware merging.
>
>
>> RE:"No other metadata format..."
>> I will refrain to pick on one format or another to prove/disprove the
>> need for SFrame.  A few formats out there use index or perform
>> de-duplication of some sort already at link-time with varying degree of
>> support from the various linkers.  They are each driven by user-needs
>> and serve their own specific usecase.
>>
>> That said, to some of us the need for SFrame is clear: fast, precise
>> stack tracing. Such a usecase needs the data to be indexed, and hence
>> some of the design choices.
>
> This doesn't address my point. I'm not questioning SFrame's *need*, but
> rather its *design choice* to require version-specific merging as default
> linker behavior. Which metadata format requires this? These differences
> matter for linker maintenance burden.
>
>> >> I think revisiting ELF and its default behavior for unknown sections is
>> >> sensible in this scenario. An unaware linker should then discard these
>> >> sections until it gains the capability to merge SFrame sections, and not
>> >> pass the buck to stack tracers to do the right thing.
>> >
>> > I believe "revisiting ELF default behavior" is not realistic. We need a
>> > solution that works within existing ELF conventions, not one that requires
>> > fundamental changes to them. I don't think the .sframe use case justifies
>> > introducing a new section flag.
>> >
>> > However, if you insist on proposing SHF_OS_NONCONFORMING_DISCARD, you can
>> > bring it to https://groups.google.com/g/generic-abi
>> >
>> > ---
>> >
>> >> re: --sframe-index, there are two issues that make it an unacceptable
>> >> solution. Both boil down to the fact that debug-info is not a good
>> >> model for this problem.
>> >>
>> >> The first is that sframes (and the index itself, in whatever form),
>> >> needs to be loadable, and indeed, *loaded* at the point the stack
>> >> trace is taken, possibly by the kernel itself. Adding a section like
>> >> .gdb_index post-link is simply a matter of objcopy --add-section=....
>> >> It doesn't need to be loaded, or have any special handling by anything
>> >> other than the debugger. Adding a loadable segment post-link is
>> >> substantially more complicated.
>> >>
>> >> The second is that even though gdb_index is easy to add post-link,
>> >> users thought it was sufficiently annoying that we added linker
>> >> support in three linkers: gnu-ld, gnu-gold, and lld itself. These
>> >> additions were completely uncontroversial. That's a pretty big
>> >> testament to the value of having the linker do it. It's easy to add a
>> >> linker flag, but much harder to add a post-link build step. Especially
>> >> with things like build-ids, read-only file systems, and whatever else.
>> >> Adding sframes would mean every package in every distro would need to
>> >> add a post-link build step, and a distro would have a very hard time
>> >> enabling it by default without hundreds of packages updating
>> >> themselves.
>> >>
>> >> I have only anecdata here, but I suspect somewhere around 90% of
>> >> gdb_index users do it at the link step, rather than post-link.
>> >
>> > I agree that SHF_ALLOC makes post-link tooling more difficult than for
>> > .gdb_index. However, this argues for *having* linker support, not for making
>> > it default behavior. The two issues can be addressed separately:
>> >
>> > .gdb_index had years of post-link usage proving its value before
>> > linkers added support.
>> > The three linkers adding support was "uncontroversial" precisely because the
>> > feature was mature and well-understood.
>> >
>> > .sframe is in a different position: v1 dropped, v3 upcoming with major changes,
>> > limited deployment, and unresolved version compatibility issues. Making an
>> > immature format with known version churn the default behavior is premature.
>> >
>> > Regarding distro enablement: if --sframe-index is opt-in, distros can enable
>> > it through compiler drivers (like -gz or -gsplit-dwarf) once the format
>> > stabilizes.
>> >
>> > For experimentation with this format, concatenation provides a viable path:
>> >
>> > - Design .sframe with minimal per-element overhead
>> > - Concatenated sections remain reasonably sized
>> > - A post-link tool can optimize and overwrite the section in-place
>>
>> As I mentioned before, this is a no-go.  We need support for SFrame
>> sections in the toolchain proper and not outside of it.
>>
>> If there is a solution to merge SFrame sections, to create stack trace
>> information along with an index, using existing/new ELF provisions, I
>> would like to hear about it.
>
> I have proposed a solution that works within existing ELF conventions:
>
> 1. Linkers concatenate .sframe sections (standard ELF behavior)
> 2. Add --sframe-index flag for opt-in merging/indexing
> 3. Unwinders handle concatenated sections when no index exists
> 4. Distros enable via compiler drivers once the format stabilizes
> This *is* toolchain-proper support. The distinction is opt-in
> vs. default.

A solution that generates invalid data is a no-solution.
Deferring the problem to your users (unwinders) is a no-solution.

Concatenating SFrame sections do not lead to valid data, and clearly you
don't have a solution for that, as awesome as that would have been.

People is starting to store non-concatenable stuff in ELF sections, like
JSON for example, as annoying as that may be.  This is happening we like
it or not, and picking on SFrame will not help a bit.  If ELF
conventions are too stringent for what people want to store in ELF
today, then ELF conventions need to be updated. It seems to me that as
tools maintainers it is our responsibility to make sure formats and
tools serve people's needs, not the other way around.

If I understand you properly, after the goalpost shifting, from

  "your format shall be concatenable to be ELF conformant"

to, after it was pointed out that SFrame is not the only ELF stored data
that needs some degree of linker awareness,

  "your format is immature, so linker awareness is ok but it should not
   be the default"

your main concern now is opt-in vs. default until some degree of
"maturity" that satisfies you is achieved.

Well, Indu already suggested a solution for that, which is to add
SHF_OS_NONCONFORMING_DISCARD, a logical complement to the existing
SHF_OS_NONCONFORMING.  Having this flag would allow ELF tools to easily
deal with unknown sections with non-concatenable data in them without
aborting the link or generating nonsense: if that flag is set and you
don't know about SHT_GNU_SFRAME, just discard the section.

In the email you link below you said her proposal is "unrealistic",
qualifying the flag as a "fundamental change" to existing ELF
conventions, so I ask you:

- What is exactly unrealistic about adding a new ELF section flag,
  especially now that the spec is actively maintained and updated at
  https://github.com/xinuos/gabi?

- Given that there are at much a dozen ELF linkers in use in the entire
  world (and that is probably too generous an estimation) what exactly
  is unrealistic about expecting them to adopt a single new, generic
  flag that hardly requires adding any logic beyond checking a flag and
  discarding an input section?

- How is adding this particular section flag a fundamental change,
  considering SHF_OS_NONCONFORMING already exists?

> Your approach:
>
> - Violates ELF conventions

SHF_OS_NONCONFORMING_DISCARD fixes this?

> - Forces all linkers to implement version-specific merging

SHF_OS_NONCONFORMING_DISCARD fixes this?

> In lld/ELF, linker-created sections are called synthetic sections.
> No synthetic section requires version-specific merging.
> Even the quite complex .gdb_index section doesn't require this, proving its
> stability after years. Why not let .sframe follow a similar maturation path
> through opt-in support first?

Sure: let's just have the new flag.  Then you will not have to concern
yourself with SFrame's maturity level, and the rest of the world won't
have to concern itself with LLD's maturity level.

> ---
>
> To move forward constructively, please address the problems I raised in:
> https://sourceware.org/pipermail/binutils/2025-October/144904.html
>
>
>> > - For cases where optimization significantly shrinks the section, place
>> >    .sframe at the end of the file (similar to BOLT moving .rodata)
>> >
>> > This approach lets the format mature and prove its value before committing
>> > to complex linker behavior.
>> >
>> > ---
>> >
>> > On a separate note regarding format direction: It appears that .sframe
>> > won't fully replace .eh_frame.
>> > In some cases, such as this sqlite3 amalgamation, .sframe is actually
>> > larger than .eh_frame.
>> > This raises the question of whether the marginal performance advantage over
>> > frame pointer unwinding justifies the additional section overhead.
>> >
>> >    [16] .eh_frame         PROGBITS        0000000000000000 0494c8
>> > 0090c8 00   A  0   0  8
>> >    [17] .rela.eh_frame    RELA            0000000000000000 079740
>> > 004620 18   I 20  16  8
>> >    [18] .sframe           LOOS+0xffffff4  0000000000000000 052590
>> > 00a909 00   A  0   0  8
>> >    [19] .rela.sframe      RELA            0000000000000000 07dd60
>> > 004620 18   I 20  18  8
>>