GNU Tools Cauldron SFrame talk notes

[Fangrui Song]

> > I agree that performance is crucial for stack tracing. However, we
> > need to distinguish between optimal performance and baseline functionality.
> >
> > For optimal portability, we must support object files from diverse origins—not
> > just those built from a single toolchain.
>
> Sure. Thats not forsaken with the current design.
>
> > In environments where almost
> > everything is built from source with a single toolchain offering strong SFrame
> > support, forcing default-on index building may be acceptable. However, we must
> > also accommodate environments with prebuilt object files using older SFrame
> > versions, or toolchains that don't support old formats.
> >
>
> RE: accommodate environments with prebuilt object files using older
> SFrame versions
> Yes, surely. An SFrame aware linker will be able to do that without much
> difficulty.
>
> RE: or toolchains that don't support old formats.
> If an older version of the format is out in the wild, a well-intentioned
> tool provider will want to include support for older version of the
> format, yes.  Is that a problem ? Its subjective. That argument applies
> to _all_ old and new ELF features and philosophically, any forward
> looking innovation: supporting old(er) specification is a choice that
> needs to be made keeping users in mind.
>
> I would like to underline that reading SFrame is simple enough: Being
> able to read _a_ version of SFrame (V1, V2, or V3 for matter) is not
> going to be wildly difficult if a previous or newer version is already
> intended to be supported. (a.k.a. implementation complexity is low).
>
> > Forward compatibility is another important consideration. Many metadata sections
> > can be used with very old linkers, which is a significant advantage
> > for adoption.
> > > The solution is to make the indexed format an optimization, not a
> requirement:
> >
> > 1. Default behavior: Concatenate .sframe sections (baseline functionality)
> > 2. Optimized behavior: Use --sframe-index to build indices (best performance)
> > 3. Runtime fallback: Unwinders can handle concatenated sections when no index
> >     is available
> >

The version mix-and-match problem requires a concrete solution—we cannot
defer it. Concatenation with opt-in indexing is the only viable approach
that handles all scenarios.

Supporting v2, v3, and v4 simultaneously in the linker is a significant
burden. No other metadata format has ever demanded this much from linkers.

Consider the LLVM-supported metadata sections: __asan_globals, .stack_sizes,
__patchable_function_entries, __llvm_prf_cnts, __sancov_bools, __sancov_pcs,
__llvm_covmap, __llvm_gcov_ctr_section, .llvmcmd, and llvm_offload_entries.
They are all well-behaved ELF citizens that do not require special linker
support. These can all be concatenated without issues. Why should SFrame be
treated differently?

> As I mentioned previously, IMO having stack tracers to handle
> concatenated SFrame sections to perform stack tracing will not the right
> thing to do. It will be a performance hit; the whole point of SFrame is
> enable fast stack tracing.

A stack unwinder already needs to support .sframe sections across the main
executable and all shared objects. Why would supporting multiple concatenated
elements within a single .sframe section be problematic?

> > LLDB already supports this model for .debug_names, which can either be
> > concatenated or an optimized index built by ld.lld --debug-names.
> >
> >
> >> So you see, we are already converging to say that for this format to be
> >> useful, implementing --sframe-index (or merging the sections in some
> >> form) is _necessary_.
> >
> > Thanks. I'm glad we agree that ld --sframe-index is necessary.
> >
> > However, I must emphasize that this should be opt-in, not default behavior.
> > Making index building opt-in provides several benefits:
> >
>
> I am not convinced that --sframe-index as opt-in is the right choice.

.sframe is far from an established technology. Why is opt-in --sframe-index
not the right choice? If .sframe ever becomes ubiquitous like RELRO, you could
make an argument for making indexing the default—but even then, you could
achieve this through compiler drivers rather than changing linker defaults.

> Stack traces needing to create index by reading in the concatenated
> sections will be
>   1. a performance hit; not recommended for the use case of fast stack
> tracing.
>   2. duplicated functionality in Linux kernel, glibc and other usecases.
>
> > 1. Linkers can support basic .sframe handling (concatenation) without
> > implementing the full index-building logic
> > 2. Mixed-version scenarios degrade gracefully to concatenation rather
> > than failing
> > 3. Consistent with precedent .debug_names
> > 4. Distros can roll out SFrame support incrementally without requiring
> > all linkers to support index building immediately
> >
>
> While this looks pleasing and sits well with the status quo (linker's
> default behaviour of concatenating unknown sections etc.),  concatenated
> SFrame sections are effectively useless because of the performance hit
> of reading in the input sections and creating the index at the time of
> stack tracing.
>
> Taking a step back though, I think we are mixing up a lot of different
> things here (backward/forward compatibility, mixing artifacts from two
> different toolchains etc).  The fundamental concern is: SFrame is a
> format that asks for merging to be as the default link-editor behavior.
> This does not align well with the existing ELF provisions.
>

I don't think we're mixing things up. The version mix-and-match problem is
central to SFrame's real-world viability, not a peripheral concern.

Discarding .sframe sections sidesteps the version mismatch issue rather than
solving it. The critical question remains: What happens when a linker only
supports v3/v4 but encounters object files with v2 .sframe sections? Given
the drop of v1 and the planned major changes for v3, I don't think linker
maintainers should be forced to play this "live at head, implement everything
we throw at you" game.

Consider this realistic scenario:

- A distribution upgrades its toolchain to support v4 and drop legacy v2.
- Users still have prebuilt libraries or object files containing v2 sections
- The v3/v4-aware linker must decide: discard v2? error? concatenate?

Again, supporting all versions a significant burden. No other metadata format
has ever demanded this much from linkers.

> I think revisiting ELF and its default behavior for unknown sections is
> sensible in this scenario. An unaware linker should then discard these
> sections until it gains the capability to merge SFrame sections, and not
> pass the buck to stack tracers to do the right thing.

I believe "revisiting ELF default behavior" is not realistic. We need a
solution that works within existing ELF conventions, not one that requires
fundamental changes to them. I don't think the .sframe use case justifies
introducing a new section flag.

However, if you insist on proposing SHF_OS_NONCONFORMING_DISCARD, you can
bring it to https://groups.google.com/g/generic-abi

---

> re: --sframe-index, there are two issues that make it an unacceptable
> solution. Both boil down to the fact that debug-info is not a good
> model for this problem.
>
> The first is that sframes (and the index itself, in whatever form),
> needs to be loadable, and indeed, *loaded* at the point the stack
> trace is taken, possibly by the kernel itself. Adding a section like
> .gdb_index post-link is simply a matter of objcopy --add-section=....
> It doesn't need to be loaded, or have any special handling by anything
> other than the debugger. Adding a loadable segment post-link is
> substantially more complicated.
>
> The second is that even though gdb_index is easy to add post-link,
> users thought it was sufficiently annoying that we added linker
> support in three linkers: gnu-ld, gnu-gold, and lld itself. These
> additions were completely uncontroversial. That's a pretty big
> testament to the value of having the linker do it. It's easy to add a
> linker flag, but much harder to add a post-link build step. Especially
> with things like build-ids, read-only file systems, and whatever else.
> Adding sframes would mean every package in every distro would need to
> add a post-link build step, and a distro would have a very hard time
> enabling it by default without hundreds of packages updating
> themselves.
>
> I have only anecdata here, but I suspect somewhere around 90% of
> gdb_index users do it at the link step, rather than post-link.

I agree that SHF_ALLOC makes post-link tooling more difficult than for
.gdb_index. However, this argues for *having* linker support, not for making
it default behavior. The two issues can be addressed separately:

.gdb_index had years of post-link usage proving its value before
linkers added support.
The three linkers adding support was "uncontroversial" precisely because the
feature was mature and well-understood.

.sframe is in a different position: v1 dropped, v3 upcoming with major changes,
limited deployment, and unresolved version compatibility issues. Making an
immature format with known version churn the default behavior is premature.

Regarding distro enablement: if --sframe-index is opt-in, distros can enable
it through compiler drivers (like -gz or -gsplit-dwarf) once the format
stabilizes.

For experimentation with this format, concatenation provides a viable path:

- Design .sframe with minimal per-element overhead
- Concatenated sections remain reasonably sized
- A post-link tool can optimize and overwrite the section in-place
- For cases where optimization significantly shrinks the section, place
  .sframe at the end of the file (similar to BOLT moving .rodata)

This approach lets the format mature and prove its value before committing
to complex linker behavior.

---

On a separate note regarding format direction: It appears that .sframe
won't fully replace .eh_frame.
In some cases, such as this sqlite3 amalgamation, .sframe is actually
larger than .eh_frame.
This raises the question of whether the marginal performance advantage over
frame pointer unwinding justifies the additional section overhead.

  [16] .eh_frame         PROGBITS        0000000000000000 0494c8
0090c8 00   A  0   0  8
  [17] .rela.eh_frame    RELA            0000000000000000 079740
004620 18   I 20  16  8
  [18] .sframe           LOOS+0xffffff4  0000000000000000 052590
00a909 00   A  0   0  8
  [19] .rela.sframe      RELA            0000000000000000 07dd60
004620 18   I 20  18  8