Re: PT_NOTE alignment, NT_GNU_PROPERTY_TYPE_0, glibc and gold

[Florian Weimer <fweimer at redhat dot com>]

At the GNU Tools Cauldron in Manchester, we had an ad-hoc side meeting
to work out a consensus among many of those involved so far.

I'm including the minutes below.  It is my hope that this can serve as a
guide for obtaining community consensus.

Cary, after reviewing the minutes, would you be willing to join the
emerging consensus, and re-review H.J.'s patch to align gold's behavior
with BFD ld?

Thanks,
Florian

# Background for the GNU property notes design

The notes are intended for subarchitecture markup, including the recent
Intel CET (CET).  (CET is an upcoming Intel CPU feature which needs
forwards and backwards compatibility for binaries, across and with the
existing ecosystem.  CPU features need to be enabled per process if all
loaded objects support it.)

Putting the markup in a new, separate program header segment (PT_*)
has more risk across the ecosystem than PT_NOTE.  We know that PT_NOTE
works because different new notes have been added to the PT_NOTE
segment over time.

gABI does not completely and unambiguously describe the layout of note
segments and sections and the format of the note header, although the
historic interpretation has been that the note header has three 32-bit
words, independently of the ELF class, particularly on GNU/Linux.
(But HP-UX uses 8-byte words in the note header in the 64-bit case.)

gABI does say that ELF notes should be 8-byte aligned for the 64-bit
ELF class.  But there are commonly used existing 4-byte-aligned notes:
the ABI tag and the build ID.  The Linux kernel, when generating
coredumps, may put 0 or 1 in the p_align field of the PT_NOTE segment;
this is a clear bug, the notes are actually 4-byte-aligned.

There could be relocations in the note; nothing currently forbids
that. The stack size in GNU_PROPERTY_STACK_SIZE is 64-bit for the 64-bit
ELF class (> 4 GiB may not make sense, but is theoretically possible).
x86-64 does not need alignment, but strict alignment targets need it,
including perhaps for relocations.

Therefore, a conscious decision was made to follow gABI and use 8-byte
alignment for the new note types (GNU property notes).

# Current state

8-byte aligned notes have been in binutils (BFD ld) for two years.
GCC 8 includes them (in <cet.h>).  glibc 2.28 ignores property notes
which are not 8-byte-aligned.

Distributions have built binaries with 8-byte-aligned GNU property notes.

Current binutils gold merges all allocated note section into a 4-byte
aligned note segment.

Relocations in notes work, and annobin uses 8-byte relocations in
4-byte-aligned notes in relocatable (ET_REL) objects on multiple
64-bit targets.

Due to the kernel bug which generates PT_NOTE headers with p_align
having values 0 or 1, some consumers ignore the p_align field and
always assume 4.

# Some available options

(1) Only 4-byte-aligned notes are valid.  Existing binaries with 8-byte
alignment become invalid.

(2) The current 8+4 alignment mix (8-byte alignment for GNU Property
notes on ELFCLASS64, 4-byte alignment for older note types such as ABI
tag and build ID) as implemented by BFD ld, GCC, and glibc.

(3) Change GCC and linkers to generate 4-byte-aligned notes
(producers), but keep backwards compatibility with existing
8-byte-aligned notes in binaries (in consumers).

(4) A new PT_* segment for property notes.

# Consensus position of those present

(4) Does not have working code today, so it was not considered
further. (It is also considered larger risk, as discussed above.)

A simplification over the status quo is not achievable with (3)
because consumers still need to be updated.  This means the choice is
between (1) and (2).

(1) makes existing binaries invalid, and there was general agreement
that this is a bad idea.  It also fails to support notes with relocation
on ELFCLASS64 strict-alignment targets.

This leaves us with (2).  There was agreement that link editor
behavior for this approach is currently underspecified.  Some
ABI-level document should indicate what linkers should do (for
example, produce separate PT_NOTE segments for different alignments,
report link failures for note sections of different alignment and the
same name in relocatable links).

In order to obtain consensus among those present, we agreed that option
(2) (that is, 8+4 alignment mix as implemented in BFD ld, GCC, and glibc
today) should only be adopted with 8-byte-alignment for the GNU property
notes, and not automatically for all future notes.  No precedent for
future notes should be established by this decision.

# Further actions

If the community adopts the consensus of those present at the meeting,
binutils gold should change to adapt to BFD ld, but only for
.note.gnu.property.

Florian agreed to try to document the expected link editor behavior
regarding notes and note merging.