[RFC] Linker script pure function plugins - TRANSFORM keyword

[Xiangfei Ding]

Hello GNU Binutils/LD Community,

We propose extending GNU ld and the shared plugin interface in
include/plugin-api.h to support link-time data transformations in
linker scripts via a new output section directive, `TRANSFORM`.

We will coordinate this design with LLVM lld developers so that both
linkers share an identical C ABI and matching linker script syntax.

==============================================================================
1. Problem Statement and Use Cases
==============================================================================

Linker scripts today cannot transform section data. When a build
system needs to manipulate sections, such as compressing binary payloads
or calculating cryptographic integrity digests, it must resort to
out-of-band scripts, intermediate assembly wrappers and post-link
binary patching.

A `TRANSFORM` directive allows linker scripts to invoke pure, side-effect free
and deterministic link-time functions provided by plugins. This approach
addresses several key use cases across systems software.

A. Firmware and Kernel Payload Compression Before Layout
--------------------------------------------------------
The Linux kernel embeds its initial ramdisk by staging raw data through
usr/initramfs_data.S and running compression tools between linker
invocations before the final vmlinux.lds link. Similarly, U-Boot packages
device tree blobs and boot stages into Flat Image Trees by invoking
mkimage and splicing binaries with objcopy. Coreboot relies on `cbfstool`
to compress and rearrange firmware stages after compilation. Because
`cbfstool` runs outside the linker, it strips DWARF debug information and
symbol tables, forcing developers to reconstruct debug metadata manually
ex post facto.

With `TRANSFORM`, the linker script can compress input sections directly
during the link. The linker automatically allocates the destination
section, propagates its size and alignment, and keeps all original DWARF
and symbol metadata intact without external packaging tools.

B. Cryptographic Module Integrity and Post-Relocation Checksums
---------------------------------------------------------------
Cryptographic libraries like BoringSSL and AWS-LC must verify their
integrity at startup. Under FIPS-140 rules, the module calculates an
HMAC-SHA256 digest over its code and data sections and compares this
digest against a stored value during startup selftest.

This digest must cover the final, relocated machine code. When calls to
functions like memcpy resolve to PLT entries or trigger target-specific
linker relaxations, external hashing tools read unresolved bytes and
produce incorrect digests, causing FIPS `selftest` to fail at startup.
Current builds attempt to avoid this by rewriting assembly files before
compilation to normalize relocations. This workaround breaks large code
models, restricts compiler optimizations, and often forces read-only
data into executable memory segments. Embedded firmware images face the
same constraint when injecting bootROM verification checksums.

A post-relocation `TRANSFORM` function resolves this problem by inspecting
the final relocated bytes of target sections and writing the digest
directly into a pre-allocated output section before the linker writes
the output file. This guarantees exact verification without assembly
hacks or post-link tools.

C. Link-Time Table Consolidation and Perfect Hashing
----------------------------------------------------
Tools like gperf generate minimal perfect hash tables for static lookup
data. When separate object files contribute keys independently, each
file builds an isolated table. This duplication increases binary size
and undermines the efficacy of search for perfect hash function parameters
due to the lack of global view of all table entries.

With `TRANSFORM`, a plugin can collect all key tables across input files
at link time and construct a single consolidated, collision-free
lookup table.

==============================================================================
2. Proposed Linker Script Syntax - `TRANSFORM`
==============================================================================

We propose adding a `TRANSFORM` directive inside output section statements.

    SECTIONS
    {
      ...
      .output_section [address] : [ALIGN(align)]
        {
          TRANSFORM ( function_name , section_spec [, section_spec ...] )
        }
      ...
    }

In an early transformation phase such as payload compression, no script
pre-allocation is necessary because the linker has not yet sized or
positioned sections. The plugin returns a dynamically sized buffer,
and the linker creates a synthetic section for it and automatically
propagates the resulting size and alignment into the enclosing output
section.

    .rodata.compressed :
      {
        TRANSFORM (zstd_compress, *(.rodata.raw*), *(foo.o(.data*)))
      }

In a post-relocation transformation phase such as an integrity digest or
cryptographic signature, the script SHALL declare an explicit capacity
envelope using the LENGTH keyword. This is because all symbol addresses and
relocations are already resolved at this stage and introducing any shifting
would invalidate the layout rather easily.
The linker validates that the returned payload fits in the reserved location,
writes the payload at the start of the section, and pads any remaining space
with zeros.

    .module_checksum :
      {
        /* Explicit 16-byte envelope allocated in-place for calc_checksum */
        TRANSFORM LENGTH(16) (calc_checksum, .text, .rodata)
      }

The first argument names the pure function registered by a plugin.
Subsequent arguments specify target sections using standard wildcards,
input file filters, or output section names.

==============================================================================
3. Plugin C ABI Additions in include/plugin-api.h
==============================================================================

The proposed interface integrates directly into the standard GNU ld
transfer vector struct ld_plugin_tv without custom entry points.

------------------------------------------------------------------------------
A. Execution Phases
------------------------------------------------------------------------------

enum ld_plugin_pure_func_phase
{
  /* Evaluated after LTO compilation, prior to garbage collection, ICF,
     and section layout.  Supports dynamic buffer sizing and alignment.  */
  LD_PHASE_POST_LTO = 1,

  /* Evaluated after all relocations and relaxations are resolved.  The
     destination section has a capacity envelope pre-allocated in the
     linker script.  Output size must satisfy *out_size <= capacity.  */
  LD_PHASE_POST_RELOC = 2
};

------------------------------------------------------------------------------
B. Input Section Descriptor
------------------------------------------------------------------------------

struct ld_plugin_input_section_desc
{
  /* Section name.  */
  const char *name;

  /* Raw section contents, or NULL for NOBITS and BSS sections.  */
  const uint8_t *data;

  /* Size of section contents in bytes.  */
  size_t size;

  /* Final virtual memory address, or 0 during POST_LTO.  */
  uint64_t vma;

  /* Section flags such as ELF sh_flags.  */
  uint64_t flags;

  /* Section alignment in bytes.  */
  uint32_t alignment;

  /* Section type such as ELF sh_type.  */
  uint32_t type;
};

------------------------------------------------------------------------------
C. Pure Function Signature and Registration Descriptor
------------------------------------------------------------------------------

/* Pure function signature.  Returns 0 on success, non-zero on error.
   Error messages are reported via the ld_plugin_message callback.  */

typedef int (*ld_plugin_pure_func) (
  const struct ld_plugin_input_section_desc *inputs,
  size_t num_inputs,
  const uint8_t **out_data,
  size_t *out_size,
  uint32_t *out_alignment);

/* Registration descriptor for a pure script function.  */

struct ld_plugin_pure_func_desc
{
  /* Function name matching the identifier in linker scripts.  */
  const char *name;

  /* Execution phase is LD_PHASE_POST_LTO or LD_PHASE_POST_RELOC.  */
  enum ld_plugin_pure_func_phase phase;

  /* Function pointer to execute.  */
  ld_plugin_pure_func func;
};

/* Transfer vector callback for registering a pure script function.  */

typedef enum ld_plugin_status (*ld_plugin_register_script_function) (
  const struct ld_plugin_pure_func_desc *desc);

------------------------------------------------------------------------------
D. Transfer Vector Tag
------------------------------------------------------------------------------

In enum ld_plugin_tag

  LDPT_REGISTER_SCRIPT_FUNCTION = 38

In struct ld_plugin_tv

  union
  {
    ...
    ld_plugin_register_script_function tv_register_script_function;
  } tv_u;

==============================================================================
4. Operational Semantics and Safety Contracts
==============================================================================

Registered functions must be deterministic and thread-safe. Given the
same input descriptors, a function must yield identical output buffers
across runs, and the linker MUST assume so.

When a function sets `*out_data = NULL` with `*out_size > 0`, the linker
allocates a zero-initialized buffer of `*out_size` bytes. In `POST_LTO`
mode, output buffers reside in persistent memory managed by BFD.

Plugins emit diagnostic messages through the `ld_plugin_message`
transfer vector callback and return non-zero from the function to abort
linking.

The linker rejects function registrations matching built-in script
keywords such as `KEEP`, `ALIGN`, `SECTIONS`, `MEMORY`, or `TRANSFORM`.

For `POST_RELOC` transformations, the linker ensures the following sequence
of operations.
- The linker passes the "reserved" size of the destination section through
  `*out_size` when invoking the pure function.
- When the pure function terminates, the linker verifies that `*out_size`
  does not exceed the pre-allocated capacity of the destination section.
- The linker also verifies that the VMA of the output buffer is correctly
  aligned as asserted by `*out_alignment`.
- If *out_size exceeds capacity or alignment fails, the linker halts with a
  fatal error.
- The linker copies the output buffer to the destination section and
  pads any remaining space with zeros.

==============================================================================
5. Minimal Working Plugin and Linker Script Example
==============================================================================

Below is a complete plugin registering both a POST_LTO zstd compression
transform and a POST_RELOC checksum, followed by the companion linker
script demonstrating their invocation.

------------------------------------------------------------------------------
/* example_plugin.c */
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <stdint.h>
#include <zstd.h>
#include <plugin-api.h>

static ld_plugin_message message_cb = NULL;

/* POST_LTO: Compress input section bytes using Zstandard.  */
static int
compress_payload (const struct ld_plugin_input_section_desc *inputs,
                  size_t num_inputs,
                  const uint8_t **out_data,
                  size_t *out_size,
                  uint32_t *out_alignment)
{
  size_t total_in = 0;
  size_t i, offset = 0;
  uint8_t *raw_buf, *comp_buf;
  size_t max_comp_size, comp_size;

  for (i = 0; i < num_inputs; i++)
    total_in += inputs[i].size;

  if (total_in == 0)
    return 0;

  /* Concatenate matched input sections into contiguous buffer.  */
  raw_buf = malloc (total_in);
  if (!raw_buf)
    return 1;

  for (i = 0; i < num_inputs; i++)
    {
      if (inputs[i].data && inputs[i].size > 0)
        {
          memcpy (raw_buf + offset, inputs[i].data, inputs[i].size);
          offset += inputs[i].size;
        }
    }

  /* Compress raw buffer using Zstandard library.  */
  max_comp_size = ZSTD_compressBound (total_in);
  comp_buf = malloc (max_comp_size);
  if (!comp_buf)
    {
      free (raw_buf);
      return 1;
    }

  comp_size = ZSTD_compress (comp_buf, max_comp_size, raw_buf, total_in, 19);
  free (raw_buf);

  if (ZSTD_isError (comp_size))
    {
      if (message_cb)
        message_cb (LDPL_ERROR, "Zstandard compression failed: %s",
                    ZSTD_getErrorName (comp_size));
      free (comp_buf);
      return 1;
    }

  *out_data = comp_buf;
  *out_size = comp_size;
  *out_alignment = 16;
  return 0;
}

/* POST_RELOC: Compute a 16-byte checksum over relocated sections.  */
static int
calc_checksum (const struct ld_plugin_input_section_desc *inputs,
               size_t num_inputs,
               const uint8_t **out_data,
               size_t *out_size,
               uint32_t *out_alignment)
{
  uint8_t *digest = malloc (16);
  if (!digest)
    return 1;

  memset (digest, 0xA5, 16);
  size_t i, j;
  for (i = 0; i < num_inputs; i++)
    {
      if (inputs[i].data)
        {
          for (j = 0; j < inputs[i].size; j++)
            digest[j % 16] ^= inputs[i].data[j];
        }
    }

  *out_data = digest;
  *out_size = 16;
  *out_alignment = 8;
  return 0;
}

/* Standard plugin initialization entry point.  */
enum ld_plugin_status
onload (struct ld_plugin_tv *tv)
{
  struct ld_plugin_tv *entry;
  ld_plugin_register_script_function register_func = NULL;

  for (entry = tv; entry->tv_tag != LDPT_NULL; ++entry)
    {
      if (entry->tv_tag == LDPT_MESSAGE)
        message_cb = entry->tv_u.tv_message;
      else if (entry->tv_tag == LDPT_REGISTER_SCRIPT_FUNCTION)
        register_func = entry->tv_u.tv_register_script_function;
    }

  if (!register_func)
    return LDPS_ERR;

  struct ld_plugin_pure_func_desc desc_lto = {
    .name = "zstd_compress",
    .phase = LD_PHASE_POST_LTO,
    .func = compress_payload
  };
  if (register_func (&desc_lto) != LDPS_OK)
    return LDPS_ERR;

  struct ld_plugin_pure_func_desc desc_reloc = {
    .name = "calc_checksum",
    .phase = LD_PHASE_POST_RELOC,
    .func = calc_checksum
  };
  if (register_func (&desc_reloc) != LDPS_OK)
    return LDPS_ERR;

  return LDPS_OK;
}
------------------------------------------------------------------------------

/* example_script.ld */
SECTIONS
{
  . = 0x400000;
  .text : { *(.text*) }
  .rodata : { *(.rodata*) }

  /* POST_LTO: Compress raw data sections into an allocated payload.  */
  .rodata.compressed :
    {
      TRANSFORM (zstd_compress, *(.raw_data*))
    }

  /* POST_RELOC: Inject 16-byte checksum over final text and rodata.  */
  . = ALIGN(8);
  .checksum :
    {
      TRANSFORM LENGTH(16) (calc_checksum, .text, .rodata)
    }

  .data : { *(.data*) }
  .bss : { *(.bss*) }
}

------------------------------------------------------------------------------

/* Example Linker Invocation */
gcc -shared -fPIC -o example_plugin.so example_plugin.c -lzstd
ld -plugin ./example_plugin.so -T example_script.ld input.o -o output.elf

==============================================================================
6. Alternatives Currently Deployed or Considered
==============================================================================

Various mentioned projects have explored alternatives to achieve the same goals,
but with caveats.

- BoringSSL, AWS-LC use custom post-processing scripts to perform manual
  relocation to stop linkers from arbitrarily shifting sections and applying
  linker relaxations. In doing so, they need to maintain their own mini-linker
  by applying many assumptions about supported compiler and assembler features,
  documented or not, and have suffered on multiple occasions breakage from
  compiler or assembler changes.
- The compression and transformation use cases are often offloaded to homegrown
  build tools without taking advantage of a standard linker feature and many
  caveats such as complication with LTO, code-model and any linker-domain
  features. Not to mention is that these custom build tools are also duplicated
  across many projects.

On a more high-level ground, there is a clear signal that these projects are
forced to breach the abstraction boundary between the build system and the
linker for use cases that would be greatly benefited from a standard interface.

==============================================================================
7. Points for Discussion
==============================================================================

We would welcome community feedback, including your input on the following
questions as starter.

First, is the `TRANSFORM` keyword with section specifications inside
output section statements the preferred representation, or are there
grammar concerns with complex wildcard lists?

Second, does struct `ld_plugin_input_section_desc` provide sufficient
information for general transformations, or should additional
target-specific attributes be exposed?

We have implemented a working prototype in GNU ld and are actively
testing it across these use cases. We look forward to community feedback
and will post the complete patch series once initial design questions
and major concerns are addressed.

Kind regards,
Xiangfei Ding

Signed-off-by: Xiangfei Ding <dingxiangfei2009@protonmail.ch>