[PATCH v4 1/2] malloc: Randomize the alignment of the thread heaps [BZ #22853]

Hector Marco-Gisbert hecmargi@upv.es
Wed Nov 11 16:41:59 GMT 2026


Hi Adhemerval, Florian,

Thank you both for the detailed review, it is very helpful. I am
answering the two patches in a single message.

I agree with almost everything and I will send a v5.  There is one
point where I would like to hear your opinion before writing it, the
behavior with ASLR disabled, so I start with that one.


1. ADDR_NO_RANDOMIZE

> How predictable are thread heap addresses today under GDB?  If they
> are predictable today, I think we should preserve this behavior.

They are fully predictable today.  With the unpatched library and
setarch -R, a thread that calls malloc gets the same address in 100 of
100 runs (x86_64).  With v4 the addresses under setarch -R are exactly
those same addresses, and the same happens under GDB, which disables
ASLR by default.  Without the personality check they would change on
every run.

Adhemerval is right that v4 does not honor
kernel.randomize_va_space=0.  With the sysctl at 0 the unpatched
library gives one thread heap address in 10,000 runs and v4 gives the
full 14 bits.  The check covers only the per-process flag, which is
what setarch -R and GDB use.

With the current information I am still inclined to keep the check.
I understand the comparison with the stack protector canary and the
tcache key, but I think they are a different kind of thing.  They are
secret values stored in memory, and they do not change where anything
is mapped.  What a user expects from setarch -R or from GDB is that
the addresses repeat from run to run, and the residue is the only item
of the three that changes addresses.

About the cost, I measured it again (x86_64, 1,000,000 calls, five
rounds, machine not isolated):

  personality (0xffffffff)      about 100 ns
  getrandom, 8 bytes            about 267 ns
  getrandom, 16 bytes           about 266 ns

If the residue comes from the getrandom call that
tcache_key_initialize already makes, as you suggest (and I agree), the
series adds only the personality call, about 0.1 us per process.

About seccomp, I tested it and Adhemerval's point is stronger than I
first thought.  If the filter returns an error for personality, v4
treats it as "ASLR enabled" and continues.  This is the case with
LockPersonality=yes in systemd and with any filter that has an errno
default action, and the default Docker and Podman profiles allow
personality (0xffffffff) explicitly.  But if an inherited filter kills
on a system call that is not listed, v4 kills the process at start-up,
threaded or not.  I checked it with a minimal seccomp filter and the
process gets SIGSYS before main.  As far as I can tell from the
documentation, this is what a systemd unit gets with an allow-list
SystemCallFilter= that does not include @system-service and has no
SystemCallErrorNumber=.  It does not seem to be the usual case (the
seven units with SystemCallFilter= on the Debian 12 system I looked at
all include @system-service), but it exists.

About the delayed initialization, there is a single place where it
can be done, _int_new_arena, with a lock or an atomic exchange because
several threads can be there at the same time.  It removes the cost
from single-threaded processes.  My doubt is that the call would then
happen after main, so it is exposed not only to the inherited filters
but also to the filter that the program installs itself.

So I see three options:

  a) Keep the check at start-up, sharing getrandom with the tcache
     key.  About 0.1 us per process and no call after main, but a
     process under an inherited filter that kills and does not list
     personality dies at start-up.
  b) Delayed check in _int_new_arena, as Florian suggests.  No cost
     and no risk for single-threaded processes, but in threaded ones
     the call happens after main, under any filter of the program.
  c) No check, and a tunable to get reproducible heaps, as Adhemerval
     suggests.  No system call and no seccomp risk, but GDB and
     setarch -R stop giving reproducible thread heaps unless the user
     also sets the tunable.

I lean towards (a), because I think that reproducible addresses under
GDB and setarch -R are worth 0.1 us and the filters that would kill
seem uncommon, and (b) is also fine for me.  What do you think?

The Hurd build is my mistake.  Whatever the option, the personality
code will be Linux only in v5.


2. Transparent huge pages and heap_trim

You are right, and it is a bigger problem than I expected.  I measured
it with one thread that grows its heap to 30 MiB, trims it and grows
it again, three cycles, with glibc.malloc.hugetlb=1 on x86_64 (Linux
6.8) and with the default configuration on AArch64 (Linux 6.12).  The
figures are the same on both:

               AnonHugePages after growing     thp_split_pmd
               cycle 1    cycle 2    cycle 3
  unpatched    32768 kB   32768 kB   32768 kB        0
  v4           30720 kB       0 kB       0 kB       15
  prototype    30720 kB   30720 kB   30720 kB        0

With v4, trimming the heap for the first time splits its 15 huge
pages, one per 2 MiB released, and when the heap grows again it has no
huge pages at all (khugepaged did not collapse them during the test,
which lasts a few seconds).

"prototype" is your proposal: the residue stays page granular and the
end of the heap, not only the accessible range, is placed on an
absolute huge page boundary in alloc_new_heap, grow_heap and
heap_trim.  It is a small change on top of v4 (arena.c only, 38 lines
added and 41 removed), heap_prot_size is replaced by a helper that
computes the heap size, and the assert in heap_trim is the original
one again, except for a heap that fills its reservation.  The residue
is not touched, so the entropy is the same as in v4 (14 bits on x86_64
and AArch64 with 4 KiB pages).  The 15 huge pages instead of 16 are
the first part of the heap, up to the first 2 MiB boundary, which
cannot be a huge page.  That is at most one huge page per heap.

It also fixes glibc.mem.decorate_maps=1.  With v4 the named VMA ends
in the middle of a huge page and the heap loses one more huge page
(28672 kB instead of 30720 kB).  With the prototype the named VMA ends
on a huge page boundary again, as in the unpatched library.

I will use this approach in v5 and fix the comment about THP.  So far
the prototype passes the malloc tests on x86_64 only, and I have only
counters, no timings.  If you would like to see timings or any other
workload, please tell me.


3. The rest of 1/2

All agreed for v5:

  - One getrandom call for the tcache key and the residue, with
    independent words, and tcache_key_initialize taking the key as an
    argument.
  - No residue code in libc_malloc_debug.so.
  - "Copyright The GNU Toolchain Authors." in arena.c.
  - NEWS adjusted to whatever we decide in point 1.


4. The test (2/2)

Also agreed:

  - Copyright line.
  - A thread that grows its arena to three heaps, checks that all of
    them have the same residue, and frees everything in reverse order
    so that heap_trim deletes the heaps.  This should also exercise
    aligned_heap_area and grow_heap.
  - The three heaps case will run with hugetlb=1 as well (the test
    is already built in the hugetlb1 and hugetlb2 variants).
  - The distinct residues check takes into account the number of bits
    of the residue.  You are right that 2^-120 does not hold with
    hugetlb=2 (5 bits with 2 MiB pages, 1 bit with 16 KiB pages and
    32 MiB huge pages).
  - support_spawn_wrap instead of building the arguments by hand.
  - The ADDR_NO_RANDOMIZE part of the test follows point 1.

Thanks again for your time.  I will prepare v5 once we settle
point 1.

Hector


More information about the Libc-alpha mailing list