[RFC] Stack allocation, hugepages and RSS implications

[Florian Weimer]

* Cupertino Miranda via Libc-alpha:

> Hi everyone,
>
> For performance purposes, one of ours in-house applications requires to enable
> TRANSPARENT_HUGEPAGES_ALWAYS option in linux kernel, actually making the
> kernel to force all of the big enough and alligned memory allocations to
> reside in hugepages.  I believe the reason behind this decision is to
> have more control on data location.
>
> For stack allocation, it seems that hugepages make resident set size
> (RSS) increase significantly, and without any apparent benefit, as the
> huge page will be split in small pages even before leaving glibc stack
> allocation code.
>
> As an example, this is what happens in case of a pthread_create with 2MB
> stack size:
>  1. mmap request for the 2MB allocation with PROT_NONE;
>       a huge page is "registered" by the kernel
>  2. the thread descriptor is writen in the end of the stack.
>       this will trigger a page exception in the kernel which will make the actual
>       memory allocation of the 2MB.
>  3. an mprotect changes protection on the guard (one of the small pages of the
>     allocated space):
>       at this point the kernel needs to break the 2MB page into many small pages
>       in order to change the protection on that memory region.
>       This will eliminate any benefit of having small pages for stack allocation,
>       but also makes RSS to be increaded by 2MB even though nothing was
>       written to most of the small pages.
>
> As an exercise I added __madvise(..., MADV_NOHUGEPAGE) right after the
> __mmap in nptl/allocatestack.c. As expected, RSS was significantly
> reduced for the application.

Interesting.  I did not expect to get hugepages right out of mmap.  I
would have expected subsequent coalescing by khugepaged, taking actual
stack usage into account.  But over-allocating memory might be
beneficial, see below.

(Something must be happening between step 1 & 2 to make the writes
possible.)

> In any case, I wonder if there is an actual use case where an hugepage would
> survive glibc stack allocation and will bring an actual benefit.

It can reduce TLB misses.  The first-level TLB might only have 64
entries for 4K pages, for example.  If the working set on the stack
(including the TCB) needs more than a couple of pages, it might
beneficial to use a 2M page and use just one TLB entry.

In your case, if your stacks are quite small, maybe you can just
allocate slightly less than 2 MiB?

The other question is whether the reported RSS is real, or if the kernel
will recover zero stack pages on memory pressure.

Thanks,
Florian