Re: Malloc NUMA Aware

["Carlos O'Donell" <carlos at systemhalted dot org>]

On Thu, Jan 2, 2020 at 11:18 PM Nicholas Krause <xerofoify@gmail.com> wrote:
> With Ryzen and other microprocessors being more NUMA based in design due to
> chiplets e.t.c., I curious as to the support for NUMA aware allocations
> in glibc.

NUMA awareness means different things to different people.

(1) A general purpose allocator that is optimal on NUMA systems?

(2) A new API that allows finer grained control over the location of
the memory requested within the hierarchy?

In general glibc's malloc binds a thread to a given arena (collection
of heaps) and attempts to avoid moving the thread during its lifetime.
This behaviour should be helpful on NUMA systems. This resistance to
moving the threads should mean that a thread's own allocations are
more likely to come from a local node and perform well (data
locality). In summary we are aware that there could be NUMA problems,
we think about them as we redesign and refine glibc's malloc, but we
haven't see any reported bad performance. Likewise there are issues to
consider with transparent huge pages (THP).

When using a producer consumer model with threads on different nodes
this becomes problematic. No matter which node you allocate from, if
your producer and consumer threads are on different nodes, then this
will cause performance degradation. Therefore it's often better to pin
such threads on a single node given knowledge of the system hierarchy.

There is a lot of discussion about this around the use of OpenMP on
large >1024 CPU systems. There is also a discussion about a NUMA-aware
allocator for OpenMP as part of the standard.

The only NUMA-aware allocator that I know about with new API's is
being developed by Intel as memkind
(https://github.com/memkind/memkind) but this is completely different
from the current glibc malloc which is a general-purpose system
allocator.

Does that answer your questions?

Cheers,
Carlos.