Malloc NUMA Aware

[Nicholas Krause]

On 1/4/20 9:54 AM, Carlos O'Donell wrote:
> On Thu, Jan 2, 2020 at 11:18 PM Nicholas Krause <xerofoify@gmail.com> wrote:
>> With Ryzen and other microprocessors being more NUMA based in design due to
>> chiplets e.t.c., I curious as to the support for NUMA aware allocations
>> in glibc.
> NUMA awareness means different things to different people.
>
> (1) A general purpose allocator that is optimal on NUMA systems?
>
> (2) A new API that allows finer grained control over the location of
> the memory requested within the hierarchy?
>
> In general glibc's malloc binds a thread to a given arena (collection
> of heaps) and attempts to avoid moving the thread during its lifetime.
> This behaviour should be helpful on NUMA systems. This resistance to
> moving the threads should mean that a thread's own allocations are
> more likely to come from a local node and perform well (data
> locality). In summary we are aware that there could be NUMA problems,
> we think about them as we redesign and refine glibc's malloc, but we
> haven't see any reported bad performance. Likewise there are issues to
> consider with transparent huge pages (THP).
>
> When using a producer consumer model with threads on different nodes
> this becomes problematic. No matter which node you allocate from, if
> your producer and consumer threads are on different nodes, then this
> will cause performance degradation. Therefore it's often better to pin
> such threads on a single node given knowledge of the system hierarchy.
>
> There is a lot of discussion about this around the use of OpenMP on
> large >1024 CPU systems. There is also a discussion about a NUMA-aware
> allocator for OpenMP as part of the standard.
>
> The only NUMA-aware allocator that I know about with new API's is
> being developed by Intel as memkind
> (https://github.com/memkind/memkind) but this is completely different
> from the current glibc malloc which is a general-purpose system
> allocator.
>
> Does that answer your questions?
>
> Cheers,
> Carlos.
I wasn't sure how much the areas helped so that does answer my
question.
Nick