[PATCH] malloc: Reduce maximum arenas

[Wilco Dijkstra]

Hi DJ,

> Do we have a sitation where we hit the limit, but didn't need to?

Yes, there are various bug reports and complaints that GLIBC creates way
too many arenas. This was before the really big servers with 128+ cores
appeared, so the situation is now far worse...

> Reading the sources, the number of arenas actually used depends on how
> many threads are simultaneously inside the malloc code.  So if you hit
> the max, that means you actually had that many threads actually doing
> malloc() at the same time...

That's not how it works. We create a new arena for every thread that happens
to use malloc, even if only once. So if say you create a bunch of worker threads
that are just sitting idle, they get their own arena and are just wasting memory.

So this is a conservative improvement - we'll need to further reduce the
number of arenas and improve concurrency.

> I mean, I agree that 8 per core is extreme for some machines - by
> definition, we shouldn't be able to have more than one thread *running*
> in malloc at a time per cpu, but for low-cpu low-ram machines that swap,
> 8 per core might be reasonable - a thread that's scheduled out shouldn't
> block all other malloc'ing threads.

Threads that are forced to wait will sleep and the thread that holds the lock
should get priority to finish and then the sleeping threads will be woken up
once finished.

Malloc could be improved significantly. Firstly it shouldn't ever hold the lock
for long (like I mentioned realloc doing a big memcpy while locked is a really
bad design). The lock should be more finegrained too, there is no need to use
one global lock when you could use different locks for different chunk sizes.
And we can do far more in tcache without needing locks. Finally, rather than
blocking on a lock, we could try the lock and if it fails use another arena
as a fallback. There is also a possibility to spin for a short time if we can stop
malloc holding onto locks for long (for example the unsorted bin scan will
still process up to 10000 blocks, each of which may need sorting into their
bin, potentially taking many millions of cycles).

Cheers,
Wilco