Sourceware Bugzilla – Bug 13137
malloc M_ARENA_MAX / MALLOC_ARENA_MAX limit not working correctly (limit exceeded)
Last modified: 2012-10-17 09:17:47 UTC
We have an huge application which makes very heavy use of threads (typical 500 - 2000 threads running in parallel for task handling). In one setup around 5 GB of physical memory are used.
The same application on Windows (using MS VC++) uses only 60% with exactly the same configuration. The reason for that is the glibc malloc() implementation using multiple arenas to prevent thread contention. Using malloc_info() we found out that 2 - 3 GB of memory are unused (free) in the memory management. As that free memory is distributed between the areans it can't be reclaimed. The application uses 64 arenas (maximum is 8 per core? we have 8 core test system) and our threads depending on the task need sometimes a lot of memory (several 100MB). Since the thread if it has bad luck gets uses only one arena it can't use the free memory of the others.
Our solution was to try to limit the number of arenas in the malloc implementation. For that the M_ARENA_MAX / MALLOC_ARENA_MAX parameters (environment or mallopt() are available. According to the documentation they define the maximum number of arenas used.
In our tests we configured for only 1 arena. But the application tends to use around 10 arenas which is more than the specified maximum. It is of course less than the 64 arenas if the parameters not sed and works also better in terms of free memory fragmentation.
But the limit is exceed which seems to be a bug since the documentation also in Ulrich Dreppers blog defines that as a hard limit.
We tested e.g. on RHEL 6 with latest updates.
The minimum arena limit is 8.
This should have been fixed by the following commit:
Author: Ulrich Drepper <email@example.com>
Date: Tue Jan 31 14:42:34 2012 -0500
Handle ARENA_TEST correctly
If you can verify with glibc-2.16, then we can close this. Otherwise please provide a reproducer so that we can analyze and fix the problem.
Closing as fixed. Please reopen if you're able to reproduce the problem.