This is the mail archive of the
mailing list for the glibc project.
Re: malloc: performance improvements and bugfixes
- From: Jörn Engel <joern at purestorage dot com>
- To: Torvald Riegel <triegel at redhat dot com>
- Cc: "GNU C. Library" <libc-alpha at sourceware dot org>, Siddhesh Poyarekar <siddhesh dot poyarekar at gmail dot com>, Joern Engel <joern at purestorage dot org>
- Date: Wed, 27 Jan 2016 15:50:16 -0800
- Subject: Re: malloc: performance improvements and bugfixes
- Authentication-results: sourceware.org; auth=none
- References: <1453767872-19161-1-git-send-email-joern at purestorage dot com> <1453810961 dot 4592 dot 100 dot camel at localhost dot localdomain> <20160126171435 dot GG5745 at Sligo dot logfs dot org> <1453828825 dot 4592 dot 108 dot camel at localhost dot localdomain> <20160126172629 dot GH5745 at Sligo dot logfs dot org> <1453829759 dot 4592 dot 116 dot camel at localhost dot localdomain> <20160126175940 dot GK5745 at Sligo dot logfs dot org> <1453900833 dot 4592 dot 134 dot camel at localhost dot localdomain> <20160127173842 dot GQ5745 at Sligo dot logfs dot org> <1453930664 dot 4592 dot 199 dot camel at localhost dot localdomain>
On Wed, Jan 27, 2016 at 10:37:44PM +0100, Torvald Riegel wrote:
> Second, you can't expect this community to look at code dumps that don't
> follow the community's rules for contributions, for obvious reasons.
That is ok. I happen to have made improvements to malloc. Those
improvements might help others as well. If they do, they are now
publically available. If you don't want them, that is fine. And it
really doesn't matter why you don't want them.
I also don't expect you to improve libc malloc for my purposes. I have
a perfectly fine fork and can use that forever. There are several other
fine allocators available as well.
If someone wants to improve libc malloc, I can help. Whether anyone
working on libc malloc actually wants my help is doubtful - we don't
seem to see eye-to-eye on very many things. But I can help.
> Third, without discussing workload patterns, it's not clear whether your
> contributions actually improve the situation because we don't know what
> "the situation" is precisely (as it depends on your use cases). (Some
> changes may be generally good, but I suspect that most affect some
> trade-off (e.g., see the discussion with Szabolcs).
You don't have to improve my situation. I would be happy if you improve
the situation for any open-source program that currently used jemalloc
or tcmalloc. Target mariadb, if you like.
> > The kernel's auto-NUMA falls apart as soon as either the memory or the
> > thread moves to the wrong node. That happens all the time. And the
> > kernel will only give you NUMA-local memory once, at allocation time.
> > If the application holds on to that memory for days or years, it such
> > one-time decisions don't matter.
> My recollection is that newer auto-NUMA (or a daemon -- I don't remember
> the names precisely) will indeed move pages to the nodes they are
> accessed from. Either way, if the page or thread moves to a place
> further away in the memory hierarchy, there's little the allocator can
> do; what it can do is try to make that less likely, for example by
> getting the initial allocation right (ie, allocate in a page that's
> local at allocation time) -- but this again depends on the allocation
> and usage patterns in the program to some extent. Hence my question
> about these in your program.
Interesting. Moving pages around shouldn't cause too much latency and
can fix long-term problems. But if you move pages back and forth too
often, there are almost certainly better solutions to the problem. Even
doing nothing is likely better.
> > That is why my code calls getcpu() to see which NUMA node we are on
> > right now and then returns memory from a NUMA-local arena. The kernel
> > might migrate the thread between getcpu() and memory access, but at
> > least you get it right 99% of the time instead of 50% (for two nodes).
> > So if you want a model for this, how about:
> > - multiple threads,
> > - no thread affinities,
> > - memory moves between threads,
> > - application runs long enough to make kernel's decision irrelevant.
> That's less than the level of detail we need. It's obvious we have to
> consider multiple threads, but how many exactly?
> Not selecting thread affinities is a practical choice, though if your
> program owns the machine as you indicated earlier, pinning threads to
> CPUs isn't an unusual thing to do.
> I'm not sure what you mean by "memory moves between threads".
I mean that threads pass memory allocated by malloc between each other.
The memory is freed on a different thread than it was allocated on.
> You also include a fork torture test. Is fork really called frequently
> in your big application?
Not by a long shot. But I was concerned about introducing a bug
somewhere in the fork handling, so I created a stresstest.
> > So if you genuinely want to help, maybe the best thing would be to
> > extract the testcases from all these projects, create a new "malloctest"
> > or whatever and make it easy to evaluate and compare allocators with
> > your test. Pass/fail would be great, benchmarks would be even better.
> > Then shame the guilty parties into improvements.
> Something similar is what we would suggest to users: If they believe
> glibc has a performance problem, and they want to help fix it, they
> should describe the problem and workload in sufficient detail, and add
> performance tests, microbenchmarks, or similar that allows glibc folks
> to (1) classify the performance problem, (2) have a measurement that is
> considered to be a good indicator of the problem. If we then can agree
> that the performance problem is likely relevant for real-world use
> cases, we have an actionable task for improving this (because it's clear
> what the goal is, whether it can be measured, and it can be
> regression-tested in the future). Note that we'll also have to
> trade-off against other uses if general-purpose code is affected, given
> that we have to serve all users.
For whatever reasons, people don't seem to do that. But maybe you can
find out why the mariadb people moved to jemalloc.
Or you could ask this person.
Or you can grep every package that your distribution carries for one of
the alternative allocators and ask the package maintainers. They should
be more willing to discuss the finer details, as their applications are
all open source and the question of keeping details secret never
I do not know how much of our implementation details we are willing to
give out. But being conservative and giving out as little as possible
seems like a good first approach. That's why I try to direct you
elsewhere. I hope you understand that.
There are two ways of constructing a software design: one way is to make
it so simple that there are obviously no deficiencies, and the other is
to make it so complicated that there are no obvious deficiencies.
-- C. A. R. Hoare