This is the mail archive of the
libc-help@sourceware.org
mailing list for the glibc project.
Re: glibc 2.26 memory leak with MySQL?
- From: Carlos O'Donell <carlos at redhat dot com>
- To: Michael Brunnbauer <brunni at netestate dot de>, libc-help at sourceware dot org
- Date: Mon, 14 Aug 2017 08:54:50 -0400
- Subject: Re: glibc 2.26 memory leak with MySQL?
- Authentication-results: sourceware.org; auth=none
- References: <20170812141938.GA26871@netestate.de>
On 08/12/2017 10:19 AM, Michael Brunnbauer wrote:
> since upgrading from glibc-2.23 to 2.26, I have observed out of memory errors
> in 2 different mysql deployments after 2-3 days of runtime:
>
> 170812 15:51:42 [ERROR] /usr/bin/mysqld: Out of memory (Needed 1675264 bytes)
> 170812 15:52:01 [ERROR] /usr/bin/mysqld: Out of memory (Needed 1677712 bytes)
This is not a lot of data to go on.
A graph of VmRSS and VmSize over time would be ideal to see what the process
is doing.
There are generally four key causes of memory usage:
* Application demand.
* External fragmentation.
* Internal fragmentation.
* Defect in glibc's malloc.
Our goal should be to identify exactly which of these cases is the problem.
Keep in mind that with the first 3 cases, there might be no easy fix,
particularly if you're deploying into a tightly memory constrained environment.
> I've never had such Problems with those mysql deployments before and one
> of them is configured for very modest memory usage.
>
> I use mysql 5.5.57 and Kernel 3.16.46. glibc has been compiled with gcc-4.9.4
> like this:
>
> echo "CFLAGS += -march=i686" > configparms
> ../configure --enable-add-ons --prefix=/usr --enable-kernel=2.6.25 --host=i686-pc-linux-gnu
>
> MySQL has not been recompiled since upgrading glibc and gcc (previous version
> 4.8.5).
>
> Have there been any changes that can explain this, e.g. the per-thread cache
> for malloc?
As Siddhesh notes, we added a thread-local cache to malloc.
Rather than rebuild with --disable-experimental-malloc, you can just use the
runtime tunables to disable the cache.
Cheers,
Carlos.