glibc 2.26 memory leak with MySQL?

Carlos O'Donell carlos@redhat.com
Mon Aug 14 12:54:00 GMT 2017


On 08/12/2017 10:19 AM, Michael Brunnbauer wrote:
> since upgrading from glibc-2.23 to 2.26, I have observed out of memory errors
> in 2 different mysql deployments after 2-3 days of runtime:
> 
> 170812 15:51:42 [ERROR] /usr/bin/mysqld: Out of memory (Needed 1675264 bytes)
> 170812 15:52:01 [ERROR] /usr/bin/mysqld: Out of memory (Needed 1677712 bytes)

This is not a lot of data to go on.

A graph of VmRSS and VmSize over time would be ideal to see what the process
is doing.

There are generally four key causes of memory usage:

* Application demand.

* External fragmentation.

* Internal fragmentation.

* Defect in glibc's malloc.

Our goal should be to identify exactly which of these cases is the problem.

Keep in mind that with the first 3 cases, there might be no easy fix,
particularly if you're deploying into a tightly memory constrained environment.

> I've never had such Problems with those mysql deployments before and one
> of them is configured for very modest memory usage.
> 
> I use mysql 5.5.57 and Kernel 3.16.46. glibc has been compiled with gcc-4.9.4 
> like this:
> 
>  echo "CFLAGS += -march=i686" > configparms
>  ../configure --enable-add-ons --prefix=/usr --enable-kernel=2.6.25 --host=i686-pc-linux-gnu
> 
> MySQL has not been recompiled since upgrading glibc and gcc (previous version
> 4.8.5).
> 
> Have there been any changes that can explain this, e.g. the per-thread cache
> for malloc?

As Siddhesh notes, we added a thread-local cache to malloc.

Rather than rebuild with --disable-experimental-malloc, you can just use the
runtime tunables to disable the cache.

Cheers,
Carlos.



More information about the Libc-help mailing list