thread heap leak?

[Carlos O'Donell]

On 3/22/19 1:53 PM, David Muse wrote:
> On Fri, 22 Mar 2019 18:31:01 +0100
> Florian Weimer <fw@deneb.enyo.de> wrote:
> 
>> * David Muse:
>>
>>> Over time, we get more and more of them until top shows the app's VIRT
>>> to be around 4G (I think, maybe just 2G).  Then it crashes.  The RES
>>> is never more than a few MB.
>>
>> Do you have backtraces from the crash?  How did you determine that the
>> crashes and the anonymous mappings are related?
>>
>> How do you launch the detached threads?  Do you use any other thread
>> attributes?
>>
> 
> I've struggled to get backtraces.  The app has a crash-handler that prints a backttrace to the log, but that also crashes inside of a malloc.  Getting a core on the production system has been a challenge too.  I'll see if I can get that.
> 
> I'm not 100% sure that they are related.
> 
> I have a monitor running that does a top every 30 seconds or so.  The pattern is always that the app's VIRT grows to about 2G and then it crashes.  We figured it was a memory leak, so we started running it through valgrind.  We'd see the same memory usage, but valgrind would report no leaks, and only a few K of "still reachable" memory.  This was the same whether the app crashed, or whether we just killed it after a few hours.  I eventually noticed the anonymous segments in /proc/<pid>/maps, did some math on them, and them + [stack] roughly added up to the VIRT size.  So, I figured they were responsible for the VIRT growth.
> 
> It's not clear why the app is crashing at about 2G.  I'd think that a 64-bit process ought to be able to address more than that.  But, since it always crashes at that size, it seems like it's related.
> 
> Code to launch detached threads:
> 
> ... main ...
> 
> 	cs->threadattr=new pthread_attr_t;
> 	pthread_attr_init(cs->threadattr);
> 	pthread_attr_setdetachstate(cs->threadattr,PTHREAD_CREATE_DETACHED);
> 	...
> 	cs->threadhandle=new pthread_t;
> 	if (pthread_create(cs->threadhandle,cs->threadattr,
> 				(void *(*)(void *))clientThread,
> 				(void *)cs)) {
> 		... error handling ...
> 	}
> 
> 
> ... inside of clientThread() ...
> 
>          pthread_attr_destroy(cs->threadattr);
> 	...
> 	pthread_exit(NULL);
> 
> 
> No other attributes.

If the detached thread exits the memory should get reaped
(eventually, the kernel is sometimes slow at reaping).

You really need to instrument the thread stack/size and
correlate the anonymous mappings to the threads, and figure
out why you crash.

When a thread exits we use madvise (MADV_DONTNEED) to mark
the pages of the stack (minus PTHREAD_STACK_MIN) as unused.
This means VmSIZE remains high, but VmRSS is reduced.
However, eventually when the stack cache exceeds 40MiB we
start unmapping the entire caches.

I wonder if your OS has a customization to increase the stack
cache size? Are you able to debug the value of 'stack_cache_maxsize'
that you see from nptl/allocatestack.c?

-- 
Cheers,
Carlos.