thread heap leak?

[David Muse]

On Fri, 22 Mar 2019 18:31:01 +0100
Florian Weimer <fw@deneb.enyo.de> wrote:

> * David Muse:
> 
> > Over time, we get more and more of them until top shows the app's VIRT
> > to be around 4G (I think, maybe just 2G).  Then it crashes.  The RES
> > is never more than a few MB.
> 
> Do you have backtraces from the crash?  How did you determine that the
> crashes and the anonymous mappings are related?
> 
> How do you launch the detached threads?  Do you use any other thread
> attributes?
> 

I've struggled to get backtraces.  The app has a crash-handler that prints a backttrace to the log, but that also crashes inside of a malloc.  Getting a core on the production system has been a challenge too.  I'll see if I can get that.

I'm not 100% sure that they are related.

I have a monitor running that does a top every 30 seconds or so.  The pattern is always that the app's VIRT grows to about 2G and then it crashes.  We figured it was a memory leak, so we started running it through valgrind.  We'd see the same memory usage, but valgrind would report no leaks, and only a few K of "still reachable" memory.  This was the same whether the app crashed, or whether we just killed it after a few hours.  I eventually noticed the anonymous segments in /proc/<pid>/maps, did some math on them, and them + [stack] roughly added up to the VIRT size.  So, I figured they were responsible for the VIRT growth.

It's not clear why the app is crashing at about 2G.  I'd think that a 64-bit process ought to be able to address more than that.  But, since it always crashes at that size, it seems like it's related.

Code to launch detached threads:

... main ...

	cs->threadattr=new pthread_attr_t;
	pthread_attr_init(cs->threadattr);
	pthread_attr_setdetachstate(cs->threadattr,PTHREAD_CREATE_DETACHED);
	...
	cs->threadhandle=new pthread_t;
	if (pthread_create(cs->threadhandle,cs->threadattr,
				(void *(*)(void *))clientThread,
				(void *)cs)) {
		... error handling ...
	}


... inside of clientThread() ...

        pthread_attr_destroy(cs->threadattr);
	...
	pthread_exit(NULL);


No other attributes.

Thanks!

David
david.muse@firstworks.com