Re: thread heap leak?

[David Muse <david dot muse at firstworks dot com>]

On Fri, 22 Mar 2019 13:45:42 -0400
"Carlos O'Donell" <codonell@redhat.com> wrote:

> On 3/22/19 1:39 PM, David Muse wrote:
> > On Fri, 22 Mar 2019 13:03:51 -0400 "Carlos O'Donell"
> > <codonell@redhat.com> wrote:
> > 
> >> On 3/22/19 10:53 AM, David Muse wrote:
> >>> We have this little server program that listens for client 
> >>> connections on an inet socket.  When it receives a connection,
> >>> it pthread_create()'s a detached thread to handle the request and
> >>> the main thread waits for more client connections.
> >> 
> >> The stack for the detached thread is recoverable only if the
> >> thread exits, at which point it calls 	__free_tcb () (only if
> >> detached, because otherwise only pthread_join() does this
> >> recovery), enqueues the stack onto the free list and leaves it
> >> there. Subsequent thread creation attemps to reuse the stack, but
> >> must wait for the kernel to clear the registered tid memory
> >> (CLONE_CHILD_CLEARTID) to mark the stack reusable (FREE_P). The
> >> total size of the thread stack cache should only be ~40MiB, and new
> >> stack creation triggers an automatic check to trim this cache back
> >> down to 40MiB.
> >> 
> >> Have you been able to reduce this to a smaller test case?
> >> 
> >> The anonymous mappings might indeed be thread stacks, but you'd 
> >> have to verify that yourself, and you can do that by asking the 
> >> thread for the stack information and printing that.
> >> 
> >> Are you using Amazon Linux or a known distro upstream?
> > 
> > I did comment out lots of features in the program to narrow down the
> > problem, but I haven't been able to reproduce the leak in a compact
> > test case.  I've been working on that though.
> 
> Good luck :-)
> 
> > How do I get the stack information from the thread?  I know that I
> > can do pthread_attr_getstacksize for the size, but what else can I
> > get?
> 
> You can use pthread_getattr_np() to take a snapshot of the thread's
> attributes and look at those, that will get you the information about
> stack address location and size.
> 
> You should try using something like systemtap to put tap points on
> your program and observe the behaviour of the detached threads.
> 
> 
> > We're using OpenSUSE Leap 42.3.  I've seen the problem directly
> > there.  I've gotten reports that it also occurs on RHEL 7, but I
> > can't confirm that.
> 
> Andreas,
> 
> Have you seen reports like this?
> 
> -- 
> Cheers,
> Carlos.
> 

Thanks for the advice.  I'll try pthread_getattr_np() and systemtap and see what info I can get.

Thanks,

David
david.muse@firstworks.com