Problem in pthread_create (allocate_stack inline code) on MIPS.

["Kaz Kylheku" <kaz at zeugmasystems dot com>]

Hey everyone,

We've discovered a problem of a large application (many processes, with
lots of threads) occupying an unusually large amount of virtual memory
on mips/n32 but not on the i686 version of the same software and OS. 

The problem is that setting the pthread stack size attribute on MIPS
doesn't seem to be doing what it's supposed to, and so the threads end
up with some incorrect stack size.

Note, by the way, that I am still on glibc 2.5 here!

If we look at allocate_stack in nptl/allocatestack.c, we have this nasty
GCC extension:

  size = attr->stacksize ?: __default_stacksize;

This is apparently being mistranslated by gcc 4.1.1. Look at this:

  # v1 := attr->stacksize
    5410:       8e630014        lw      v1,20(s3)

  # if (v1 == 0) goto 5778
    5414:       106000d8        beqz    v1,5778
<pthread_create@@GLIBC_2.2+0x3d8>

  # s1 := v1
    5418:       0060882d        move    s1,v1

  # v0 := attr->flags
    541c:       8e620008        lw      v0,8(s3)

  # if ((v0 & ATTR_FLAG_STACKADDR) != 0) goto 5b30
    5420:       30420008        andi    v0,v0,0x8
    5424:       144001c2        bnez    v0,5b30
<pthread_create@@GLIBC_2.2+0x790

  # ... etc

So the logic is: the stack size is loaded into v1. If it's zero, then we
branch to some fixup code located elsewhere which implements the
alternative clause. Either way, the size is then copied into s1. Here is
the alternate code at 5778:

  # v0 := __default_stacksize     (presumably! Haven't verified this
global offset)
    5778:       8f82804c        lw      v0,-32692(gp)
 
  # goto 541c
    577c:       1000ff27        b       541c
<pthread_create@@GLIBC_2.2+0x7c>
 
This is inconsistent register use! The v0 register is wrong, because it
gets clobbered right away by the instruction at the branch target 541c.
How can the default stack size ever be used, in the common case?

This should be loading v1, and jumping back to 5418, so v1 is copied
into s1.

I have no idea what the stack calculation will actually end up doing; I
stopped looking when I found this.

Our application does use a nonzero value for the stack size, and so it
shouldn't be taking the branch to 5778, yet it's not behaving as it
should.