mmap use in linuxthreads

[Jes Sorensen]

Hi

I was pointed at a problem with the way we are using mmap() within
libpthread. It seems that the Linux kernel will kill an existing mapping
silently if you try to mmap(MAP_FIXED) on a range overlaying an existing
mapping for a task. According to SUSv2 this seems to be within spec
( http://www.opengroup.org/public/pubs/online/7908799/xsh/mmap.html ):

      When MAP_FIXED is set in the flags argument, the implementation is
      informed that the value of pa must be addr, exactly. If MAP_FIXED
      is set, mmap() may return MAP_FAILED and set errno to [EINVAL]. If
      a MAP_FIXED request is successful, the mapping established by
      mmap() replaces any previous mappings for the process' pages in
      the range [pa, pa + len).

However linuxthreads/manager.c seems to rely on mmap() returning
MAP_FAILED in case you have a conflict, for instance this part:

      if (mmap((caddr_t)((char *)(new_thread + 1) - stacksize / 2),
		stacksize / 2, PROT_READ | PROT_WRITE | PROT_EXEC,
		MAP_PRIVATE | MAP_ANONYMOUS | MAP_FIXED, -1, 0)
          == MAP_FAILED)
	/* Bad luck, this segment is already mapped. */
	return -1;
      /* Then the register stack:       */
      if (mmap((caddr_t)new_thread_bottom, stacksize/2,
		PROT_READ | PROT_WRITE | PROT_EXEC,
		MAP_PRIVATE | MAP_ANONYMOUS | MAP_FIXED, -1, 0)
          == MAP_FAILED)
	{
          munmap((caddr_t)((char *)(new_thread + 1) - stacksize/2),
                 stacksize/2);
          return -1;
	}

Hence the question is what to do here, should one try to fix
linuxthreads to check for a conflict before trying a mapping or lobby
for the kernel to be changed?

Ideas?

Jes