We were running a unit test on malloc and ran into an edge case that causes
corruption.  I will attach the test program + an strace.

The critical part of the test is:

  /* malloc a lot of blocks near the max possible size. */
  for (i = 0; i < 10000; i++) {
    if (malloc(kMaxSize - i) != NULL) {
      fprintf(stderr, "Oops\n");
      return 1;
    }
  }

When I strace this, I see these mmap calls:

...
mmap2(0xade06220, 110048, PROT_NONE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0)
= -1 EINVAL (Invalid argument)
mmap2(NULL, 2097152, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0)
= 0xb7d94000
munmap(0xb7d94000, 442368)              = 0
munmap(0xb7f00000, 606208)              = 0
mprotect(0xb7e00000, 135168, PROT_READ|PROT_WRITE) = 0
munmap(0xb7e00000, 1048576)             = 0
brk(0x80a0000)                          = 0x80a0000
mmap2(0xade06220, 110048, PROT_NONE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0)
= -1 EINVAL (Invalid argument)
mmap2(NULL, 2097152, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0)
= 0xb7d94000
munmap(0xb7d94000, 442368)              = 0
munmap(0xb7f00000, 606208)              = 0
mprotect(0xb7e00000, 135168, PROT_READ|PROT_WRITE) = 0
munmap(0xb7e00000, 1048576)             = 0
...

It's an mmap on a fixed address which is not an integral page.

Later on:
mmap2(0xade06008, 110584, PROT_NONE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0)
= -1 EINVAL (Invalid argument)
mmap2(NULL, 2097152, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0)
= 0xb7d94000
munmap(0xb7d94000, 442368)              = 0
munmap(0xb7f00000, 606208)              = 0
mprotect(0xb7e00000, 131072, PROT_READ|PROT_WRITE) = 0
munmap(0xb7e00000, 1048576)             = 0
mmap2(0xade06008, 110584, PROT_NONE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0)
= -1 EINVAL (Invalid argument)
mmap2(NULL, 2097152, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0)
= 0xb7d94000
munmap(0xb7d94000, 442368)              = 0
munmap(0xb7f00000, 606208)              = 0
mprotect(0xb7e00000, 131072, PROT_READ|PROT_WRITE) = 0
munmap(0xb7e00000, 1048576)             = 0
mmap2(0xade06000, 110592, PROT_NONE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0)
= 0xade06000
--- SIGSEGV (Segmentation fault) @ 0 (0) ---
+++ killed by SIGSEGV +++

When the test program runs long enough, eventually it mmap's a fixed address
which is on a page boundary.  It's a PROT_NONE so it disables access to
something.  Shortly after that, _int_malloc gets a SIGSEGV and dies.  Here is a
stack trace:

(gdb) bt
#0  0x47270ce3 in _int_malloc () from /lib/tls/i686/cmov/libc.so.6
#1  0x472723b1 in malloc () from /lib/tls/i686/cmov/libc.so.6
#2  0x08048601 in main ()

I think the problem is in this code:

  /* sYSMALLOc */
  if (grow_heap(old_heap, MINSIZE + nb - old_size) == 0) {

And then in grow_heap:

    if((char *)MMAP((char *)h + new_size, -diff, PROT_NONE,
                    MAP_PRIVATE|MAP_FIXED) == (char *) MAP_FAILED)

That line in grow_heap is the only place mmap call with a fixed address and
MAP_PRIVATE|MAP_FIXED.

Our environment is an in-house distro, native i686-pc-linux-gnu, with glibc 2.3.5.

This isn't an important bug because normal programs don't try to allocate
0xFFFFnnnn bytes of memory.  Since we turned it up, I thought it would be useful
to report it.

Let me know if you need any more info, and thanks for your attention.

------- Additional Comment #1 From Michael Elizabeth Chastain 2006-06-15 06:59 -------

Created an attachment (id=1089)
Test program

Run this lots of times.  In 200 runs, I got 168 PASS and 32 seg fault.

------- Additional Comment #2 From Michael Elizabeth Chastain 2006-06-15 07:01 -------

Created an attachment (id=1090)
strace output from running malloc-large

strace output from malloc-large on a crashing run.

------- Additional Comment #3 From Jakub Jelinek 2006-09-07 20:39 -------

Should be fixed in CVS.