Segfault in MAP_NORESERVE mmap above ~4GB

[Erik Bray <erik dot m dot bray at gmail dot com>]

Hi all,

This issue pertains to Cygwin 64-bit.  The following example program
demonstrates the issue:

$ cat mmap_test.c
#include <sys/mman.h>
#include <stdio.h>
#include <windows.h>


#define VSIZE 0x100001000
#define SIZE 0x1000


void foo() {
    void *top, *bot, *c;

    c = mmap(NULL, VSIZE, PROT_READ|PROT_WRITE,
             MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0);

    top = c + VSIZE;
    bot = top - SIZE;

    printf("     c = 0x%016lx\n", c);
    printf("   top = 0x%016lx\n", top);
    printf("   bot = 0x%016lx\n", bot);

    printf("  c[0] = %ul\n", *((unsigned long *)c));
    printf("bot[0] = %ul\n", *((unsigned long *)bot));
}


int main(void) {
    foo();
    return 0;
}

$ gcc mmap_test.c -o mmap_test

$ ./mmap_test.exe
     c = 0x000006feffff0000
   top = 0x000006ffffff1000
   bot = 0x000006ffffff0000
  c[0] = 0l
Segmentation fault (core dumped)

--------------------------------------------------

As you can see, the address stored in `bot` is within the mmap'd
region, but trying to access it results in an access violation, while
addresses low in the region can be accessed.  This is only an issue
because of MAP_NORESERVE, so when the mmap is created it only reserves
an address range for it but does not commit any resources.

Instead, when Cygwin's exception handler receives a
STATUS_ACCESS_VIOLATION [1] it calls into
mmap_is_attached_or_noreserve [2] with the address associated with
access violation.  This searches the records of existing mmap'd
regions until it finds a matching region, and commits enough of that
region to be able to service the memory access.

This search works for lower addresses in the region, but fails for
higher addresses.  From a bit of debugging it seems this is occurring
because the constructor for mmap_record is squeezing the mmap length
into a DWORD [3], and the rest of the problems stem from there.  There
are a few other places related to mmap_record that seem to be treating
the mmap length as a DWORD instead of a size_t.  I would offer a patch
but I've never built Cygwin before so I haven't tested this yet.

Thanks,
Erik

[1] https://github.com/openunix/cygwin/blob/99590589326b5537d549cdd41ca4177ce7051d4a/winsup/cygwin/exceptions.cc#L722
[2] https://github.com/openunix/cygwin/blob/99590589326b5537d549cdd41ca4177ce7051d4a/winsup/cygwin/mmap.cc#L704
[3] https://github.com/openunix/cygwin/blob/99590589326b5537d549cdd41ca4177ce7051d4a/winsup/cygwin/mmap.cc#L275

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple