Address space clobbers during fork() (was Re: Extending /proc/*/maps)
Ryan Johnson
ryan.johnson@cs.utoronto.ca
Tue Apr 19 18:17:00 GMT 2011
On 19/04/2011 12:01 PM, Corinna Vinschen wrote:
> On Apr 19 11:38, Ryan Johnson wrote:
>> BTW, /cygdrive/c/Windows/System32/locale.nls seems to be the reason
>> that even statically linked dlls don't always load in the same place
>> twice in a row.
> How so? And then, when is it loaded? I assume that this file is
> fetched as soon as the GetLocaleInfo function is called. What OS
> are you using? If it's Vista or later, there's a chance that the
> LocaleNameToLCID function is the culprit as well. But the real
> problem is, how can that be worked around? For the locale stuff
> we need these functions.
(subject changed to reflect what we're actually discussing)
I'm using Win7-x64.
When locale.nls gets mapped in, Windows doesn't always put it in the
same place. As a result, fork() can run into all the usual problems with
DLL base addresses, except it can bite even statically-linked dlls.
Below is an example of how the memory layouts can differ right from the
start. The diff comes from memory maps taken just before the parent
forked and just after the child detected a problem (in dll_list::alloc,
called by dll_dllcrt0_1).**
Recall that cygfoo.dll and cygbar.dll both want to load at 00660000. The
latter always gets the coveted base address because it always loads
first (last in link order? luck?). In the parent, locale.nls maps to
002B0000, while cygfoo.dll loads at 00320000. The child, on the other
hand, puts cygfoo.dll at 002B0000, and maps locale.nls to 00380000. That
latter address is reserved for some sort of anonymous memory in the
parent (most likely a thread stack,*** since there are some guarded
pages at 003B9000 and it's not a heap). My best guess is that this stack
lives at 00410000 in the child, which is free space in the parent.
The other main difference is the lack of shared memory in the child and
completely different heaps. I assume the former is because cygwin hasn't
finished initializing yet, and will not be a problem. I don't know
whether the latter matters wrt fork() semantics, but it certainly is
another potential source of address space clobbers. It's also rather odd
that the child has 3-4 heaps when its parent has only 1-2.
I actually hadn't noticed that thread stack before making the diff, so
at this point it's a toss-up which of the thread or locale.nls got there
first and messed up the other. Does Windows tend to map files at the
same address every time, or just executable images it hopes to share
as-is between processes? Without knowing that it's hard to say what
happened.
Regardless of file mapping behavior, though, I don't see right off how
to make this problem go away. Nothing stops thread stacks or heaps from
causing problems with other dlls, and they seem to move around even when
they could have stayed put. If we do it early enough, we might prevent
some clobbers by reserving address space we know dlls will eventually
need, but it's going to be hard to get in before statically-linked dlls,
let alone that first thread stack...
Thoughts?
Ryan
**If I try to print the process map too early it blows up because not
all the needed dlls have loaded yet...
*** I can't figure out how to identify a process's thread stacks.
SysInternals' newly-released VMMap utility can identify them, but is
unfortunately closed-source, stripped, and undocumented. I suspect it
exploits non-public/non-stable (fields of) structures such as PEB and
TEB, which I'd prefer to avoid pulling into cygwin even if I knew how
to. Also unfortunately, VMMap locks up trying to examine the child
process, most likely because the problem arises while we hold the
dll/loader lock.
> $ diff parent-maps.txt child-maps.txt
> 3d2
> < 00030000-00031000 rw-p 00000000 0000:0000 0 [shared]
> 13,22c12,20
> < 002B0000-00317000 r--p 00000000 2C36:17C8 281474976927380
> /cygdrive/c/Windows/System32/locale.nls
> < 00320000-00321000 r--p 00000000 2C36:17C8 2533274791177101
> /cygdrive/c/cygwin/home/Ryan/experiments/fork-tests/cygfoo.dll
> < 00321000-00322000 r-xp 00001000 2C36:17C8 2533274791177101
> /cygdrive/c/cygwin/home/Ryan/experiments/fork-tests/cygfoo.dll
> < 00322000-00325000 rw-p 00002000 2C36:17C8 2533274791177101
> /cygdrive/c/cygwin/home/Ryan/experiments/fork-tests/cygfoo.dll
> < 00325000-00326000 r--p 00005000 2C36:17C8 2533274791177101
> /cygdrive/c/cygwin/home/Ryan/experiments/fork-tests/cygfoo.dll
> < 00326000-00327000 rw-p 00006000 2C36:17C8 2533274791177101
> /cygdrive/c/cygwin/home/Ryan/experiments/fork-tests/cygfoo.dll
> < 00327000-00332000 r--p 00007000 2C36:17C8 2533274791177101
> /cygdrive/c/cygwin/home/Ryan/experiments/fork-tests/cygfoo.dll
> < 00380000-003B9000 ---p 00000000 0000:0000 0
> < 003B9000-003BC000 rw-s 00039000 0000:0000 0
> < 003BC000-003C0000 rw-p 0003C000 0000:0000 0
> ---
> > 002B0000-002B1000 r--p 00000000 2C36:17C8 2533274791177101
> /cygdrive/c/cygwin/home/Ryan/experiments/fork-tests/cygfoo.dll
> > 002B1000-002B2000 r-xp 00001000 2C36:17C8 2533274791177101
> /cygdrive/c/cygwin/home/Ryan/experiments/fork-tests/cygfoo.dll
> > 002B2000-002B5000 rw-p 00002000 2C36:17C8 2533274791177101
> /cygdrive/c/cygwin/home/Ryan/experiments/fork-tests/cygfoo.dll
> > 002B5000-002B6000 r--p 00005000 2C36:17C8 2533274791177101
> /cygdrive/c/cygwin/home/Ryan/experiments/fork-tests/cygfoo.dll
> > 002B6000-002B7000 rw-p 00006000 2C36:17C8 2533274791177101
> /cygdrive/c/cygwin/home/Ryan/experiments/fork-tests/cygfoo.dll
> > 002B7000-002C2000 r--p 00007000 2C36:17C8 2533274791177101
> /cygdrive/c/cygwin/home/Ryan/experiments/fork-tests/cygfoo.dll
> > 00300000-00306000 rw-p 00000000 0000:0000 0
> > 00306000-00380000 ---p 00006000 0000:0000 0
> > 00380000-003E7000 r--p 00000000 2C36:17C8 281474976927380
> /cygdrive/c/Windows/System32/locale.nls
> 27,37c25,40
> < 004F0000-004F3000 rw-p 00000000 0000:0000 0 [heap]
> < 004F3000-00500000 ---p 00003000 0000:0000 0 [heap]
> < 005C0000-005C6000 rw-p 00000000 0000:0000 0
> < 005C6000-00640000 ---p 00006000 0000:0000 0
> < 007F0000-0083A000 rw-p 00000000 0000:0000 0 [heap]
> < 0083A000-008F0000 ---p 0004A000 0000:0000 0 [heap]
> < 00AF0000-00EF0000 ---p 00000000 0000:0000 0
> < 00EF0000-00F60000 rw-p 00400000 0000:0000 0
> < 00F60000-18EF0000 ---p 00470000 0000:0000 0
> < 190EB000-190EC000 rw-s 001FB000 0000:0000 0
> < 190EC000-190F0000 rw-p 001FC000 0000:0000 0
> ---
> > 00410000-00449000 ---p 00000000 0000:0000 0
> > 00449000-0044C000 rw-s 00039000 0000:0000 0
> > 0044C000-00450000 rw-p 0003C000 0000:0000 0
> > 00460000-00499000 rw-p 00000000 0000:0000 0 [heap]
> > 00499000-00560000 ---p 00039000 0000:0000 0 [heap]
> > 00599000-0059C000 rw-s 00039000 0000:0000 0 [heap]
> > 0059C000-005A0000 rw-p 0003C000 0000:0000 0 [heap]
> > 006B0000-006B3000 rw-p 00000000 0000:0000 0 [heap]
> > 006B3000-006C0000 ---p 00003000 0000:0000 0 [heap]
> > 008BA000-008BC000 rw-s 001FA000 0000:0000 0 [heap]
> > 008BC000-008C0000 rw-p 001FC000 0000:0000 0 [heap]
> > 008C0000-00ABD000 ---p 00000000 0000:0000 0
> > 00ABD000-00ABF000 rw-s 001FD000 0000:0000 0
> > 00ABF000-00AC0000 rw-p 001FF000 0000:0000 0
> > 00EF0000-00FC0000 rw-p 00000000 0000:0000 0
> > 00FC0000-18EF0000 ---p 000D0000 0000:0000 0
> 163a167
> > 7EFD8000-7EFDB000 rw-p 00000000 0000:0000 0
More information about the Cygwin-developers
mailing list