Address space clobbers during fork() (was Re: Extending /proc/*/maps)

Ryan Johnson ryan.johnson@cs.utoronto.ca
Tue Apr 19 18:17:00 GMT 2011


On 19/04/2011 12:01 PM, Corinna Vinschen wrote:
> On Apr 19 11:38, Ryan Johnson wrote:
>> BTW, /cygdrive/c/Windows/System32/locale.nls seems to be the reason
>> that even statically linked dlls don't always load in the same place
>> twice in a row.
> How so?  And then, when is it loaded?  I assume that this file is
> fetched as soon as the GetLocaleInfo function is called.  What OS
> are you using?  If it's Vista or later, there's a chance that the
> LocaleNameToLCID function is the culprit as well.  But the real
> problem is, how can that be worked around?  For the locale stuff
> we need these functions.
(subject changed to reflect what we're actually discussing)

I'm using Win7-x64.

When locale.nls gets mapped in, Windows doesn't always put it in the 
same place. As a result, fork() can run into all the usual problems with 
DLL base addresses, except it can bite even statically-linked dlls.

Below is an example of how the memory layouts can differ right from the 
start. The diff comes from memory maps taken just before the parent 
forked and just after the child detected a problem (in dll_list::alloc, 
called by dll_dllcrt0_1).**

Recall that cygfoo.dll and cygbar.dll both want to load at 00660000. The 
latter always gets the coveted base address because it always loads 
first (last in link order? luck?). In the parent, locale.nls maps to 
002B0000, while cygfoo.dll loads at 00320000. The child, on the other 
hand, puts cygfoo.dll at 002B0000, and maps locale.nls to 00380000. That 
latter address is reserved for some sort of anonymous memory in the 
parent (most likely a thread stack,*** since there are some guarded 
pages at 003B9000 and it's not a heap). My best guess is that this stack 
lives at 00410000 in the child, which is free space in the parent.

The other main difference is the lack of shared memory in the child and 
completely different heaps. I assume the former is because cygwin hasn't 
finished initializing yet, and will not be a problem. I don't know 
whether the latter matters wrt fork() semantics, but it certainly is 
another potential source of address space clobbers. It's also rather odd 
that the child has 3-4 heaps when its parent has only 1-2.

I actually hadn't noticed that thread stack before making the diff, so 
at this point it's a toss-up which of the thread or locale.nls got there 
first and messed up the other. Does Windows tend to map files at the 
same address every time, or just executable images it hopes to share 
as-is between processes? Without knowing that it's hard to say what 
happened.

Regardless of file mapping behavior, though, I don't see right off how 
to make this problem go away. Nothing stops thread stacks or heaps from 
causing problems with other dlls, and they seem to move around even when 
they could have stayed put. If we do it early enough, we might prevent 
some clobbers by reserving address space we know dlls will eventually 
need, but it's going to be hard to get in before statically-linked dlls, 
let alone that first thread stack...

Thoughts?
Ryan

**If I try to print the process map too early it blows up because not 
all the needed dlls have loaded yet...

*** I can't figure out how to identify a process's thread stacks. 
SysInternals' newly-released VMMap utility can identify them, but is 
unfortunately closed-source, stripped, and undocumented. I suspect it 
exploits non-public/non-stable (fields of) structures such as PEB and 
TEB, which I'd prefer to avoid pulling into cygwin even if I knew how 
to. Also unfortunately, VMMap locks up trying to examine the child 
process, most likely because the problem arises while we hold the 
dll/loader lock.

> $ diff parent-maps.txt child-maps.txt
> 3d2
> < 00030000-00031000 rw-p 00000000 0000:0000 0                   [shared]
> 13,22c12,20
> < 002B0000-00317000 r--p 00000000 2C36:17C8 281474976927380     
> /cygdrive/c/Windows/System32/locale.nls
> < 00320000-00321000 r--p 00000000 2C36:17C8 2533274791177101    
> /cygdrive/c/cygwin/home/Ryan/experiments/fork-tests/cygfoo.dll
> < 00321000-00322000 r-xp 00001000 2C36:17C8 2533274791177101    
> /cygdrive/c/cygwin/home/Ryan/experiments/fork-tests/cygfoo.dll
> < 00322000-00325000 rw-p 00002000 2C36:17C8 2533274791177101    
> /cygdrive/c/cygwin/home/Ryan/experiments/fork-tests/cygfoo.dll
> < 00325000-00326000 r--p 00005000 2C36:17C8 2533274791177101    
> /cygdrive/c/cygwin/home/Ryan/experiments/fork-tests/cygfoo.dll
> < 00326000-00327000 rw-p 00006000 2C36:17C8 2533274791177101    
> /cygdrive/c/cygwin/home/Ryan/experiments/fork-tests/cygfoo.dll
> < 00327000-00332000 r--p 00007000 2C36:17C8 2533274791177101    
> /cygdrive/c/cygwin/home/Ryan/experiments/fork-tests/cygfoo.dll
> < 00380000-003B9000 ---p 00000000 0000:0000 0
> < 003B9000-003BC000 rw-s 00039000 0000:0000 0
> < 003BC000-003C0000 rw-p 0003C000 0000:0000 0
> ---
> > 002B0000-002B1000 r--p 00000000 2C36:17C8 2533274791177101    
> /cygdrive/c/cygwin/home/Ryan/experiments/fork-tests/cygfoo.dll
> > 002B1000-002B2000 r-xp 00001000 2C36:17C8 2533274791177101    
> /cygdrive/c/cygwin/home/Ryan/experiments/fork-tests/cygfoo.dll
> > 002B2000-002B5000 rw-p 00002000 2C36:17C8 2533274791177101    
> /cygdrive/c/cygwin/home/Ryan/experiments/fork-tests/cygfoo.dll
> > 002B5000-002B6000 r--p 00005000 2C36:17C8 2533274791177101    
> /cygdrive/c/cygwin/home/Ryan/experiments/fork-tests/cygfoo.dll
> > 002B6000-002B7000 rw-p 00006000 2C36:17C8 2533274791177101    
> /cygdrive/c/cygwin/home/Ryan/experiments/fork-tests/cygfoo.dll
> > 002B7000-002C2000 r--p 00007000 2C36:17C8 2533274791177101    
> /cygdrive/c/cygwin/home/Ryan/experiments/fork-tests/cygfoo.dll
> > 00300000-00306000 rw-p 00000000 0000:0000 0
> > 00306000-00380000 ---p 00006000 0000:0000 0
> > 00380000-003E7000 r--p 00000000 2C36:17C8 281474976927380     
> /cygdrive/c/Windows/System32/locale.nls
> 27,37c25,40
> < 004F0000-004F3000 rw-p 00000000 0000:0000 0                   [heap]
> < 004F3000-00500000 ---p 00003000 0000:0000 0                   [heap]
> < 005C0000-005C6000 rw-p 00000000 0000:0000 0
> < 005C6000-00640000 ---p 00006000 0000:0000 0
> < 007F0000-0083A000 rw-p 00000000 0000:0000 0                   [heap]
> < 0083A000-008F0000 ---p 0004A000 0000:0000 0                   [heap]
> < 00AF0000-00EF0000 ---p 00000000 0000:0000 0
> < 00EF0000-00F60000 rw-p 00400000 0000:0000 0
> < 00F60000-18EF0000 ---p 00470000 0000:0000 0
> < 190EB000-190EC000 rw-s 001FB000 0000:0000 0
> < 190EC000-190F0000 rw-p 001FC000 0000:0000 0
> ---
> > 00410000-00449000 ---p 00000000 0000:0000 0
> > 00449000-0044C000 rw-s 00039000 0000:0000 0
> > 0044C000-00450000 rw-p 0003C000 0000:0000 0
> > 00460000-00499000 rw-p 00000000 0000:0000 0                   [heap]
> > 00499000-00560000 ---p 00039000 0000:0000 0                   [heap]
> > 00599000-0059C000 rw-s 00039000 0000:0000 0                   [heap]
> > 0059C000-005A0000 rw-p 0003C000 0000:0000 0                   [heap]
> > 006B0000-006B3000 rw-p 00000000 0000:0000 0                   [heap]
> > 006B3000-006C0000 ---p 00003000 0000:0000 0                   [heap]
> > 008BA000-008BC000 rw-s 001FA000 0000:0000 0                   [heap]
> > 008BC000-008C0000 rw-p 001FC000 0000:0000 0                   [heap]
> > 008C0000-00ABD000 ---p 00000000 0000:0000 0
> > 00ABD000-00ABF000 rw-s 001FD000 0000:0000 0
> > 00ABF000-00AC0000 rw-p 001FF000 0000:0000 0
> > 00EF0000-00FC0000 rw-p 00000000 0000:0000 0
> > 00FC0000-18EF0000 ---p 000D0000 0000:0000 0
> 163a167
> > 7EFD8000-7EFDB000 rw-p 00000000 0000:0000 0




More information about the Cygwin-developers mailing list