This is the mail archive of the cygwin-developers mailing list for the Cygwin project.


Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]
Other format: [Raw text]

Address space clobbers during fork() (was Re: Extending /proc/*/maps)


On 19/04/2011 12:01 PM, Corinna Vinschen wrote:
On Apr 19 11:38, Ryan Johnson wrote:
BTW, /cygdrive/c/Windows/System32/locale.nls seems to be the reason
that even statically linked dlls don't always load in the same place
twice in a row.
How so?  And then, when is it loaded?  I assume that this file is
fetched as soon as the GetLocaleInfo function is called.  What OS
are you using?  If it's Vista or later, there's a chance that the
LocaleNameToLCID function is the culprit as well.  But the real
problem is, how can that be worked around?  For the locale stuff
we need these functions.
(subject changed to reflect what we're actually discussing)

I'm using Win7-x64.

When locale.nls gets mapped in, Windows doesn't always put it in the same place. As a result, fork() can run into all the usual problems with DLL base addresses, except it can bite even statically-linked dlls.

Below is an example of how the memory layouts can differ right from the start. The diff comes from memory maps taken just before the parent forked and just after the child detected a problem (in dll_list::alloc, called by dll_dllcrt0_1).**

Recall that cygfoo.dll and cygbar.dll both want to load at 00660000. The latter always gets the coveted base address because it always loads first (last in link order? luck?). In the parent, locale.nls maps to 002B0000, while cygfoo.dll loads at 00320000. The child, on the other hand, puts cygfoo.dll at 002B0000, and maps locale.nls to 00380000. That latter address is reserved for some sort of anonymous memory in the parent (most likely a thread stack,*** since there are some guarded pages at 003B9000 and it's not a heap). My best guess is that this stack lives at 00410000 in the child, which is free space in the parent.

The other main difference is the lack of shared memory in the child and completely different heaps. I assume the former is because cygwin hasn't finished initializing yet, and will not be a problem. I don't know whether the latter matters wrt fork() semantics, but it certainly is another potential source of address space clobbers. It's also rather odd that the child has 3-4 heaps when its parent has only 1-2.

I actually hadn't noticed that thread stack before making the diff, so at this point it's a toss-up which of the thread or locale.nls got there first and messed up the other. Does Windows tend to map files at the same address every time, or just executable images it hopes to share as-is between processes? Without knowing that it's hard to say what happened.

Regardless of file mapping behavior, though, I don't see right off how to make this problem go away. Nothing stops thread stacks or heaps from causing problems with other dlls, and they seem to move around even when they could have stayed put. If we do it early enough, we might prevent some clobbers by reserving address space we know dlls will eventually need, but it's going to be hard to get in before statically-linked dlls, let alone that first thread stack...

Thoughts?
Ryan

**If I try to print the process map too early it blows up because not all the needed dlls have loaded yet...

*** I can't figure out how to identify a process's thread stacks. SysInternals' newly-released VMMap utility can identify them, but is unfortunately closed-source, stripped, and undocumented. I suspect it exploits non-public/non-stable (fields of) structures such as PEB and TEB, which I'd prefer to avoid pulling into cygwin even if I knew how to. Also unfortunately, VMMap locks up trying to examine the child process, most likely because the problem arises while we hold the dll/loader lock.

$ diff parent-maps.txt child-maps.txt
3d2
< 00030000-00031000 rw-p 00000000 0000:0000 0 [shared]
13,22c12,20
< 002B0000-00317000 r--p 00000000 2C36:17C8 281474976927380 /cygdrive/c/Windows/System32/locale.nls
< 00320000-00321000 r--p 00000000 2C36:17C8 2533274791177101 /cygdrive/c/cygwin/home/Ryan/experiments/fork-tests/cygfoo.dll
< 00321000-00322000 r-xp 00001000 2C36:17C8 2533274791177101 /cygdrive/c/cygwin/home/Ryan/experiments/fork-tests/cygfoo.dll
< 00322000-00325000 rw-p 00002000 2C36:17C8 2533274791177101 /cygdrive/c/cygwin/home/Ryan/experiments/fork-tests/cygfoo.dll
< 00325000-00326000 r--p 00005000 2C36:17C8 2533274791177101 /cygdrive/c/cygwin/home/Ryan/experiments/fork-tests/cygfoo.dll
< 00326000-00327000 rw-p 00006000 2C36:17C8 2533274791177101 /cygdrive/c/cygwin/home/Ryan/experiments/fork-tests/cygfoo.dll
< 00327000-00332000 r--p 00007000 2C36:17C8 2533274791177101 /cygdrive/c/cygwin/home/Ryan/experiments/fork-tests/cygfoo.dll
< 00380000-003B9000 ---p 00000000 0000:0000 0
< 003B9000-003BC000 rw-s 00039000 0000:0000 0
< 003BC000-003C0000 rw-p 0003C000 0000:0000 0
---
> 002B0000-002B1000 r--p 00000000 2C36:17C8 2533274791177101 /cygdrive/c/cygwin/home/Ryan/experiments/fork-tests/cygfoo.dll
> 002B1000-002B2000 r-xp 00001000 2C36:17C8 2533274791177101 /cygdrive/c/cygwin/home/Ryan/experiments/fork-tests/cygfoo.dll
> 002B2000-002B5000 rw-p 00002000 2C36:17C8 2533274791177101 /cygdrive/c/cygwin/home/Ryan/experiments/fork-tests/cygfoo.dll
> 002B5000-002B6000 r--p 00005000 2C36:17C8 2533274791177101 /cygdrive/c/cygwin/home/Ryan/experiments/fork-tests/cygfoo.dll
> 002B6000-002B7000 rw-p 00006000 2C36:17C8 2533274791177101 /cygdrive/c/cygwin/home/Ryan/experiments/fork-tests/cygfoo.dll
> 002B7000-002C2000 r--p 00007000 2C36:17C8 2533274791177101 /cygdrive/c/cygwin/home/Ryan/experiments/fork-tests/cygfoo.dll
> 00300000-00306000 rw-p 00000000 0000:0000 0
> 00306000-00380000 ---p 00006000 0000:0000 0
> 00380000-003E7000 r--p 00000000 2C36:17C8 281474976927380 /cygdrive/c/Windows/System32/locale.nls
27,37c25,40
< 004F0000-004F3000 rw-p 00000000 0000:0000 0 [heap]
< 004F3000-00500000 ---p 00003000 0000:0000 0 [heap]
< 005C0000-005C6000 rw-p 00000000 0000:0000 0
< 005C6000-00640000 ---p 00006000 0000:0000 0
< 007F0000-0083A000 rw-p 00000000 0000:0000 0 [heap]
< 0083A000-008F0000 ---p 0004A000 0000:0000 0 [heap]
< 00AF0000-00EF0000 ---p 00000000 0000:0000 0
< 00EF0000-00F60000 rw-p 00400000 0000:0000 0
< 00F60000-18EF0000 ---p 00470000 0000:0000 0
< 190EB000-190EC000 rw-s 001FB000 0000:0000 0
< 190EC000-190F0000 rw-p 001FC000 0000:0000 0
---
> 00410000-00449000 ---p 00000000 0000:0000 0
> 00449000-0044C000 rw-s 00039000 0000:0000 0
> 0044C000-00450000 rw-p 0003C000 0000:0000 0
> 00460000-00499000 rw-p 00000000 0000:0000 0 [heap]
> 00499000-00560000 ---p 00039000 0000:0000 0 [heap]
> 00599000-0059C000 rw-s 00039000 0000:0000 0 [heap]
> 0059C000-005A0000 rw-p 0003C000 0000:0000 0 [heap]
> 006B0000-006B3000 rw-p 00000000 0000:0000 0 [heap]
> 006B3000-006C0000 ---p 00003000 0000:0000 0 [heap]
> 008BA000-008BC000 rw-s 001FA000 0000:0000 0 [heap]
> 008BC000-008C0000 rw-p 001FC000 0000:0000 0 [heap]
> 008C0000-00ABD000 ---p 00000000 0000:0000 0
> 00ABD000-00ABF000 rw-s 001FD000 0000:0000 0
> 00ABF000-00AC0000 rw-p 001FF000 0000:0000 0
> 00EF0000-00FC0000 rw-p 00000000 0000:0000 0
> 00FC0000-18EF0000 ---p 000D0000 0000:0000 0
163a167
> 7EFD8000-7EFDB000 rw-p 00000000 0000:0000 0



Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]