Address space clobbers during fork() (was Re: Extending /proc/*/maps)

[Ryan Johnson]

On 21/04/2011 3:17 PM, Ryan Johnson wrote:
> I'm starting to notice a disturbing pattern. When I use windbg to run 
> my toy program, it never fails (at least, never yet). When I run it 
> within gdb, it invariably fails. If I run from the command line it 
> fails 10-15% of the time. Could there be some sort of timing issue 
> involved here? The process seems thoroughly single-threaded at this 
> point in its life, but yet the pattern is there. Does windbg somehow 
> change dll load order, or something?
>
> Further, when the fork fails, it seems due to the *parent* having a 
> messed-up address space: locale.nls prefers to load at 002B0000, so a 
> misbehaving parent pretty much guarantees a base address mismatch... 
> unless the child also happens to misbehave.
*sigh*

I analyzed the output of 100 runs of my fork test and:
16 parents had cygfoo.dll at 002B0000
22 parents had locale.nls not at 002B0000
11 forks failed
3 forks failed with cygfoo.dll not at 002B0000
1 fork (at least) failed with locale.nls at 002B0000 (!)

For the cases where cygfoo.dll and locale.nls get along and yet the fork 
fails, a mystery memory region, 512kB in size (with 24kB of that 
mapped), gets in the way or one or the other. Neither vmmap nor windbg 
knows what it is.

The implication seems to be that neither rebaseall nor messing with 
locale.nls is going to fix this completely, though both would reduce the 
probability of failure somewhat.

Next up... to see how often fork fails when cygfoo.dll and cygbar.dll 
don't have base address conflicts.

Ryan