[PATCH RFC] fork: reduce chances for "address space is already occupied" errors

[Michael Haubenwallner]

On 3/28/19 10:15 AM, Corinna Vinschen wrote:
> On Mar 28 09:34, Michael Haubenwallner wrote:
>> Hi Corinna,
>>
>> On 3/27/19 10:16 AM, Corinna Vinschen wrote:
>>> On Mar 27 09:26, Michael Haubenwallner wrote:
>>>> On 3/26/19 7:28 PM, Corinna Vinschen wrote:
>>> Wait, let me understand what's going on.  IIUC you're building DLLs
>>> which are then used during the build job itself, right?
>>
>> Exactly.
>> FWIW, the CI builds also set up a Cygwin instance from scratch,
>> as I'm also after testing Cygwin (v3) itself to some degree:
>> https://dev.azure.com/gentoo-prefix/ci-builds/_build
>>
>> However, I've not found a commandline option for setup.exe to install
>> "test" versions...
>>
>>> As you know, 64 bit has a defined memory layout.  Binutils ld is
>>> supposed to base the DLLs to a pseudo-random address in the area between
>>> 0x4:00000000 and 0x6:00000000.  This area is occupied by un-rebased DLLs
>>> only.  8 Gigs is a *lot* of space for DLLs.
>>>
>>> That also means that the DLLs should not at all collide with windows
>>> objects (typically reserved in the lesser 2 Gigs area), unless they
>>> collide with themselves.  At least that's the idea.
>>>
>>> Can you check what addresses the freshly built DLLs are based on by LD?
>>> Is there a chance that the algorithm used in LD is too dumb?
>>
>> I've also added system_printf to dll_list::reserve_space() when a dynloaded
>> dll was relocated, and each new address was below 0x0:01000000. The attached
>> output also contains the preferred address, above 0x4:00000000 each.
> 
> Do they actually collide with each other?  Did you check the addresses?

Yes, there is a real collision between installed dlls:
$ rebase -i /home/haubi/test-20190327/gentoo-prefix/usr/bin/cygcrypto-1.1.dll /home/haubi/test-20190327/gentoo-prefix/usr/lib/python2.7/lib-dynload/_locale.dll
/home/haubi/test-20190327/gentoo-prefix/usr/bin/cygcrypto-1.1.dll                 base 0x00041c650000 size 0x0027d000 *
/home/haubi/test-20190327/gentoo-prefix/usr/lib/python2.7/lib-dynload/_locale.dll base 0x00041c6a0000 size 0x0002c000 *

> 
> There must be collisions in your case.  Can you please check if
> Achim's solution works for you?

The flexibility of rebase regarding multiple rebase databases seems not there yet,
but in theory this can help to avoid conflicts between dlls finally *installed*.

It will not help for conflicts between dlls within a single package while this
package is built.  I'm thinking of python modules built within the python package
itself, where the just built modules are used within the very build process.  Not
sure if packages using local modules during build also do use fork then, though.

> 
> In the meantime I pushed your patch to the master branch (but not
> yet to the 3.0 branch).

Is the cygwin1.dll from master branch available via setup.exe cmdline somehow?

Thanks!
/haubi/