RE: CVS setup.exe crashes on Windows 2003 Server x64

["Thrall, Bryan" <bryan dot thrall at flightsafety dot com>]

Dave Korn wrote on Monday, December 18, 2006 9:04 AM:
> On 18 December 2006 14:45, Thrall, Bryan wrote:
> 
>> Dave Korn wrote on Friday, December 15, 2006 7:13 PM:
>>> On 15 December 2006 21:08, Thrall, Bryan wrote:
>>> 
>>>> I'm seeing setup.exe built from CVS head crash on Windows 2003
>>>> Server x64 (Standard version, SP 1).
>>> 
>>>> Program received signal SIGSEGV, Segmentation fault.
>>>> 0x004eb157 in AllocateAndInitializeSid@44 ()
>>>> (gdb) bt
>>>> #0  0x004eb157 in AllocateAndInitializeSid@44 ()
>>>> #1  0x0043d8f8 in NTSecurity::initialiseEveryOneSID (this=0x23f780)
>>>> at main.cc:269 #2  0x0043daca in NTSecurity::setDefaultDACL
>>>> (this=0x23f780) at main.cc:287 #3  0x0043e1ec in
>>>> NTSecurity::setDefaultSecurity (this=0x23f780) at main.cc:340 #4
>>>> 0x0043ed33 in set_default_sec () at main.cc:237 #5  0x0043f3a2 in
>>>>     WinMain (h=0x400000, hPrevInstance=0x0, command_line=0xe0245d
>>>> "", cmd_show=10) at main.cc:482 #6  0x00499238 in main (argc=1,
>>>>     argv=0x346b8, __p__environ=0x33090) at ../../runtime/main.c:73
>>>> 
>>>> After a little debugging, it looks like the this pointer is getting
>>>> corrupted between frames 1 and 0 (it is valid in frame 1 and
>>>> invalid -- 0x105 IIRC -- in frame 0).
> 
>> Sorry, I meant to say that the this pointer is fine in
>> NTSecurity::setDefaultDACL (frame 2), but bad in
>> NTSecurity::initialiseEveryOneSID (frame 1).
> 
>   Oh well, that blows out my theory you might have been getting
> linked against the wrong version of the libs (64-vs-32 bit) and hence
> get disagreements about sizes of various integer types being passed
> on the stack as arguments. 
> 
>   Wait a minute!  That stack trace shows 'this as having the same
> value in frames 1, 2 and 3, it's 0x23f780 in all of them.  That
> suggests that gdb is parsing the stack right and the code is getting
> it wrong.  I think you still need to worry about sizes of stack
> arguments.  Maybe there's a header file somewhere that uses an int
> where it should have written long int, and nobody's noticed until
> LP64 made them become different sizes? 

Er... why yes it certainly does! Don't know how I missed that...

Ok, I checked just to be sure, and if I set a breakpoint on main:269,
then the this pointer is invalid (0x105) in
NTSecurity::initialiseEveryOneSID; after the segfault, the this pointer
is valid again (0x23f780). Weird.

Also, I rebooted the machine this morning, and got a bunch of message
popups dated from my last test session regarding DEP. This seems like a
big clue that Windows DEP is trying to close setup.exe, which is causing
this crash. It seems extremely odd that the "DEP is closing your
process" messages wouldn't pop up until you reboot, though.

... And in fact, disabling DEP for setup.exe fixes the problem. So, it
looks like setup.exe is trying to execute some memory that Windows
thinks it shouldn't be (the invalid this pointer, probably, but why then
is it invalid at that point and valid after the segfault?).

>>>   Don't have 64bits or 2k3 but I'll see how win2k likes it.
> 
>   Haven't tested this yet owing to wanting to keep my installation
> stable while doing gcc testsuite runs.  Should be able to give it a
> go tonight or tomorrow.
> 
> 
>     cheers,
>       DaveK

-- 
Bryan Thrall
FlightSafety International
Bryan.Thrall@flightsafety.com