[RFC] stdlib: Make atexit to not act as __cxa_atexit

[Adhemerval Zanella]

On 02/07/2019 10:24, Florian Weimer wrote:
> * Adhemerval Zanella:
> 
>> On 02/07/2019 04:44, Florian Weimer wrote:
>>> * Adhemerval Zanella:
>>>
>>>>   3. exit is the only way where atexit handlers are actually called.
>>>>      Unloading a library through dlclose does not issue the atexit
>>>>      anymore (this is the main semantic change).
>>>
>>> Sorry, I don't think this is the right direction.  We got the current
>>> atexit behavior from Solaris, and I think it is useful.
>>>
>>>   <https://docs.oracle.com/cd/E88353_01/html/E37843/atexit-3c.html>
>>
>> I am not sure this is also a good design, specially that we have elf
>> destructors that provide a similar semantic.
> 
> It's what's in various quasi-standard documents, so we need to support
> it.
> 
>>> I think the bug is that our implementation of that behavior interferes
>>> with the execution order when there is no dlclose at all.  There should
>>> be a different way for fixing that.
>>>
>>> Maybe we can turn __cxa_finalize into a NOP and have the dynamic loader
>>> handle atexit handlers directly on dlclose (assuming that it's always
>>> called before the other ELF destructors)?
>>
>> We need __cxa_finalize because it is called by __do_global_dtors_aux
>> from initfini callbacks.
> 
> (Note: The comment is about crtbeginS.o built from libgcc sources.)
> 
> If all versions of the ELF constructor in crtbeginS.o call
> __cxa_finalize as their first action, then we can turn __cxa_finalize
> into a NOP and have the dynamic linker perform the original actions of
> __cxa_finalize directly, before running the ELF destructors for this
> object.

In fact I though about it and one issue I found is __cxa_finalize uses
__dso_handle and it is a hidden symbol for each DSO.  So, at _dl_fini
we will need to resolve __cxa_finalize from libc.so (or potentially
other function that handles the exit handlers), then resolve the
__dso_handle from the referenced object (with potentially a hack to
lookup for local objects), and finally call __cxa_finalize with
__dso_handler as argument. Not sure it is really a better design.


> 
>> Also making the loader itself handler atexit
>> is not the best option IMHO: we will either to call a private symbol on 
>> libc (which might require an additional lookup) or export the internal 
>> data so loader can implement the callback issuing itself.
> 
> Anything would be better than this pointless indirection through libgcc.
> We have accumlated lots of cruft in the crt* objects which serve no
> purpose at all for current binaries, and eventually, we will have to
> clean this up.
> 
> I think crtbegin*.o could completely go away if we turn __cxa_finalize
> into a NOP on the glibc side.

I think if __dso_handle was exported as a global symbol it could
simplify the indirection by a bit.

> 
>>> Or otherwise use flags to mark things, but in such a way that the
>>> handlers are processed in reverse atexit order, unless a dlclose
>>> happens.
>>
>> I think one possible option to still allow atexit handler being called
>> by atexit is to still register them as ef_at (using the new __atexit
>> symbol) and on __run_exit_handler first process et_at and them
>> ef_on and ef_cxa. This is still a semantic change of atexit regarding
>> the order of on_exit and __cxa_atexit, but we keep the Solaris like
>> one to call them on dlclose.
> 
> Sorry, can you rephrase?  There seem to be a few typos in this
> paragraph.
> 

One idea is to change the order atexit handlers are handled slightly:

  - atexit handlers are not registered through __cxa_atexit, a new
    symbol __atexit adds a new entry as ef_at (similar to my proposal).

  - __cxa_finalize is changed to handle ef_at similar to ef_cxa by
    calling the function pointer and setting it as ef_free. dlclose
    will issue atexit as expected.

  - __run_exit_handlers, used on both exit and quick_exit, will issue
    atexit handler (ef_at) first and then handle on_exit (ef_on), and
    __cxa_atexit (ef_cxa).  This keeps the expected order or atexit
    handlers, however the order regarding ef_cxa will change.

  - It will required some changes internally to handle atexit and
    __cxa_atexit differently regarding lock acquisition since before
    ef_cxa handler we can't get newly added ef_at ones.