atexit, __cxa_atexit and __cxa_finalize interaction

[Adhemerval Zanella <adhemerval dot zanella at linaro dot org>]

Recently on libc-help [1], a user reported it observed some issues regarding
atexit order calling in some scenarios (callbacks were not issued in the 
expected order for some scenarios). After some time, he could provide an 
actual testcase that shows the issue he was referring [2].

After some debugging, it turned out to be how atexit, __cxa_atexit and
__cxa_finalize are defined [3]. In a short:

  - __cxa_atexit register a function to be called by exit or when a shared 
    library is unloaded. 

  - __cxa_finalize is called on library unload (either when a program is exiting
    or by a dlopen)

  - atexit is implemented by calling __cxa_atexit through a static object

It then makes atexit acts exactly as __cxa_atexit, which makes unload shared 
libraries to call atexit handlers are well to __cxa_atexit handlers.

As on my later findings [4], it seems to come from LSB definition (although
I am not sure which one comes first, the standard or the implementation). 

In any case, I do think that making atexit works as __cxa_atexit is problematic
for glibc, since it does try to actually unload the shared libraries on dlclose.

POSIX [5] standard does not really specify the dlclose interaction with atexit 
handlers, so I think we might assume it is an implementation detail. And by 
making atexit handlers being called on __cxa_finalize it makes then run in 
non-expected order when they are registered in shared libraries constructors.

So the question is anything use preventing to change __cxa_finalize to no handle 
atexit handlers, only __cxa_atexit, and quick_exit ones)?

If we could, my idea would be to:

  1. Either add an atexit symbol or an __atexit (since currently atexit
     is build statically) which add a ef_at exit_function entry on __exit_funcs.
     Old binaries would still call __cxa_atexit, so we do not actually
     need to add a compat symbol.

  2. Make __cxa_finalize set ef_at handlers to ef_free to avoid them being
     called later (similar to quick_exit).

The only issue I can think of is c++ compilers that might rely on atexit
to works as __cxa_atexit, however __cxa_atexit is provided by glibc
since forever so I don't think it should be a problem.

[1] https://sourceware.org/ml/libc-help/2019-05/msg00021.html
[2] https://github.com/mulle-nat/ld-so-breakage
[3] https://sourceware.org/ml/libc-help/2019-06/msg00008.html
[4] https://sourceware.org/ml/libc-help/2019-06/msg00009.html
[5] https://pubs.opengroup.org/onlinepubs/9699919799/functions/atexit.html