Re: atexit, __cxa_atexit and __cxa_finalize interaction

[Carlos O'Donell <carlos at redhat dot com>]

On 6/11/19 3:06 PM, Adhemerval Zanella wrote:
> Recently on libc-help [1], a user reported it observed some issues regarding
> atexit order calling in some scenarios (callbacks were not issued in the 
> expected order for some scenarios). After some time, he could provide an 
> actual testcase that shows the issue he was referring [2].

This issue has been around since the dawn of time.

https://freebsd-arch.freebsd.narkive.com/zzyHX3Bw/dlclose-vs-atexit

The options as outlined in the old FreeBSD discussion are informative.

(1) Run atexit() at exit() after dlclose() and crash.
(2) Run atexit() at dlclose() and try not to crash (impossible for the cyclic
    case where two libraries call eachothers atexit functions i.e. no total
    order possible, and we could detect this).
(3) Have atexit() increment the library ref count, and decrement it after
    it runs during exit(), making the library unloadable (what SunOS does?)

My opinion is this:

(a) glibc promises to try and unload your DSO on dlclose().

(b) glibc has always called atexit() on dlclose(), because the intent of
    atexit() for a DSO was to handle the case where it's going away and
    you want to cleanup state. It minimizes the memory needed to store the
    list of these calls, and fails early, both are good.

(c) LSB is a trailing standard that needs to document what GNU/Linux
    defines as the standard.

I don't think there is anything we need to do to document this better in
our manual.

The linux man pages already say:

  Linux notes
       Since  glibc 2.2.3, atexit() (and on_exit(3)) can be used within a shared library to establish functions that
       are called when the shared library is unloaded.

This is a perfectly acceptable use case for dlclose () IMO.

> After some debugging, it turned out to be how atexit, __cxa_atexit and
> __cxa_finalize are defined [3]. In a short:
> 
>   - __cxa_atexit register a function to be called by exit or when a shared 
>     library is unloaded. 
> 
>   - __cxa_finalize is called on library unload (either when a program is exiting
>     or by a dlopen)
> 
>   - atexit is implemented by calling __cxa_atexit through a static object
> 
> It then makes atexit acts exactly as __cxa_atexit, which makes unload shared 
> libraries to call atexit handlers are well to __cxa_atexit handlers.
> 
> As on my later findings [4], it seems to come from LSB definition (although
> I am not sure which one comes first, the standard or the implementation). 
> 
> In any case, I do think that making atexit works as __cxa_atexit is problematic
> for glibc, since it does try to actually unload the shared libraries on dlclose.
> 
> POSIX [5] standard does not really specify the dlclose interaction with atexit 
> handlers, so I think we might assume it is an implementation detail. And by 
> making atexit handlers being called on __cxa_finalize it makes then run in 
> non-expected order when they are registered in shared libraries constructors.
> 
> So the question is anything use preventing to change __cxa_finalize to no handle 
> atexit handlers, only __cxa_atexit, and quick_exit ones)?
> 
> If we could, my idea would be to:
> 
>   1. Either add an atexit symbol or an __atexit (since currently atexit
>      is build statically) which add a ef_at exit_function entry on __exit_funcs.
>      Old binaries would still call __cxa_atexit, so we do not actually
>      need to add a compat symbol.
> 
>   2. Make __cxa_finalize set ef_at handlers to ef_free to avoid them being
>      called later (similar to quick_exit).

What is the use case for making the chnage?

New developers do stumble over this issue, but it's a complex issue and we
could do better to document it, like including Ben's patch on dynamic loading:

https://www.sourceware.org/ml/libc-alpha/2017-10/msg00582.html

So we have good docs to read.

> The only issue I can think of is c++ compilers that might rely on atexit
> to works as __cxa_atexit, however __cxa_atexit is provided by glibc
> since forever so I don't think it should be a problem.
> 
> [1] https://sourceware.org/ml/libc-help/2019-05/msg00021.html
> [2] https://github.com/mulle-nat/ld-so-breakage
> [3] https://sourceware.org/ml/libc-help/2019-06/msg00008.html
> [4] https://sourceware.org/ml/libc-help/2019-06/msg00009.html
> [5] https://pubs.opengroup.org/onlinepubs/9699919799/functions/atexit.html
> 


-- 
Cheers,
Carlos.