Problem with atexit and _dl_fini

[Adhemerval Zanella]

On 09/06/2019 17:59, Nat! wrote:
> Another datapoint to support my claim that _dl-fini breaks atexit. This time its very easy to reproduce ;)
> 
> Here 's the README.md from the Github Repo https://github.com/mulle-nat/atexit-breakage-linux
> 
> 
> ```
> 
> # Shows another breakage involving `atexit` on linux
> 
> Here the `atexit` callback is invoked mistakenly multiple times.

This 'example' does not really show the issue because ldd script issues
the loader multiple times, see below. You can check exactly what ldd is
doing by calling with sh -x. 


I will try to use your instruction to run on docker to see what exactly
is happening in your environment.


> 
> ## Build
> 
> Build with [mulle-make](//github.com/mulle-sde/mulle-make) or alternatively :
> 
> ```
> (
> Â Â  mkdir build &&
> Â Â  cd build &&
> Â Â  cmake .. &&
> Â Â  make
> )
> ```
> 
> ## Run
> 
> Use `ldd` to trigger the misbehaviour:
> 
> ```
> LD_PRELOAD="${PWD}/build/libld-preload.so" ldd ./build/main
> ```
> 
> ## Output
> 
> ```
> load
> unload

First and second time is done on:

158         dummy=`$rtld 2>&1`
159         if test $? = 127; then
160           verify_out=`${rtld} --verify "$file"`
161           ret=$?

Where the rtld list is, for x86_64, /lib/ld-linux.so.2 and /lib64/ld-linux-x86-64.so.2

> unload
> unload

This time is done at:

176     0|2)
177       try_trace "$RTLD" "$file" || result=1
178       ;;
179     *)

Where is call the loader as 

eval LD_TRACE_LOADED_OBJECTS=1 LD_WARN= LD_BIND_NOW= 'LD_LIBRARY_VERSION=$verify_out' LD_VERBOSE= '"$@"'

With same rtld list as before.

> Â Â  linux-vdso.so.1 (0x00007ffd2b2bd000)
> Â Â  /home/src/srcO/mulle-core/mulle-testallocator/research/ld-preload/build/libld-preload.so (0x00007f83853c1000)
> Â Â  libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f838518c000)
> Â Â  /lib64/ld-linux-x86-64.so.2 (0x00007f83853cd000)
> unload
> unload
> ```
> 
> 
> Ciao
> Â Â  Nat!
> 
> 
> On 19.05.19 21:37, Nat! wrote:
>>
>> On 19.05.19 18:23, Florian Weimer wrote:
>>> * Nat!:
>>>
>>>> So my problem is, that I observe that my atexit calls are not executed
>>>> in the correct order.
>>>>
>>>> i.e. atexit( a); atexit( b);Â  should result in b(), a() being called in
>>>> that order. To quote the man page,
>>>>
>>>> "the registered functions are invoked in reverse order".
>>>>
>>>>
>>>> When I register with `atexit` I can see my functions being added
>>>> properly within `__internal_atexit` in the
>>>>
>>>> correct order. Finally after my functions, the elf-loader ? also adds
>>>> itself there. So it is being called first by
>>>>
>>>> `__run_exit_handlers`.
>>>>
>>>>
>>>> Then comes the part where it goes wrong. I registered my two function
>>>> with `__internal_atexit`, but for some reason
>>>>
>>>> `_dl_fini` is calling `__cxa_finalize` and that is calling the wrong
>>>> function first.
>>> When atexit is called from a DSO, glibc calls the registered function
>>> before the DSO is unloaded.  This choice was made because after
>>> unloading, the function pointer becomes invalid.
>>>
>>> I haven't checked, but I suspect atexit still works this way even if
>>> it doesn't have to (because the DSO is never unloaded).
>>>
>>
>> I understand, but the behavior is wrong :) The C standard (or the C++ standard for this matter) http://www.cplusplus.com/reference/cstdlib/atexit/ states that
>>
>>
>> ```
>>
>> If more than one atexit function has been specified by different calls to this function, they are all executed in reverse order as a stack (i.e. the last function specified is the first to be executed at exit).
>>
>> ```
>>
>> I think its been shown that glibc can violate this C standard, so for me the argument would be over here already. That one should unwind in the reverse order is, I assume, not a interesting discussion topic. Currently atexit as a reliable mechanism is broken.
>>
>>
>> But I also don't think the way this is currently handled in glibc, can't be of much use to anyone.
>>
>> Case 1: a regular exe linked with shared libraries, nothing gets unloaded at exit, so what's the point ?
>>
>> Case 2: someone manually unloads a shared library, that contains atexit code. The bug is either using `atexit` for a shared library that gets unloaded, or unloading a shared library that contains atexit code. But it's not really glibcs business IMO.
>>
>> Case 3: some automatism unloads shared libraries. Then the automatism should check if atexit code is affected and not unload, because the shared library is still clearly needed. It's a bug in the automatism.
>>
>> If one was hellbent on trying to support atexit for unloading shared libraries, an atexit contained in a shared library should up the reference count of the shared library during the atexit call and decrement after the callback has executed.
>>
>> Ciao
>>
>> Â Â  Nat!
>>
>>