Problem with atexit and _dl_fini

[Nat! <nat at mulle-kybernetik dot com>]

So my problem is, that I observe that my atexit calls are not executed 
in the correct order.

i.e. atexit( a); atexit( b);  should result in b(), a() being called in 
that order. To quote the man page,

"the registered functions are invoked in reverse order".


When I register with `atexit` I can see my functions being added 
properly within `__internal_atexit` in the

correct order. Finally after my functions, the elf-loader ? also adds 
itself there. So it is being called first by

`__run_exit_handlers`.


Then comes the part where it goes wrong. I registered my two function 
with `__internal_atexit`, but for some reason

`_dl_fini` is calling `__cxa_finalize` and that is calling the wrong 
function first.


I would like to question the wisdom of `_dl_fini` calling my destructor, 
which I never registered via

__attribute__((destructor)) or somesuch. To me that is a bug. If 
`_dl_fini` would leave that to `__internal_atexit`,

there would be no problem. The comments in `_dl_fini` say, that it does 
some complicated dependency checks to

call destructors in the correct order. This seemingly interferes with 
the atexit order.



Here is a stacktrace of the wrong call back:

```

* thread #1, name = 'noleak.debug.ex', stop reason = breakpoint 2.1
  * frame #0: 0x00007ffff7f86740 
libmulle-testallocator.so`mulle_testallocator_exit at 
mulle-testallocator.c:451:14
    frame #1: 0x00007ffff7e131af 
libc.so.6`__cxa_finalize(d=0x00007ffff7f890e0) at cxa_finalize.c:83:6
    frame #2: 0x00007ffff7f85243 
libmulle-testallocator.so`__do_global_dtors_aux + 35
    frame #3: 0x00007ffff7fe2ce6 ld-2.29.so`_dl_fini at dl-fini.c:138:9
    frame #4: 0x00007ffff7e12c65 
libc.so.6`__run_exit_handlers(status=0, listp=0x00007ffff7f7d718, 
run_list_atexit=<unavailable>, run_dtors=<unavailable>) at exit.c:108:8
    frame #5: 0x00007ffff7e12d8c 
libc.so.6`__GI_exit(status=<unavailable>) at exit.c:139:3
    frame #6: 0x00007ffff7dfe587 
libc.so.6`__libc_start_main(main=(noleak.debug.exe`main at noleak.m:18), 
argc=1, argv=0x00007fffffffd728, init=(noleak.debug.exe`__libc_csu_init 
at elf-init.c:68:1), fini=<unavailable>, rtld_fini=<unavailable>, 
stack_end=0x00007fffffffd718) at libc-start.c:342:3
    frame #7: 0x000000000040110a noleak.debug.exe`_start + 42
```


Stacktrace of the corresponding atexit :

```

  * frame #0: 0x00007ffff7f869a0 libmulle-testallocator.so`atexit
    frame #1: 0x00007ffff7f868c7 
libmulle-testallocator.so`mulle_testallocator_initialize at 
mulle-testallocator.c:492:7
    frame #2: 0x00007ffff7fe295a ld-2.29.so`call_init(l=<unavailable>, 
argc=1, argv=0x00007fffffffd728, env=0x00007fffffffd738) at dl-init.c:72:3
    frame #3: 0x00007ffff7fe2a59 ld-2.29.so`_dl_init at dl-init.c:30:6
    frame #4: 0x00007ffff7fe2a43 
ld-2.29.so`_dl_init(main_map=0x00007ffff7ffe190, argc=1, 
argv=0x00007fffffffd728, env=0x00007fffffffd738) at dl-init.c:119
    frame #5: 0x00007ffff7fd30ca ld-2.29.so`_dl_start_user + 50
```


A tweet/screenshot of the problem in action, where you can see that 
libmulle-testallocator.so is calling atexit first, but also gets called 
back first:


https://twitter.com/mulle_nat/status/1129131042001043456


I tried to reproduce this with a smaller test case, but wasn't 
successful. I have the problem with system glibc as well as with my 
self-built debugging glibc-2.29.


```

(lldb) image list
[  0] 047FB15C 0x0000000000400000 
/home/src/srcO/mulle-objc/MulleObjC/test/0_noleak/noleak.debug.exe
[  1] 7A9876B1-3173-65A8-3AC8-D6F6E88FD53F-25BEA927 0x00007ffff7fd2000 
/lib/x86_64-linux-gnu/ld-2.29.so
      /usr/lib/debug/lib/x86_64-linux-gnu/ld-2.29.so
[  2] 76B794BB-7305-068D-C4A6-B2C6330A23DE-0BC95526 0x00007ffff7fd1000 
[vdso] (0x00007ffff7fd1000)
[  3] 05639C58 0x00007ffff7f8a000 
/home/src/srcO/mulle-objc/MulleObjC/test/dependency/lib/libMulleObjC.so
[  4] D1F022AE-06A7-DD85-43B7-D26B34D45415-59240C1E 0x00007ffff7f84000 
/home/src/srcO/mulle-objc/MulleObjC/test/dependency/lib/libmulle-testallocator.so 

[  5] 0F5AACB3-8162-0D16-F936-1C8C44F8B6D6-82423DF9 0x00007ffff7dd9000 
/home/src/srcO/mulle-objc/MulleObjC/test/dependency/lib/libc.so.6
[  6] 13E7B8ED-A5D0-4B2C-05BE-0926D37365CE-A861C019 0x00007ffff7d89000 
/home/src/srcO/mulle-objc/MulleObjC/test/dependency/lib/libmulle-objc-runtime.so 

[  7] 96F58694-13A4-8019-C8C7-47993BC8B10E-C11F8498 0x00007ffff7d84000 
/home/src/srcO/mulle-objc/MulleObjC/test/dependency/lib/libdl.so.2
[  8] 8BBE3351-7A55-AB93-23D3-E05E6DBF19C4-842503AB 0x00007ffff7d7a000 
/home/src/srcO/mulle-objc/MulleObjC/test/dependency/lib/libmulle-concurrent.so 

[  9] 4D189D24-CA86-BCAD-32DC-2C4E15D6DB70-C51C94FB 0x00007ffff7d6c000 
/home/src/srcO/mulle-objc/MulleObjC/test/dependency/lib/libmulle-aba.so
[ 10] DEB7E042-331F-B188-43B1-91C1225C51EB-DA4DAC39 0x00007ffff7d67000 
/home/src/srcO/mulle-objc/MulleObjC/test/dependency/lib/libmulle-allocator.so 

[ 11] 30F22D5C-1699-A1E2-C135-14F5A2199065-7DCC8A02 0x00007ffff7d62000 
/home/src/srcO/mulle-objc/MulleObjC/test/dependency/lib/libmulle-thread.so
[ 12] 6AC089FD-3867-9707-F7F1-4E2B6347F639-60D504FF 0x00007ffff7d5d000 
/home/src/srcO/mulle-objc/MulleObjC/test/dependency/lib/libmulle-vararg.so
[ 13] C18E6B6F-2BD8-DC81-92BE-E3CA239F4B92-D1FBB101 0x00007ffff7d56000 
/home/src/srcO/mulle-objc/MulleObjC/test/dependency/lib/libmulle-stacktrace.so 

[ 14] 272D4479-EC66-502F-C535-BBC9286963F6-09CA9E9D 0x00007ffff7d3d000 
/home/src/srcO/mulle-objc/MulleObjC/test/dependency/lib/libmulle-container.so 

[ 15] A6F82062-5433-D33D-4BC8-F81DA55BC85A-76B9D8AF 0x00007ffff7d1d000 
/home/src/srcO/mulle-objc/MulleObjC/test/dependency/lib/libpthread.so.0
[ 16] FB3F5F86-E867-5910-864C-3E064CEAC7D3-4FFFD5BC 0x00007ffff7cb2000 
/lib/x86_64-linux-gnu/libgcc_s.so.1
```

LD_DEBUG: doesn't say much:


```

     13670:
     13670:    calling fini: 
/home/src/srcO/mulle-objc/MulleObjC/test/0_noleak/noleak.debug.exe [0]
     13670:
     13670:
     13670:    calling fini: 
/home/src/srcO/mulle-objc/MulleObjC/test/dependency/lib/libMulleObjC.so [0]
     13670:
     13670:
     13670:    calling fini: 
/home/src/srcO/mulle-objc/MulleObjC/test/dependency/lib/libmulle-testallocator.so 
[0]
     13670:
mulle_testallocator: exit (0x7f78f9ee873c)
```

Ciao

   Nat!