ld.so mysterious behavior

[Olivier Langlois]

I am experiencing a mysterious result from ld.so.

I have 2 shared libraries built from the same source code. The only
differences are possibly the linking options and maybe they have few
differences in the obj code they contain but they should both contain
all the symbols that a.out needs. One is generated by bazel so I have
little exposure of what it is doing.

I have a small test program built like this:

# link against the bazel lib
lano1106@whippet2 ~/dev/kraken/test $ g++ -g -std=c++26 -
I/home/lano1106/dev/tcmalloc tcmalloc_new_test.cpp -
L/home/lano1106/dev/tcmalloc/bazel-bin/tcmalloc/ -ltcmalloc_new

# ld preload against the manually built so
lano1106@whippet2 ~/dev/kraken/test $
LD_PRELOAD=../tcmalloc/libtcmalloc_new.so ./a.out
a PerCpuCachesActive:0

# ld preload the bazel so
lano1106@whippet2 ~/dev/kraken/test $
LD_PRELOAD=/home/lano1106/dev/tcmalloc/bazel-
bin/tcmalloc/libtcmalloc_new.so ./a.out 
./a.out: error while loading shared libraries: libtcmalloc_new.so:
cannot open shared object file: No such file or directory

# copy bazel lib in the LD_LIBRARY_PATH
lano1106@whippet2 ~/dev/kraken/test :( $ cp
/home/lano1106/dev/tcmalloc/bazel-bin/tcmalloc/libtcmalloc_new.so ..
lano1106@whippet2 ~/dev/kraken/test $ ./a.out 
a PerCpuCachesActive:0

I have performed the same experiment with LD_DEBUG=all.

The successful execution gives:

   1035779:	symbol=__vdso_clock_gettime;  lookup in file=linux-
vdso.so.1 [0]
   1035779:	binding file linux-vdso.so.1 [0] to linux-vdso.so.1
[0]: normal symbol `__vdso_clock_gettime' [LINUX_2.6]
   1035779:	symbol=__vdso_gettimeofday;  lookup in file=linux-
vdso.so.1 [0]
   1035779:	binding file linux-vdso.so.1 [0] to linux-vdso.so.1
[0]: normal symbol `__vdso_gettimeofday' [LINUX_2.6]
   1035779:	symbol=__vdso_time;  lookup in file=linux-vdso.so.1
[0]
   1035779:	binding file linux-vdso.so.1 [0] to linux-vdso.so.1
[0]: normal symbol `__vdso_time' [LINUX_2.6]
   1035779:	symbol=__vdso_getcpu;  lookup in file=linux-vdso.so.1
[0]
   1035779:	binding file linux-vdso.so.1 [0] to linux-vdso.so.1
[0]: normal symbol `__vdso_getcpu' [LINUX_2.6]
   1035779:	symbol=__vdso_clock_getres;  lookup in file=linux-
vdso.so.1 [0]
   1035779:	binding file linux-vdso.so.1 [0] to linux-vdso.so.1
[0]: normal symbol `__vdso_clock_getres' [LINUX_2.6]
   1035779:	
   1035779:	file=../tcmalloc/libtcmalloc_new.so [0];  needed by
./a.out [0]
   1035779:	file=../tcmalloc/libtcmalloc_new.so [0];  generating
link map
   1035779:	  dynamic: 0x00007e56ebe596e0  base:
0x00007e56eba00000   size: 0x0000000000f94fc0
   1035779:	    entry: 0x00007e56eba00000  phdr:
0x00007e56eba00040  phnum:                 12
   1035779:	
   1035779:	
   1035779:	file=libstdc++.so.6 [0];  needed by ./a.out [0]
   1035779:	find library=libstdc++.so.6 [0]; searching

In the successful case, ld.so appears to be recognize the LD_PRELOAD
lib as one of the libs it needs to load and move on the next one in its
list.

with the bazel lib:
   1035804:	symbol=__vdso_clock_gettime;  lookup in file=linux-
vdso.so.1 [0]
   1035804:	binding file linux-vdso.so.1 [0] to linux-vdso.so.1
[0]: normal symbol `__vdso_clock_gettime' [LINUX_2.6]
   1035804:	symbol=__vdso_gettimeofday;  lookup in file=linux-
vdso.so.1 [0]
   1035804:	binding file linux-vdso.so.1 [0] to linux-vdso.so.1
[0]: normal symbol `__vdso_gettimeofday' [LINUX_2.6]
   1035804:	symbol=__vdso_time;  lookup in file=linux-vdso.so.1
[0]
   1035804:	binding file linux-vdso.so.1 [0] to linux-vdso.so.1
[0]: normal symbol `__vdso_time' [LINUX_2.6]
   1035804:	symbol=__vdso_getcpu;  lookup in file=linux-vdso.so.1
[0]
   1035804:	binding file linux-vdso.so.1 [0] to linux-vdso.so.1
[0]: normal symbol `__vdso_getcpu' [LINUX_2.6]
   1035804:	symbol=__vdso_clock_getres;  lookup in file=linux-
vdso.so.1 [0]
   1035804:	binding file linux-vdso.so.1 [0] to linux-vdso.so.1
[0]: normal symbol `__vdso_clock_getres' [LINUX_2.6]
   1035804:	
   1035804:	file=/home/lano1106/dev/tcmalloc/bazel-
bin/tcmalloc/libtcmalloc_new.so [0];  needed by ./a.out [0]
   1035804:	file=/home/lano1106/dev/tcmalloc/bazel-
bin/tcmalloc/libtcmalloc_new.so [0];  generating link map
   1035804:	  dynamic: 0x0000752dd722fe38  base:
0x0000752dd6e00000   size: 0x0000000000f6afb8
   1035804:	    entry: 0x0000752dd6e00000  phdr:
0x0000752dd6e00040  phnum:                 10
   1035804:	
   1035804:	
   1035804:	file=libtcmalloc_new.so [0];  needed by ./a.out [0]
   1035804:	find library=libtcmalloc_new.so [0]; searching
   1035804:	 search path=/home/lano1106/dev/kraken/glibc-
hwcaps/x86-64-
v2:/home/lano1106/dev/kraken:/home/lano1106/dev/qt/chap02/krakenqtlib/g
libc-hwcaps/x86-64-
v2:/home/lano1106/dev/qt/chap02/krakenqtlib		(LD_LIBRARY_PATH)
   1035804:	  trying file=/home/lano1106/dev/kraken/glibc-
hwcaps/x86-64-v2/libtcmalloc_new.so
   1035804:	  trying
file=/home/lano1106/dev/kraken/libtcmalloc_new.so
   1035804:	  trying
file=/home/lano1106/dev/qt/chap02/krakenqtlib/glibc-hwcaps/x86-64-
v2/libtcmalloc_new.so
   1035804:	  trying
file=/home/lano1106/dev/qt/chap02/krakenqtlib/libtcmalloc_new.so
   1035804:	 search cache=/etc/ld.so.cache
   1035804:	 search path=/usr/lib/glibc-hwcaps/x86-64-
v2:/usr/lib		(system search path)
   1035804:	  trying file=/usr/lib/glibc-hwcaps/x86-64-
v2/libtcmalloc_new.so
   1035804:	  trying file=/usr/lib/libtcmalloc_new.so
   1035804:	
./a.out: error while loading shared libraries: libtcmalloc_new.so:
cannot open shared object file: No such file or directory

I can provide more details if requested but I want the discussion to
remain focussed on ld.so.

>From the ld.so point of view, what makes it accept the LD_PRELOAD lib
as one of the libs that it needs to load, vs not recognizing it and
still searching for it in the LD_LIBRARY_PATH directories?

thank you,