Bug 15006 - Libraries in ld.so.cache ignored by ld-linux-armhf.so.3 on armv6l
Summary: Libraries in ld.so.cache ignored by ld-linux-armhf.so.3 on armv6l
Status: RESOLVED FIXED
Alias: None
Product: glibc
Classification: Unclassified
Component: dynamic-link (show other bugs)
Version: 2.17
: P2 normal
Target Milestone: 2.18
Assignee: Carlos O'Donell
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2013-01-09 00:49 UTC by lfs4pi
Modified: 2014-06-13 18:53 UTC (History)
8 users (show)

See Also:
Host:
Target:
Build:
Last reconfirmed:
fweimer: security-


Attachments
LFS system build script - Toolchain (3.63 KB, text/plain)
2013-01-09 16:02 UTC, lfs4pi
Details
LFS system build script - Final system (7.84 KB, text/plain)
2013-01-09 16:04 UTC, lfs4pi
Details

Note You need to log in before you can comment on or make changes to this bug.
Description lfs4pi 2013-01-09 00:49:53 UTC
I build Linux From Scratch systems for the Raspberry Pi and completed a new build where glibc 2.16 has been replaced by glibc 2.17.

The system runs great in general except for a peculiar behavior of the dynamic runtime linker to ignore needed libraries, in non-standard locations, that are clearly in ld.so.cache.

As an example I will try to execute a binary that depends on two libraries, that I compiled myself, located in /opt/vc/lib:

[root@0] [/] cd /opt/vc/bin
[root@0] [/opt/vc/bin] ./vcgencmd measure_temp
./vcgencmd: error while loading shared libraries: libvcos.so: cannot open shared object file: No such file or directory

Let's verify with ldd:

[root@0] [/opt/vc/bin] ldd vcgencmd
        libvcos.so => not found
        libvchiq_arm.so => not found
        libpthread.so.0 => /lib/libpthread.so.0 (0xb6f4f000)
        libdl.so.2 => /lib/libdl.so.2 (0xb6f44000)
        librt.so.1 => /lib/librt.so.1 (0xb6f35000)
        libc.so.6 => /lib/libc.so.6 (0xb6e03000)
        /lib/ld-linux-armhf.so.3 (0xb6f6e000)

But it's in the cache!
[root@0] [/opt/vc/bin] cat /etc/ld.so.conf
/opt/vc/lib
[root@0] [/opt/vc/bin] rm /etc/ld.so.cache
[root@0] [/opt/vc/bin] ldconfig -v | grep vcos
        libvcos.so -> libvcos.so
[root@0] [/opt/vc/bin] ldconfig -p | grep vcos
        libvcos.so (libc6) => /opt/vc/lib/libvcos.so

Using LD_LIBRARY_PATH, it works:
[root@0] [/opt/vc/bin] LD_LIBRARY_PATH=/opt/vc/lib ./vcgencmd measure_temp
temp=50.8'C

A debug output, where it at least seems to look at ld.so.cache:
[root@0] [/opt/vc/bin] LD_DEBUG=libs ./vcgencmd
       522:     find library=libvcos.so [0]; searching
       522:      search cache=/etc/ld.so.cache
       522:      search path=/lib/tls/v6l/vfp:/lib/tls/v6l:/lib/tls/vfp:/lib/tls:/lib/v6l/vfp:/lib/v6l:/lib/vfp:/lib:/usr/lib/tls/v6l/vfp:/usr/lib/tls/v6l:/usr/lib/tls/vfp:/usr/lib/tls:/usr/lib/v6l/vfp:/usr/lib/v6l:/usr/lib/vfp:/usr/lib            (system search path)
       522:       trying file=/lib/tls/v6l/vfp/libvcos.so
       522:       trying file=/lib/tls/v6l/libvcos.so
       522:       trying file=/lib/tls/vfp/libvcos.so
       522:       trying file=/lib/tls/libvcos.so
       522:       trying file=/lib/v6l/vfp/libvcos.so
       522:       trying file=/lib/v6l/libvcos.so
       522:       trying file=/lib/vfp/libvcos.so
       522:       trying file=/lib/libvcos.so
       522:       trying file=/usr/lib/tls/v6l/vfp/libvcos.so
       522:       trying file=/usr/lib/tls/v6l/libvcos.so
       522:       trying file=/usr/lib/tls/vfp/libvcos.so
       522:       trying file=/usr/lib/tls/libvcos.so
       522:       trying file=/usr/lib/v6l/vfp/libvcos.so
       522:       trying file=/usr/lib/v6l/libvcos.so
       522:       trying file=/usr/lib/vfp/libvcos.so
       522:       trying file=/usr/lib/libvcos.so
       522:    
./vcgencmd: error while loading shared libraries: libvcos.so: cannot open shared object file: No such file or directory

I don't have the time and probably not the skills necessary for a lengthy gdb session, but if I had to guess I'd point to the recent changes in ports/sysdeps/unix/sysv/linux/arm as a possible clue.

Apart from ld-linux-armhf.so.3, there's also a ld-linux.so.3 compatibility symlink. Not sure if that could play a part in this.
Comment 1 Steve McIntyre 2013-01-09 13:54:52 UTC
Hi,

What does ldconfig -p say about your libraries? I'm guessing there's a mismatch of the hard-float / soft-float ABI flags here.
Comment 2 lfs4pi 2013-01-09 14:38:45 UTC
(In reply to comment #1)
> Hi,
> 
> What does ldconfig -p say about your libraries? I'm guessing there's a mismatch
> of the hard-float / soft-float ABI flags here.

ldconfig -p says:
	libvcos.so (libc6) => /opt/vc/lib/libvcos.so
	libvchiq_arm.so (libc6) => /opt/vc/lib/libvchiq_arm.so
        ...
	ld-linux-armhf.so.3 (ELF) => /lib/ld-linux-armhf.so.3

Also, I can simply make symlinks to the libraries in /usr/lib and it works fine:
[root@0] [/] ln -s /opt/vc/lib/libvcos.so /usr/lib
[root@0] [/] ln -s /opt/vc/lib/libvchiq_arm.so /usr/lib
[root@0] [/] ldd /opt/vc/bin/vcgencmd
	libvcos.so => /usr/lib/libvcos.so (0xb6f96000)
	libvchiq_arm.so => /usr/lib/libvchiq_arm.so (0xb6f88000)
Comment 3 Steve McIntyre 2013-01-09 14:49:07 UTC
(In reply to comment #2)
> (In reply to comment #1)
> > Hi,
> > 
> > What does ldconfig -p say about your libraries? I'm guessing there's a mismatch
> > of the hard-float / soft-float ABI flags here.
> 
> ldconfig -p says:
>     libvcos.so (libc6) => /opt/vc/lib/libvcos.so
>     libvchiq_arm.so (libc6) => /opt/vc/lib/libvchiq_arm.so
>         ...
>     ld-linux-armhf.so.3 (ELF) => /lib/ld-linux-armhf.so.3
> 
> Also, I can simply make symlinks to the libraries in /usr/lib and it works
> fine:
> [root@0] [/] ln -s /opt/vc/lib/libvcos.so /usr/lib
> [root@0] [/] ln -s /opt/vc/lib/libvchiq_arm.so /usr/lib
> [root@0] [/] ldd /opt/vc/bin/vcgencmd
>     libvcos.so => /usr/lib/libvcos.so (0xb6f96000)
>     libvchiq_arm.so => /usr/lib/libvchiq_arm.so (0xb6f88000)

Hmmmm, that's weird. The ld-linux-armhf.so.3 entry should be saying that it's "libc6" at the very least, and most likely "hard-float", not just "ELF".

Also: what's in your ld.so.conf? And have you rebuilt the cache using ldconfig? What's the history of the system - is this newly-built, or upgraded from an older installation?
Comment 4 lfs4pi 2013-01-09 15:03:20 UTC
(In reply to comment #3)
> Hmmmm, that's weird. The ld-linux-armhf.so.3 entry should be saying that it's
> "libc6" at the very least, and most likely "hard-float", not just "ELF".
> 
> Also: what's in your ld.so.conf? And have you rebuilt the cache using ldconfig?
> What's the history of the system - is this newly-built, or upgraded from an
> older installation?

I just verified the output of ldconfig -p on a working "Raspbian" hard float distribution that also prints:
ld-linux-armhf.so.3 (ELF) => /lib/ld-linux-armhf.so.3
On my x86 linux it's also ld-linux.so.2 (ELF) => /lib/ld-linux.so.2
So I don't think that's it. Could be that ld-linux-armhf.so.3 is a symlink to ld-2.17.so which is more of a executable binary than an actual library if I understand it correctly?

In my /etc/ld.so.conf is the path you would expect:
/opt/vc/lib
And yes, I have rebuilt the cache.

The system is newly-built from scratch, not an upgrade.
Comment 5 Steve McIntyre 2013-01-09 15:46:10 UTC
> > Hmmmm, that's weird. The ld-linux-armhf.so.3 entry should be saying that it's
> > "libc6" at the very least, and most likely "hard-float", not just "ELF".
> > 
> > Also: what's in your ld.so.conf? And have you rebuilt the cache using ldconfig?
> > What's the history of the system - is this newly-built, or upgraded from an
> > older installation?
> 
> I just verified the output of ldconfig -p on a working "Raspbian" hard float
> distribution that also prints:
> ld-linux-armhf.so.3 (ELF) => /lib/ld-linux-armhf.so.3
> On my x86 linux it's also ld-linux.so.2 (ELF) => /lib/ld-linux.so.2
> So I don't think that's it. Could be that ld-linux-armhf.so.3 is a symlink to
> ld-2.17.so which is more of a executable binary than an actual library if I
> understand it correctly?
> 
> In my /etc/ld.so.conf is the path you would expect:
> /opt/vc/lib
> And yes, I have rebuilt the cache.
> 
> The system is newly-built from scratch, not an upgrade.

OK, so what version of binutils are you using? There are matching changes there to add the new HF/SF ABI tags.
Comment 6 lfs4pi 2013-01-09 16:02:08 UTC
Created attachment 6807 [details]
LFS system build script - Toolchain
Comment 7 lfs4pi 2013-01-09 16:04:06 UTC
Created attachment 6808 [details]
LFS system build script - Final system
Comment 8 lfs4pi 2013-01-09 16:05:37 UTC
(In reply to comment #5)
> OK, so what version of binutils are you using? There are matching changes there
> to add the new HF/SF ABI tags.

binutils-2.23.1

It's a bit messy but I'm attaching the build scripts that I used to produce the system.
They might not make much sense to someone not familiar with the Linux From Scratch project but at least all the tarballs that were used are in there and every command line statement used to compile them.
ch5-wip.sh builds the basic toolchain.
ch6-wip.sh builds the finished system from within a chroot.
Comment 9 Jason Plum 2013-01-10 00:33:07 UTC
We are also experiencing this exact problem with the armv6l & armv7l gnueabihf.

our build scripts are here: 
https://github.com/archlinuxarm/PKGBUILDs/tree/master/core/glibc
https://github.com/archlinuxarm/PKGBUILDs/tree/master/core/binutils

Here the output from pacman, showing relavant build information and dates:
# pacman -Qi binutils glibc
Name           : binutils
Version        : 2.23.1-2
URL            : http://www.gnu.org/software/binutils/
Licenses       : GPL
Groups         : base-devel
Provides       : None
Depends On     : glibc>=2.17  zlib
Optional Deps  : None
Required By    : gcc
Conflicts With : None
Replaces       : None
Installed Size : 15704.00 KiB
Packager       : Kevin Mihelich <kevin@archlinuxarm.org>
Architecture   : armv6h
Build Date     : Mon Dec 31 01:43:51 2012
Install Date   : Wed Jan 9 23:42:06 2013
Install Reason : Explicitly installed
Install Script : Yes
Description    : A set of programs to assemble and manipulate binary and object files

Name           : glibc
Version        : 2.17-1
URL            : http://www.gnu.org/software/libc
Licenses       : GPL  LGPL
Groups         : base
Provides       : None
Depends On     : linux-api-headers>=3.7  tzdata
Optional Deps  : None
Required By    : alsa-lib  attr  bash  binutils  bison  bzip2  coreutils  cracklib  dash
                 device-mapper  dhcpcd  diffutils  eventlog  expat  fakeroot  file  findutils  flex
                 fribidi  fuse  gawk  gcc-libs  gdbm  grep  gsm  gzip  iproute2  iptables  json-c
                 kbd  keyutils  kmod  libasyncns  libcap  libcap-ng  libcddb  libdrm  libffi
                 libglapi  libgpg-error  libgssglue  libice  libidn  libjpeg-turbo  libksba  libmad
                 libmpeg2  libnfs  libnl  libogg  libpcap  libpciaccess  libpipeline  libsigsegv
                 libtasn1  libusbx  libvpx  libwbclient  libxau  libxdmcp  lzo2  m4  make  mdadm
                 mkinitcpio-busybox  ncurses  net-tools  nss-myhostname  ntfs-3g  opencore-amr  orc
                 p11-kit  pacman  pam  patch  pciutils  perl  popt  ppp  pth  readline  recode
                 run-parts  sdl  sysfsutils  sysvinit-tools  talloc  tar  tree  util-linux
                 v4l-utils  which  x264  xvidcore  zlib
Conflicts With : None
Replaces       : None
Installed Size : 33728.00 KiB
Packager       : Kevin Mihelich <kevin@archlinuxarm.org>
Architecture   : armv6h
Build Date     : Mon Dec 31 03:01:45 2012
Install Date   : Wed Jan 9 23:50:19 2013
Install Reason : Explicitly installed
Install Script : Yes
Description    : GNU C Library
Comment 10 Steve McIntyre 2013-01-10 11:22:08 UTC
(In reply to comment #2)
> (In reply to comment #1)
> > Hi,
> > 
> > What does ldconfig -p say about your libraries? I'm guessing there's a mismatch
> > of the hard-float / soft-float ABI flags here.
> 
> ldconfig -p says:
>     libvcos.so (libc6) => /opt/vc/lib/libvcos.so
>     libvchiq_arm.so (libc6) => /opt/vc/lib/libvchiq_arm.so
>         ...
>     ld-linux-armhf.so.3 (ELF) => /lib/ld-linux-armhf.so.3
> 
> Also, I can simply make symlinks to the libraries in /usr/lib and it works
> fine:
> [root@0] [/] ln -s /opt/vc/lib/libvcos.so /usr/lib
> [root@0] [/] ln -s /opt/vc/lib/libvchiq_arm.so /usr/lib
> [root@0] [/] ldd /opt/vc/bin/vcgencmd
>     libvcos.so => /usr/lib/libvcos.so (0xb6f96000)
>     libvchiq_arm.so => /usr/lib/libvchiq_arm.so (0xb6f88000)

Hi,

Could you show me all the output from ldconfig -p please? I'd like to see what the rest of your libs look like. Also: I'm curious why you've got something under /opt - is that something you've built as part of your system, or is /opt/vc a precompiled binary from somewhere else?

readelf -A /lib/libc.so.6
readelf -A /opt/vc/lib/libvcos.so

might also be instructive here.
Comment 11 Steve McIntyre 2013-01-10 11:28:51 UTC
(In reply to comment #9)
> We are also experiencing this exact problem with the armv6l & armv7l gnueabihf.
> 
> our build scripts are here: 
> https://github.com/archlinuxarm/PKGBUILDs/tree/master/core/glibc
> https://github.com/archlinuxarm/PKGBUILDs/tree/master/core/binutils
> 
> Here the output from pacman, showing relavant build information and dates:
> # pacman -Qi binutils glibc
> Name           : binutils
> Version        : 2.23.1-2
> URL            : http://www.gnu.org/software/binutils/
> Licenses       : GPL
> Groups         : base-devel
> Provides       : None
> Depends On     : glibc>=2.17  zlib
> Optional Deps  : None
> Required By    : gcc
> Conflicts With : None
> Replaces       : None
> Installed Size : 15704.00 KiB
> Packager       : Kevin Mihelich <kevin@archlinuxarm.org>
> Architecture   : armv6h
> Build Date     : Mon Dec 31 01:43:51 2012
> Install Date   : Wed Jan 9 23:42:06 2013
> Install Reason : Explicitly installed
> Install Script : Yes
> Description    : A set of programs to assemble and manipulate binary and object
> files
> 
> Name           : glibc
> Version        : 2.17-1
> URL            : http://www.gnu.org/software/libc
> Licenses       : GPL  LGPL
> Groups         : base
> Provides       : None
> Depends On     : linux-api-headers>=3.7  tzdata
> Optional Deps  : None
> Required By    : alsa-lib  attr  bash  binutils  bison  bzip2  coreutils 
> cracklib  dash
>                  device-mapper  dhcpcd  diffutils  eventlog  expat  fakeroot 
> file  findutils  flex
>                  fribidi  fuse  gawk  gcc-libs  gdbm  grep  gsm  gzip  iproute2
>  iptables  json-c
>                  kbd  keyutils  kmod  libasyncns  libcap  libcap-ng  libcddb 
> libdrm  libffi
>                  libglapi  libgpg-error  libgssglue  libice  libidn 
> libjpeg-turbo  libksba  libmad
>                  libmpeg2  libnfs  libnl  libogg  libpcap  libpciaccess 
> libpipeline  libsigsegv
>                  libtasn1  libusbx  libvpx  libwbclient  libxau  libxdmcp  lzo2
>  m4  make  mdadm
>                  mkinitcpio-busybox  ncurses  net-tools  nss-myhostname 
> ntfs-3g  opencore-amr  orc
>                  p11-kit  pacman  pam  patch  pciutils  perl  popt  ppp  pth 
> readline  recode
>                  run-parts  sdl  sysfsutils  sysvinit-tools  talloc  tar  tree 
> util-linux
>                  v4l-utils  which  x264  xvidcore  zlib
> Conflicts With : None
> Replaces       : None
> Installed Size : 33728.00 KiB
> Packager       : Kevin Mihelich <kevin@archlinuxarm.org>
> Architecture   : armv6h
> Build Date     : Mon Dec 31 03:01:45 2012
> Install Date   : Wed Jan 9 23:50:19 2013
> Install Reason : Explicitly installed
> Install Script : Yes
> Description    : GNU C Library

Hi Jason,

You say "the exact same problem" - is this with exactly the same binaries (vcgencmd etc.) or with others? Again, are they binaries you've built yourself recently or older ones? Do you know if they are using the soft-float or hard-float ABI?
Comment 12 lfs4pi 2013-01-10 12:37:17 UTC
(In reply to comment #10)
> Hi,
> 
> Could you show me all the output from ldconfig -p please? I'd like to see what
> the rest of your libs look like. Also: I'm curious why you've got something
> under /opt - is that something you've built as part of your system, or is
> /opt/vc a precompiled binary from somewhere else?
> 
> readelf -A /lib/libc.so.6
> readelf -A /opt/vc/lib/libvcos.so
> 
> might also be instructive here.

/opt/vc is the common location for the userland libraries on the Raspberry Pi computer.
I built both the vcgencmd binrary and the libraries it links to myself, natively on the same system.
The same problem remains if I move the libraries into /usr/local/lib for example, any non-default library location shows the same strangeness.

Here is the full output of ldconfig -p:

[root@0] [/] ldconfig -p
159 libs found in cache `/etc/ld.so.cache'
	libz.so.1 (libc6) => /lib/libz.so.1
	libz.so (libc6) => /usr/lib/libz.so
	libvcos.so (libc6) => /opt/vc/lib/libvcos.so
	libvchiq_arm.so (libc6) => /opt/vc/lib/libvchiq_arm.so
	libuuid.so.1 (libc6) => /lib/libuuid.so.1
	libuuid.so (libc6) => /usr/lib/libuuid.so
	libutil.so.1 (libc6, OS ABI: Linux 2.6.25) => /lib/libutil.so.1
	libutil.so (libc6, OS ABI: Linux 2.6.25) => /usr/lib/libutil.so
	libudev.so.1 (libc6) => /lib/libudev.so.1
	libudev.so (libc6) => /usr/lib/libudev.so
	libthread_db.so.1 (libc6, OS ABI: Linux 2.6.25) => /lib/libthread_db.so.1
	libthread_db.so (libc6, OS ABI: Linux 2.6.25) => /usr/lib/libthread_db.so
	libstdc++.so.6 (libc6) => /usr/lib/libstdc++.so.6
	libstdc++.so (libc6) => /usr/lib/libstdc++.so
	libssp.so.0 (libc6) => /usr/lib/libssp.so.0
	libssp.so (libc6) => /usr/lib/libssp.so
	libssl.so.1.0.0 (libc6) => /usr/lib/libssl.so.1.0.0
	libssl.so (libc6) => /usr/lib/libssl.so
	libss.so.2 (libc6) => /lib/libss.so.2
	libss.so (libc6) => /usr/lib/libss.so
	libsqlite3.so.0 (libc6) => /usr/lib/libsqlite3.so.0
	libsqlite3.so (libc6) => /usr/lib/libsqlite3.so
	librt.so.1 (libc6, OS ABI: Linux 2.6.25) => /lib/librt.so.1
	librt.so (libc6, OS ABI: Linux 2.6.25) => /usr/lib/librt.so
	libresolv.so.2 (libc6, OS ABI: Linux 2.6.25) => /lib/libresolv.so.2
	libresolv.so (libc6, OS ABI: Linux 2.6.25) => /usr/lib/libresolv.so
	libreadline.so.6 (libc6) => /lib/libreadline.so.6
	libreadline.so (libc6) => /usr/lib/libreadline.so
	libpython2.7.so.1.0 (libc6) => /usr/lib/libpython2.7.so.1.0
	libpython2.7.so (libc6) => /usr/lib/libpython2.7.so
	libpthread.so.0 (libc6, OS ABI: Linux 2.6.25) => /lib/libpthread.so.0
	libproc-3.2.8.so (libc6) => /lib/libproc-3.2.8.so
	libpipeline.so.1 (libc6) => /usr/lib/libpipeline.so.1
	libpipeline.so (libc6) => /usr/lib/libpipeline.so
	libpcreposix.so.0 (libc6) => /usr/lib/libpcreposix.so.0
	libpcreposix.so (libc6) => /usr/lib/libpcreposix.so
	libpcrecpp.so.0 (libc6) => /usr/lib/libpcrecpp.so.0
	libpcrecpp.so (libc6) => /usr/lib/libpcrecpp.so
	libpcre.so.1 (libc6) => /lib/libpcre.so.1
	libpcre.so (libc6) => /usr/lib/libpcre.so
	libpcprofile.so (libc6, OS ABI: Linux 2.6.25) => /lib/libpcprofile.so
	libpanelw.so.5 (libc6) => /usr/lib/libpanelw.so.5
	libpanelw.so (libc6) => /usr/lib/libpanelw.so
	libopenmaxil.so (libc6) => /opt/vc/lib/libopenmaxil.so
	libopcodes-2.23.1.so (libc6) => /usr/lib/libopcodes-2.23.1.so
	libnss_nisplus.so.2 (libc6, OS ABI: Linux 2.6.25) => /lib/libnss_nisplus.so.2
	libnss_nisplus.so (libc6, OS ABI: Linux 2.6.25) => /usr/lib/libnss_nisplus.so
	libnss_nis.so.2 (libc6, OS ABI: Linux 2.6.25) => /lib/libnss_nis.so.2
	libnss_nis.so (libc6, OS ABI: Linux 2.6.25) => /usr/lib/libnss_nis.so
	libnss_hesiod.so.2 (libc6, OS ABI: Linux 2.6.25) => /lib/libnss_hesiod.so.2
	libnss_hesiod.so (libc6, OS ABI: Linux 2.6.25) => /usr/lib/libnss_hesiod.so
	libnss_files.so.2 (libc6, OS ABI: Linux 2.6.25) => /lib/libnss_files.so.2
	libnss_files.so (libc6, OS ABI: Linux 2.6.25) => /usr/lib/libnss_files.so
	libnss_dns.so.2 (libc6, OS ABI: Linux 2.6.25) => /lib/libnss_dns.so.2
	libnss_dns.so (libc6, OS ABI: Linux 2.6.25) => /usr/lib/libnss_dns.so
	libnss_db.so.2 (libc6, OS ABI: Linux 2.6.25) => /lib/libnss_db.so.2
	libnss_db.so (libc6, OS ABI: Linux 2.6.25) => /usr/lib/libnss_db.so
	libnss_compat.so.2 (libc6, OS ABI: Linux 2.6.25) => /lib/libnss_compat.so.2
	libnss_compat.so (libc6, OS ABI: Linux 2.6.25) => /usr/lib/libnss_compat.so
	libnsl.so.1 (libc6, OS ABI: Linux 2.6.25) => /lib/libnsl.so.1
	libnsl.so (libc6, OS ABI: Linux 2.6.25) => /usr/lib/libnsl.so
	libnl-3.so.200 (libc6) => /usr/lib/libnl-3.so.200
	libnl-3.so (libc6) => /usr/lib/libnl-3.so
	libnl-route-3.so.200 (libc6) => /usr/lib/libnl-route-3.so.200
	libnl-route-3.so (libc6) => /usr/lib/libnl-route-3.so
	libnl-nf-3.so.200 (libc6) => /usr/lib/libnl-nf-3.so.200
	libnl-nf-3.so (libc6) => /usr/lib/libnl-nf-3.so
	libnl-genl-3.so.200 (libc6) => /usr/lib/libnl-genl-3.so.200
	libnl-genl-3.so (libc6) => /usr/lib/libnl-genl-3.so
	libnl-cli-3.so.200 (libc6) => /usr/lib/libnl-cli-3.so.200
	libnl-cli-3.so (libc6) => /usr/lib/libnl-cli-3.so
	libncursesw.so.5 (libc6) => /lib/libncursesw.so.5
	libncursesw.so (libc6) => /usr/lib/libncursesw.so
	libmudflapth.so.0 (libc6) => /usr/lib/libmudflapth.so.0
	libmudflapth.so (libc6) => /usr/lib/libmudflapth.so
	libmudflap.so.0 (libc6) => /usr/lib/libmudflap.so.0
	libmudflap.so (libc6) => /usr/lib/libmudflap.so
	libmpfr.so.4 (libc6) => /usr/lib/libmpfr.so.4
	libmpfr.so (libc6) => /usr/lib/libmpfr.so
	libmpc.so.3 (libc6) => /usr/lib/libmpc.so.3
	libmpc.so (libc6) => /usr/lib/libmpc.so
	libmount.so.1 (libc6) => /lib/libmount.so.1
	libmount.so (libc6) => /usr/lib/libmount.so
	libmmal_vc_client.so (libc6) => /opt/vc/lib/libmmal_vc_client.so
	libmmal.so (libc6) => /opt/vc/lib/libmmal.so
	libmenuw.so.5 (libc6) => /usr/lib/libmenuw.so.5
	libmenuw.so (libc6) => /usr/lib/libmenuw.so
	libmemusage.so (libc6, OS ABI: Linux 2.6.25) => /lib/libmemusage.so
	libmagic.so.1 (libc6) => /usr/lib/libmagic.so.1
	libmagic.so (libc6) => /usr/lib/libmagic.so
	libm.so.6 (libc6, OS ABI: Linux 2.6.25) => /lib/libm.so.6
	libm.so (libc6, OS ABI: Linux 2.6.25) => /usr/lib/libm.so
	liblzma.so.5 (libc6) => /lib/liblzma.so.5
	liblzma.so (libc6) => /lib/liblzma.so
	libltdl.so.7 (libc6) => /usr/lib/libltdl.so.7
	libltdl.so (libc6) => /usr/lib/libltdl.so
	libkmod.so.2 (libc6) => /lib/libkmod.so.2
	libkmod.so (libc6) => /lib/libkmod.so
	libitm.so.1 (libc6) => /usr/lib/libitm.so.1
	libitm.so (libc6) => /usr/lib/libitm.so
	libhistory.so.6 (libc6) => /lib/libhistory.so.6
	libhistory.so (libc6) => /usr/lib/libhistory.so
	libgomp.so.1 (libc6) => /usr/lib/libgomp.so.1
	libgomp.so (libc6) => /usr/lib/libgomp.so
	libgmpxx.so.4 (libc6) => /usr/lib/libgmpxx.so.4
	libgmpxx.so (libc6) => /usr/lib/libgmpxx.so
	libgmp.so.10 (libc6) => /usr/lib/libgmp.so.10
	libgmp.so (libc6) => /usr/lib/libgmp.so
	libgettextsrc-0.18.2.so (libc6) => /usr/lib/libgettextsrc-0.18.2.so
	libgettextpo.so.0 (libc6) => /usr/lib/libgettextpo.so.0
	libgettextpo.so (libc6) => /usr/lib/libgettextpo.so
	libgettextlib-0.18.2.so (libc6) => /usr/lib/libgettextlib-0.18.2.so
	libgdbm_compat.so.4 (libc6) => /usr/lib/libgdbm_compat.so.4
	libgdbm_compat.so (libc6) => /usr/lib/libgdbm_compat.so
	libgdbm.so.4 (libc6) => /usr/lib/libgdbm.so.4
	libgdbm.so (libc6) => /usr/lib/libgdbm.so
	libgcc_s.so.1 (libc6) => /usr/lib/libgcc_s.so.1
	libformw.so.5 (libc6) => /usr/lib/libformw.so.5
	libformw.so (libc6) => /usr/lib/libformw.so
	libe2p.so.2 (libc6) => /lib/libe2p.so.2
	libe2p.so (libc6) => /usr/lib/libe2p.so
	libext2fs.so.2 (libc6) => /lib/libext2fs.so.2
	libext2fs.so (libc6) => /usr/lib/libext2fs.so
	libexpat.so.1 (libc6) => /usr/lib/libexpat.so.1
	libexpat.so (libc6) => /usr/lib/libexpat.so
	libdl.so.2 (libc6, OS ABI: Linux 2.6.25) => /lib/libdl.so.2
	libdl.so (libc6, OS ABI: Linux 2.6.25) => /usr/lib/libdl.so
	libcurl.so.4 (libc6) => /usr/lib/libcurl.so.4
	libcurl.so (libc6) => /usr/lib/libcurl.so
	libcrypto.so.1.0.0 (libc6) => /usr/lib/libcrypto.so.1.0.0
	libcrypto.so (libc6) => /usr/lib/libcrypto.so
	libcrypt.so.1 (libc6, OS ABI: Linux 2.6.25) => /lib/libcrypt.so.1
	libcrypt.so (libc6, OS ABI: Linux 2.6.25) => /usr/lib/libcrypt.so
	libcom_err.so.2 (libc6) => /lib/libcom_err.so.2
	libcom_err.so (libc6) => /usr/lib/libcom_err.so
	libcofi_rpi.so (libc6) => /usr/lib/libcofi_rpi.so
	libcidn.so.1 (libc6, OS ABI: Linux 2.6.25) => /lib/libcidn.so.1
	libcidn.so (libc6, OS ABI: Linux 2.6.25) => /usr/lib/libcidn.so
	libc.so.6 (libc6, OS ABI: Linux 2.6.25) => /lib/libc.so.6
	libbz2.so.1.0 (libc6) => /lib/libbz2.so.1.0
	libbz2.so (libc6) => /usr/lib/libbz2.so
	libblkid.so.1 (libc6) => /lib/libblkid.so.1
	libblkid.so (libc6) => /usr/lib/libblkid.so
	libbfd-2.23.1.so (libc6) => /usr/lib/libbfd-2.23.1.so
	libbcm_host.so (libc6) => /opt/vc/lib/libbcm_host.so
	libasprintf.so.0 (libc6) => /usr/lib/libasprintf.so.0
	libasprintf.so (libc6) => /usr/lib/libasprintf.so
	libarchive.so.12 (libc6) => /usr/lib/libarchive.so.12
	libarchive.so (libc6) => /usr/lib/libarchive.so
	libanl.so.1 (libc6, OS ABI: Linux 2.6.25) => /lib/libanl.so.1
	libanl.so (libc6, OS ABI: Linux 2.6.25) => /usr/lib/libanl.so
	libWFC.so (libc6) => /opt/vc/lib/libWFC.so
	libSegFault.so (libc6, OS ABI: Linux 2.6.25) => /lib/libSegFault.so
	libOpenVG.so (libc6) => /opt/vc/lib/libOpenVG.so
	libGLESv2.so (libc6) => /opt/vc/lib/libGLESv2.so
	libEGL.so (libc6) => /opt/vc/lib/libEGL.so
	libBrokenLocale.so.1 (libc6, OS ABI: Linux 2.6.25) => /lib/libBrokenLocale.so.1
	libBrokenLocale.so (libc6, OS ABI: Linux 2.6.25) => /usr/lib/libBrokenLocale.so
	ld-linux-armhf.so.3 (ELF) => /lib/ld-linux-armhf.so.3

Here's the output of readelf -A /lib/libc.so.6:

[root@0] [/] readelf -A /lib/libc.so.6
Attribute Section: aeabi
File Attributes
  Tag_CPU_name: "6ZK"
  Tag_CPU_arch: v6KZ
  Tag_ARM_ISA_use: Yes
  Tag_THUMB_ISA_use: Thumb-1
  Tag_FP_arch: VFPv2
  Tag_ABI_PCS_wchar_t: 4
  Tag_ABI_FP_rounding: Needed
  Tag_ABI_FP_denormal: Needed
  Tag_ABI_FP_exceptions: Needed
  Tag_ABI_FP_number_model: IEEE 754
  Tag_ABI_align_needed: 8-byte
  Tag_ABI_align_preserved: 8-byte, except leaf SP
  Tag_ABI_enum_size: int
  Tag_ABI_HardFP_use: SP and DP
  Tag_ABI_VFP_args: VFP registers
  Tag_CPU_unaligned_access: v6
  Tag_Virtualization_use: TrustZone

Here's the output of readelf -A /opt/vc/lib/libvcos.so:

[root@0] [/] readelf -A /opt/vc/lib/libvcos.so
Attribute Section: aeabi
File Attributes
  Tag_CPU_name: "6ZK"
  Tag_CPU_arch: v6KZ
  Tag_ARM_ISA_use: Yes
  Tag_THUMB_ISA_use: Thumb-1
  Tag_FP_arch: VFPv2
  Tag_ABI_PCS_wchar_t: 4
  Tag_ABI_FP_denormal: Needed
  Tag_ABI_FP_exceptions: Needed
  Tag_ABI_FP_number_model: IEEE 754
  Tag_ABI_align_needed: 8-byte
  Tag_ABI_align_preserved: 8-byte, except leaf SP
  Tag_ABI_enum_size: int
  Tag_ABI_HardFP_use: SP and DP
  Tag_ABI_VFP_args: VFP registers
  Tag_CPU_unaligned_access: v6
  Tag_Virtualization_use: TrustZone
Comment 13 Steve McIntyre 2013-01-10 13:49:35 UTC
(In reply to comment #12)
> (In reply to comment #10)
> > Hi,
> > 
> > Could you show me all the output from ldconfig -p please? I'd like to see what
> > the rest of your libs look like. Also: I'm curious why you've got something
> > under /opt - is that something you've built as part of your system, or is
> > /opt/vc a precompiled binary from somewhere else?
> > 
> > readelf -A /lib/libc.so.6
> > readelf -A /opt/vc/lib/libvcos.so
> > 
> > might also be instructive here.
> 
> /opt/vc is the common location for the userland libraries on the Raspberry Pi
> computer.
> I built both the vcgencmd binrary and the libraries it links to myself,
> natively on the same system.
> The same problem remains if I move the libraries into /usr/local/lib for
> example, any non-default library location shows the same strangeness.

OK, fine - just wanted to check!

> Here is the full output of ldconfig -p:
> 
> [root@0] [/] ldconfig -p
> 159 libs found in cache `/etc/ld.so.cache'
>     libz.so.1 (libc6) => /lib/libz.so.1
>     libz.so (libc6) => /usr/lib/libz.so
>     libvcos.so (libc6) => /opt/vc/lib/libvcos.so
>     libvchiq_arm.so (libc6) => /opt/vc/lib/libvchiq_arm.so

<snip>

>     libcidn.so (libc6, OS ABI: Linux 2.6.25) => /usr/lib/libcidn.so
>     libc.so.6 (libc6, OS ABI: Linux 2.6.25) => /lib/libc.so.6
>     libbz2.so.1.0 (libc6) => /lib/libbz2.so.1.0

<snip>

OK, so all of your libs look consistent. That's good.

> Here's the output of readelf -A /lib/libc.so.6:
> 
> [root@0] [/] readelf -A /lib/libc.so.6
> Attribute Section: aeabi
> File Attributes
>   Tag_CPU_name: "6ZK"
>   Tag_CPU_arch: v6KZ
>   Tag_ARM_ISA_use: Yes
>   Tag_THUMB_ISA_use: Thumb-1
>   Tag_FP_arch: VFPv2
>   Tag_ABI_PCS_wchar_t: 4
>   Tag_ABI_FP_rounding: Needed
>   Tag_ABI_FP_denormal: Needed
>   Tag_ABI_FP_exceptions: Needed
>   Tag_ABI_FP_number_model: IEEE 754
>   Tag_ABI_align_needed: 8-byte
>   Tag_ABI_align_preserved: 8-byte, except leaf SP
>   Tag_ABI_enum_size: int
>   Tag_ABI_HardFP_use: SP and DP
>   Tag_ABI_VFP_args: VFP registers
>   Tag_CPU_unaligned_access: v6
>   Tag_Virtualization_use: TrustZone
> 
> Here's the output of readelf -A /opt/vc/lib/libvcos.so:
> 
> [root@0] [/] readelf -A /opt/vc/lib/libvcos.so
> Attribute Section: aeabi
> File Attributes
>   Tag_CPU_name: "6ZK"
>   Tag_CPU_arch: v6KZ
>   Tag_ARM_ISA_use: Yes
>   Tag_THUMB_ISA_use: Thumb-1
>   Tag_FP_arch: VFPv2
>   Tag_ABI_PCS_wchar_t: 4
>   Tag_ABI_FP_denormal: Needed
>   Tag_ABI_FP_exceptions: Needed
>   Tag_ABI_FP_number_model: IEEE 754
>   Tag_ABI_align_needed: 8-byte
>   Tag_ABI_align_preserved: 8-byte, except leaf SP
>   Tag_ABI_enum_size: int
>   Tag_ABI_HardFP_use: SP and DP
>   Tag_ABI_VFP_args: VFP registers
>   Tag_CPU_unaligned_access: v6
>   Tag_Virtualization_use: TrustZone

However, I'm a little worried by the fact that all of your libs are
showing up as using the soft-float ABI (lack of ",hard-float" in the
ldconfig output), but readelf shows that these two libraries are both
meant to be HFABI (Tag_ABI_VFP_args: VFP registers). That's more
expected on a system with older binaries, but with new binutils and
new glibc all the HFABI flags should be there. However, even given
that I don't see why on earth that would be affecting the library
search *paths* in the way you're seeing. If you had a mix of ABIs in
your libraries, that would be blocking incompatible loading
etc. regardless of where they live.

I'm going to do an LFS build on a machine locally now so that I can
see what on earth is going on, but that's going to take a short
while. At least I have a reasonably fast v7 machine to work with
here...
Comment 14 Jason Plum 2013-01-10 14:41:33 UTC
We also see the problem with other programs, such as clang/llvm http://archlinuxarm.org/forum/viewtopic.php?f=31&t=4832

Here is the ld.so.config(s) & cache:
cat /etc/ld.so.conf : http://sprunge.us/MjXi
cat /etc/ld.so.conf.d/* :  http://sprunge.us/BdPa
ldconfig -p :  http://sprunge.us/SRiN

Here are some readelfs:
readelf -A /opt/vc/bin/vcgencmd  : http://sprunge.us/dYBZ
readelf -A /usr/bin/clang : http://sprunge.us/JgXN
readelf -A /usr/lib/llvm/libLLVM-3.2.so : http://sprunge.us/UdHL

Here is package information & listing:
pacman -Ql raspberrypi-firmware llvm clang (complete file listing and location) :  http://sprunge.us/jMDa
pacman -Qi raspberrypi-firmware llvm clang (package information): http://sprunge.us/MURg
Comment 15 Steve McIntyre 2013-01-11 18:23:29 UTC
Lots of failures trying to do the LFS build here. :-(

Is there any chance you could give me a copy of your environment easily?
Comment 16 Jason Plum 2013-01-11 18:36:20 UTC
Is that directed at me, lfs4pi@gmail, or both?
Comment 17 lfs4pi 2013-01-11 18:41:01 UTC
(In reply to comment #15)
> Lots of failures trying to do the LFS build here. :-(
> 
> Is there any chance you could give me a copy of your environment easily?
Sure, I could either prepare a QEMU disk image or you can have a fat tarball of
all the binaries to run on any real armv6 hardware you have available.

The QEMU route is probably easier...

I don't have the unstripped glibc 2.17 binaries or source tree around but I
could produce those in a day or so if you think it would help.
Comment 18 lfs4pi 2013-01-11 22:16:29 UTC
(In reply to comment #17)
> (In reply to comment #15)
> > Lots of failures trying to do the LFS build here. :-(
> > 
> > Is there any chance you could give me a copy of your environment easily?
> Sure, I could either prepare a QEMU disk image or you can have a fat tarball of
> all the binaries to run on any real armv6 hardware you have available.
> 
> The QEMU route is probably easier...
> 
> I don't have the unstripped glibc 2.17 binaries or source tree around but I
> could produce those in a day or so if you think it would help.

Alright, you can grab the QEMU image here: http://circu.it/pilfs/glibc217.ext4.xz
And a suitable kernel here: http://circu.it/pilfs/pilfs-qemu-kernel.zip

Start QEMU something like so: qemu-system-arm -M versatilepb -cpu arm1176 -m 256 -kernel kernel.img -drive file=glibc217.ext4,cache=none -append "root=/dev/sda rootfstype=ext4"

Login as root, password "pilfs".

Let me know if you have any problems running the images.
Comment 19 lfs4pi 2013-01-13 20:29:48 UTC
(In reply to comment #18)
> Alright, you can grab the QEMU image here:
> http://circu.it/pilfs/glibc217.ext4.xz
> And a suitable kernel here: http://circu.it/pilfs/pilfs-qemu-kernel.zip
> 
> Start QEMU something like so: qemu-system-arm -M versatilepb -cpu arm1176 -m
> 256 -kernel kernel.img -drive file=glibc217.ext4,cache=none -append
> "root=/dev/sda rootfstype=ext4"
> 
> Login as root, password "pilfs".
> 
> Let me know if you have any problems running the images.

New QEMU disk image with unstripped binaries, gdb and compiled glibc source tree in /sources:
http://circu.it/pilfs/glibc217debug.ext4.xz (408 Mb)

Hope it proves helpful.
Comment 20 Steve McIntyre 2013-01-23 18:25:20 UTC
Thanks for the images, they've been very helpful in finally(!) tracking down the problem. I've worked it out.

The new hard-float ABI detection code in ld.so and ldconfig is designed to stop the problems that might happen if you have an inconsistent mix of soft- and hard-float ABI programs and libraries. ldconfig looks for the EF_ARM_ABI_FLOAT_HARD or EF_ARM_ABI_FLOAT_SOFT flags in the libraries it scans, and those flags are added by binutils as libraries and programs are built.

Unfortunately... there's a problem with binutils here. The code to add those flags at compile time has been accepted upstream back in October, but just too late for the 2.23 release. It also wasn't merged for the 2.23.1 release. Apologies - looks like I didn't leave enough time for things to sync before glibc changes went in (in December). All my testing used the latest trunk code from both codebases. Arse. :-(

See the ARM-related binutils commits in http://sourceware.org/git/?p=binutils.git;a=commit;h=cd49363b8ae1725522ea29ccc0dda7b138f3d5aa and http://sourceware.org/git/?p=binutils.git;a=commit;h=c7d345740f2244e15e04030938b7e4f79e2902af for the changes if you want a quick fix.

Joseph - do you have any suggestions on what we could/should do to fix this please?
Comment 21 Kevin Mihelich 2013-01-23 20:08:17 UTC
Should there not be a way to accept the way binaries and libraries are built now, along with the correct hard/soft flag going forward?
Comment 22 joseph@codesourcery.com 2013-01-23 20:31:07 UTC
If programs / libraries are built using old binutils, they should have 
neither of the flags EF_ARM_ABI_FLOAT_SOFT and EF_ARM_ABI_FLOAT_HARD set.  
In that case (both old binaries, and new binaries built with old 
binutils), I'd think ld.so / ldconfig should treat them as unknown-ABI and 
loadable by both hard-float and soft-float ld.so.  So this suggests to me 
that there is something wrong with the logic in glibc for handling such 
binaries without ABI markers.
Comment 23 Carlos O'Donell 2013-01-28 20:02:03 UTC
Working on this.
Comment 24 Carlos O'Donell 2013-02-06 19:51:48 UTC
Testing patch.
Comment 25 Carlos O'Donell 2013-02-07 05:14:11 UTC
Patch sent to libc-alpha/libc-ports:
http://sourceware.org/ml/libc-alpha/2013-02/msg00120.html
Comment 26 Carlos O'Donell 2013-02-08 17:33:25 UTC
Checked into master, closing this bug report.

Backport bug here: http://sourceware.org/bugzilla/show_bug.cgi?id=15122

commit b39949d21179bd0b08a083e4f7e35d0c98e11294
Author: Carlos O'Donell <carlos@redhat.com>
Date:   Fri Feb 8 12:26:12 2013 -0500

    ARM: Support loading unmarked objects from cache.
    
    ARM now supports loading unmarked objects from
    the dynamic loader cache. Unmarked objects can
    be used with the hard-float or soft-float ABI.
    We must support loading unmarked objects during
    the transition period from a binutils that does
    not mark objects to one that does mark them with
    the correct ELF flags.
    
    Signed-off-by: Carlos O'Donell <carlos@redhat.com>
Comment 27 Jackie Rosen 2014-02-16 19:41:27 UTC Comment hidden (spam)