We need a complete reproducer for this issue please.

Include exactly the steps you took and the expected versus observed results.

This includes the exact gcc command you used to compile your application, and exactly what version of glibc you were using.

If you are using a distro glibc then you need to file this issue upstream with the distro so they can determine if any of their patches caused the problem.

Created attachment 7085 [details]
multiple definition of `__lll_lock_wait_private'  example

OS version : Ubuntu 12.04
gcc version : gcc version 4.6.3 (Ubuntu/Linaro 4.6.3-1ubuntu5)


untar the tarball, and use make to reproduce the bug.

I expect there is no error message.
But I get error message when linking.

compiler options "-ftree-parallelize-loops=2 -flto -O1" are necessary.

gcc -ftree-parallelize-loops=2 -flto -O1 core_list_join.i -c -o core_list_join.o
gcc -ftree-parallelize-loops=2 -flto -O1 core_main.i -c -o core_main.o
gcc -ftree-parallelize-loops=2 -flto -O1 core_matrix.i -c -o core_matrix.o
gcc -ftree-parallelize-loops=2 -flto -O1 core_state.i -c -o core_state.o
gcc -ftree-parallelize-loops=2 -flto -O1 core_util.i -c -o core_util.o
gcc -ftree-parallelize-loops=2 -flto -O1 core_portme.i -c -o core_portme.o
gcc -ftree-parallelize-loops=2 -flto -O1 *.o -static -o coremark.exe
/usr/lib/gcc/i686-linux-gnu/4.6/../../../i386-linux-gnu/libpthread.a(lowlevellock.o): In function `__lll_lock_wait_private':
/build/buildd/eglibc-2.15/nptl/../nptl/sysdeps/unix/sysv/linux/i386/i686/../i486/lowlevellock.S:76: multiple definition of `__lll_lock_wait_private'
/usr/lib/gcc/i686-linux-gnu/4.6/../../../i386-linux-gnu/libc.a(libc-lowlevellock.o):(.text+0x0): first defined here
/usr/lib/gcc/i686-linux-gnu/4.6/../../../i386-linux-gnu/libpthread.a(lowlevellock.o): In function `__lll_unlock_wake_private':
/build/buildd/eglibc-2.15/nptl/../nptl/sysdeps/unix/sysv/linux/i386/i686/../i486/lowlevellock.S:332: multiple definition of `__lll_unlock_wake_private'
/usr/lib/gcc/i686-linux-gnu/4.6/../../../i386-linux-gnu/libc.a(libc-lowlevellock.o):(.text+0x30): first defined here
collect2: ld returned 1 exit status
make: *** [all] Error 1

I also try linaro gcc 4.8 toolchain

http://releases.linaro.org/13.05/components/toolchain/binaries/gcc-linaro-arm-linux-gnueabihf-4.8-2013.05_linux.tar.bz2

the same situation

/work/linaro_tools/gcc-linaro-arm-linux-gnueabihf-4.8-2013.04-20130417_linux/bin/arm-linux-gnueabihf-gcc -ftree-parallelize-loops=2 -flto -O1 core_list_join.i -c -o core_list_join.o
/work/linaro_tools/gcc-linaro-arm-linux-gnueabihf-4.8-2013.04-20130417_linux/bin/arm-linux-gnueabihf-gcc -ftree-parallelize-loops=2 -flto -O1 core_main.i -c -o core_main.o
/work/linaro_tools/gcc-linaro-arm-linux-gnueabihf-4.8-2013.04-20130417_linux/bin/arm-linux-gnueabihf-gcc -ftree-parallelize-loops=2 -flto -O1 core_matrix.i -c -o core_matrix.o
/work/linaro_tools/gcc-linaro-arm-linux-gnueabihf-4.8-2013.04-20130417_linux/bin/arm-linux-gnueabihf-gcc -ftree-parallelize-loops=2 -flto -O1 core_state.i -c -o core_state.o
/work/linaro_tools/gcc-linaro-arm-linux-gnueabihf-4.8-2013.04-20130417_linux/bin/arm-linux-gnueabihf-gcc -ftree-parallelize-loops=2 -flto -O1 core_util.i -c -o core_util.o
/work/linaro_tools/gcc-linaro-arm-linux-gnueabihf-4.8-2013.04-20130417_linux/bin/arm-linux-gnueabihf-gcc -ftree-parallelize-loops=2 -flto -O1 core_portme.i -c -o core_portme.o
/work/linaro_tools/gcc-linaro-arm-linux-gnueabihf-4.8-2013.04-20130417_linux/bin/arm-linux-gnueabihf-gcc -ftree-parallelize-loops=2 -flto -O1 *.o -static -o coremark.exe
/work/linaro_tools/gcc-linaro-arm-linux-gnueabihf-4.8-2013.04-20130417_linux/bin/../arm-linux-gnueabihf/libc/usr/lib/arm-linux-gnueabihf/libpthread.a(lowlevellock.o): In function `__lll_lock_wait_private':
lowlevellock.c:(.text+0x0): multiple definition of `__lll_lock_wait_private'
/home/neilh/work/linaro_tools/gcc-linaro-arm-linux-gnueabihf-4.8-2013.04-20130417_linux/bin/../arm-linux-gnueabihf/libc/usr/lib/arm-linux-gnueabihf/libc.a(libc-lowlevellock.o):(.text+0x0): first defined here
collect2: error: ld returned 1 exit status
make: *** [all] Error 1

Hope these information are enough to reproduce the issue.

Regards,
Neil

This doesn't work at all for me for entirely different reasons.

I expect that you haven't actually tried to use -flto.

glibc master is currently unbuildable with -flto because of:
http://sourceware.org/bugzilla/show_bug.cgi?id=15648

You should always see:
checking for .preinit_array/.init_array/.fini_array support... no
configure: error: Need linker with .init_array/.fini_array support.

This is because -flto throws out the .init_array support.

A potential fix is:
diff --git a/configure.in b/configure.in
index 8b11081..415e777 100644
--- a/configure.in
+++ b/configure.in
@@ -1383,7 +1383,7 @@ AC_CACHE_CHECK(for .preinit_array/.init_array/.fini_array support,
               libc_cv_initfini_array, [dnl
 LIBC_TRY_LINK_STATIC([
 int foo (void) { return 1; }
-int (*fp) (void) __attribute__ ((section (".init_array"))) = foo;
+int (*fp) (void) __attribute__ ((section (".init_array"), used)) = foo;
 ],
   [if $READELF -S conftest | fgrep INIT_ARRAY > /dev/null; then
     libc_cv_initfini_array=yes
~~~

However, given that nobody has built with -flto it's going to require considerable work to make it happen.

I'm retitling and marking as enhancement.

(In reply to Carlos O'Donell from comment #4)
> glibc master is currently unbuildable with -flto because of:
> http://sourceware.org/bugzilla/show_bug.cgi?id=15648

Should have been:

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=51255

Created attachment 7086 [details]
multiple definition of `__lll_lock_wait_private' example2

I don't want to build glibc with -flto option.
I build application with -flto.

The problem is like building thread program which static link with -lc -lpthread

For example:
gcc thread_program.c -lc -lpthread -static
/usr/lib/gcc/i686-linux-gnu/4.6/../../../i386-linux-gnu/libpthread.a(lowlevellock.o): In function `__lll_lock_wait_private':
/build/buildd/eglibc-2.15/nptl/../nptl/sysdeps/unix/sysv/linux/i386/i686/../i486/lowlevellock.S:76: multiple definition of `__lll_lock_wait_private'
/usr/lib/gcc/i686-linux-gnu/4.6/../../../i386-linux-gnu/libc.a(libc-lowlevellock.o):(.text+0x0): first defined here
/usr/lib/gcc/i686-linux-gnu/4.6/../../../i386-linux-gnu/libpthread.a(lowlevellock.o): In function `__lll_unlock_wake_private':
/build/buildd/eglibc-2.15/nptl/../nptl/sysdeps/unix/sysv/linux/i386/i686/../i486/lowlevellock.S:332: multiple definition of `__lll_unlock_wake_private'
/usr/lib/gcc/i686-linux-gnu/4.6/../../../i386-linux-gnu/libc.a(libc-lowlevellock.o):(.text+0x30): first defined here
collect2: ld returned 1 exit status

But, I don't know why -flto causes the problem for my first example. :(

This has nothing to do with building glibc with -flto.  There are simply two defintions of __lll_lock_wait_private and __lll_unlock_wait_private, one in libpthread.a(lowlevellock.o) and another one in libc.a(libc-lowlevellock.o).  One of them needs to be elided from the static library.

Yes, I see the libc.a and libpthread.a

nm /usr/lib/i386-linux-gnu/libpthread.a
lowlevellock.o:
         U __have_futex_clock_realtime
00000030 T __lll_lock_wait
00000000 T __lll_lock_wait_private
00000060 T __lll_timedlock_wait
000001c0 T __lll_timedwait_tid
00000190 T __lll_unlock_wake
00000160 T __lll_unlock_wake_private
         U _dl_sysinfo

nm /usr/lib/i386-linux-gnu/libc.a
libc-lowlevellock.o:
00000000 T __lll_lock_wait_private
00000030 T __lll_unlock_wake_private
         U _dl_sysinfo

Why not to make __lll_lock_wait_private and __lll_unlock_wake_private in a individual file?

If ld include libc-lowlevellock.o first, it always cause error when includeing lowlevellock.o. Right?

(In reply to Neil from comment #6)
> Created attachment 7086 [details]
> multiple definition of `__lll_lock_wait_private' example2
> 
> I don't want to build glibc with -flto option.
> I build application with -flto.

Thanks, I misunderstood your problem.

(In reply to Carlos O'Donell from comment #9)
> (In reply to Neil from comment #6)
> > Created attachment 7086 [details]
> > multiple definition of `__lll_lock_wait_private' example2
> > 
> > I don't want to build glibc with -flto option.
> > I build application with -flto.
> 
> Thanks, I misunderstood your problem.

Filed:
Bug 15658 - Support building glibc with -flto.
http://sourceware.org/bugzilla/show_bug.cgi?id=15658

(In reply to Neil from comment #8)
> If ld include libc-lowlevellock.o first, it always cause error when
> includeing lowlevellock.o. Right?

But why does that happen?
You link -lc before -lpthread?  That is never going to work properly when linking the libraries statically.

(In reply to Jakub Jelinek from comment #11)
> (In reply to Neil from comment #8)
> > If ld include libc-lowlevellock.o first, it always cause error when
> > includeing lowlevellock.o. Right?
> 
> But why does that happen?
> You link -lc before -lpthread?  That is never going to work properly when
> linking the libraries statically.

A `gcc -v...` would help show what's going on here.

Neil, can you give us the verbose output of the compile?

(In reply to Jakub Jelinek from comment #11)
> (In reply to Neil from comment #8)
> > If ld include libc-lowlevellock.o first, it always cause error when
> > includeing lowlevellock.o. Right?
> 
> But why does that happen?
> You link -lc before -lpthread?  That is never going to work properly when
> linking the libraries statically.

Exactly, statically link -c before -pthread never works well.

I mean I encounter the similar problem like that but I don't link -c before -pthread.

I use gcc4.6 and the option "-ftree-parallelize-loops=2 -flto -O1" to compile program.
Because of option "-ftree-parallelize-loops=2", gcc will auto apply -lpthread to linker.
Then the error message show up.

In this case, I'm not sure this is gcc issue or glibc issue.
But I modify glibc source, the issue can be resolved.

Created attachment 7088 [details]
link error message

I add -v and -Wl,--verbose.
There are more information.
Please see the attachment file.

Following is linking information.

gcc -ftree-parallelize-loops=2 -flto -O1 -save-temps *.o -static -o coremark.exe -v -Wl,--verbose
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/lib/gcc/i686-linux-gnu/4.6/lto-wrapper
Target: i686-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Ubuntu/Linaro 4.6.3-1ubuntu5' --with-bugurl=file:///usr/share/doc/gcc-4.6/README.Bugs --enable-languages=c,c++,fortran,objc,obj-c++ --prefix=/usr --program-suffix=-4.6 --enable-shared --enable-linker-build-id --with-system-zlib --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --with-gxx-include-dir=/usr/include/c++/4.6 --libdir=/usr/lib --enable-nls --with-sysroot=/ --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-gnu-unique-object --enable-plugin --enable-objc-gc --enable-targets=all --disable-werror --with-arch-32=i686 --with-tune=generic --enable-checking=release --build=i686-linux-gnu --host=i686-linux-gnu --target=i686-linux-gnu
Thread model: posix
gcc version 4.6.3 (Ubuntu/Linaro 4.6.3-1ubuntu5)
COMPILER_PATH=/usr/lib/gcc/i686-linux-gnu/4.6/:/usr/lib/gcc/i686-linux-gnu/4.6/:/usr/lib/gcc/i686-linux-gnu/:/usr/lib/gcc/i686-linux-gnu/4.6/:/usr/lib/gcc/i686-linux-gnu/
LIBRARY_PATH=/usr/lib/gcc/i686-linux-gnu/4.6/:/usr/lib/gcc/i686-linux-gnu/4.6/../../../i386-linux-gnu/:/usr/lib/gcc/i686-linux-gnu/4.6/../../../../lib/:/lib/i386-linux-gnu/:/lib/../lib/:/usr/lib/i386-linux-gnu/:/usr/lib/../lib/:/usr/lib/gcc/i686-linux-gnu/4.6/../../../:/lib/:/usr/lib/
Reading specs from /usr/lib/gcc/i686-linux-gnu/4.6/libgomp.spec
COLLECT_GCC_OPTIONS='-ftree-parallelize-loops=2' '-flto' '-O1' '-save-temps' '-static' '-o' 'coremark.exe' '-v' '-mtune=generic' '-march=i686' '-pthread'
 /usr/lib/gcc/i686-linux-gnu/4.6/collect2 -plugin /usr/lib/gcc/i686-linux-gnu/4.6/liblto_plugin.so -plugin-opt=/usr/lib/gcc/i686-linux-gnu/4.6/lto-wrapper -plugin-opt=-fresolution=--verbose.res -plugin-opt=-pass-through=-lgcc -plugin-opt=-pass-through=-lgcc_eh -plugin-opt=-pass-through=-lpthread -plugin-opt=-pass-through=-lc -flto --sysroot=/ --build-id --no-add-needed --as-needed -m elf_i386 --hash-style=gnu -static -z relro -o coremark.exe /usr/lib/gcc/i686-linux-gnu/4.6/../../../i386-linux-gnu/crt1.o /usr/lib/gcc/i686-linux-gnu/4.6/../../../i386-linux-gnu/crti.o /usr/lib/gcc/i686-linux-gnu/4.6/crtbeginT.o -L/usr/lib/gcc/i686-linux-gnu/4.6 -L/usr/lib/gcc/i686-linux-gnu/4.6/../../../i386-linux-gnu -L/usr/lib/gcc/i686-linux-gnu/4.6/../../../../lib -L/lib/i386-linux-gnu -L/lib/../lib -L/usr/lib/i386-linux-gnu -L/usr/lib/../lib -L/usr/lib/gcc/i686-linux-gnu/4.6/../../.. core_list_join.o core_main.o coremark.exe.ltrans0.o core_matrix.o core_portme.o core_state.o core_util.o --verbose -lgomp -lrt --start-group -lgcc -lgcc_eh -lpthread -lc --end-group /usr/lib/gcc/i686-linux-gnu/4.6/crtend.o /usr/lib/gcc/i686-linux-gnu/4.6/../../../i386-linux-gnu/crtn.o

Hope this will help you.
Thanks,
Neil

The problem is that even though libpthread.a(lowlevellock.o) would resolve the reference to __lll_lock_wait_private and/or __lll_unlock_wake_private the first reference is from libc.a, and libpthread.a will only be rescanned after libc.a cannot resolve any more references.

Static linking of -lpthread never worked well, there are tons of issues with that, which is why in Fedora/RHEL we are just linking all the content of libpthread.a using ld -r into libpthread.o and libpthread.a contains just that single object.  Then this bug doesn't exist.  There are many assumptions (unwritten dependencies) in between libpthread object files, with dynamic linking you simply have all of them together or nothing, but with libpthread, unless you do that ld -r hack, you don't.  Not to mention that many programs (e.g. gcc gthr.h) check using weakref for availability of some pthread_* symbol(s) and if non-NULL, assume all of libpthread is available, rather than random parts from it.

Why exactly are you linking statically?  http://www.akkadia.org/drepper/no_static_linking.html

Created attachment 7089 [details]
Make __lll_lock_wait_private and __lll_unlock_wake_private weak in libc.

Could you test the following patch to see if it fixes the issue for you?

http://sourceware.org/glibc/wiki/Testing/Builds

(In reply to Jakub Jelinek from comment #16)
> Static linking of -lpthread never worked well, there are tons of issues with
> that, which is why in Fedora/RHEL we are just linking all the content of
> libpthread.a using ld -r into libpthread.o and libpthread.a contains just
> that single object.  Then this bug doesn't exist.  There are many
> assumptions (unwritten dependencies) in between libpthread object files,
> with dynamic linking you simply have all of them together or nothing, but
> with libpthread, unless you do that ld -r hack, you don't.  Not to mention
> that many programs (e.g. gcc gthr.h) check using weakref for availability of
> some pthread_* symbol(s) and if non-NULL, assume all of libpthread is
> available, rather than random parts from it.

Then we need to file a bug to fix glibc to use ld -r or gather community consensus that static linking is no longer supported. Current community consensus is that we will continue to support static linking where possible.

> Why exactly are you linking statically? 
> http://www.akkadia.org/drepper/no_static_linking.html

Good question, but it doesn't answer the more fundamental question of glibc support for static linking.

(In reply to Carlos O'Donell from comment #17)
> Created attachment 7089 [details]
> Make __lll_lock_wait_private and __lll_unlock_wake_private weak in libc.
> 
> Could you test the following patch to see if it fixes the issue for you?
> 
> http://sourceware.org/glibc/wiki/Testing/Builds

That's awesome!

This patch fixes the issue for me.

Is this patch OK for the trunk?

Carlos, is your patch ready to libc-alpha?

(In reply to Ondrej Bilka from comment #20)
> Carlos, is your patch ready to libc-alpha?

No. This issue needs to be driven to consensus over the solution to use for libpthread, either we link with `ld -r' and force all of libpthread into the application during a static link or we do it piece-meal dependency by dependency trying to keep the link minimal.

Previous comments from Roland and myself was that we should try to fix these kinds of issues one at a time, but it's looking liked `ld -r' is going to be the only real solution given the number of applications and core tools that use a weakref and a check to test for conditional library inclusion.

This has to go to libca-alpha for discussion, but I don't have time to champion this right now. If you are interested I encourage you to get consensus and implement a solution, I'd be happy to review.

This should be fixed in glibc 2.34 by:

commit 2f4019de81024cc0d05a5b5cedb1afb28b59a836
Author: Florian Weimer <fweimer@redhat.com>
Date:   Mon Feb 22 18:30:52 2021 +0100

    nptl: Move lowlevellock into libc [BZ #15648]
    
    And export __lll_lock_wait and __lll_lock_wait_private as
    GLIBC_PRIVATE.  This should eliminate the need for the previous
    riscv kludge.

Similar issues for different symbols may still exist.