Bug 24723 - Linker fails with "too many open files"
Summary: Linker fails with "too many open files"
Status: RESOLVED DUPLICATE of bug 28138
Alias: None
Product: binutils
Classification: Unclassified
Component: ld (show other bugs)
Version: 2.32
: P2 normal
Target Milestone: ---
Assignee: Not yet assigned to anyone
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2019-06-23 20:42 UTC by Benjamin Redelings
Modified: 2021-08-18 13:17 UTC (History)
3 users (show)

See Also:
Host:
Target:
Build:
Last reconfirmed: 2019-06-24 00:00:00


Attachments
An strace log of the linker running out of open files. (233.54 KB, application/gzip)
2019-06-23 20:44 UTC, Benjamin Redelings
Details
Example containing also the boot.o object file (70.61 KB, application/x-zip-compressed)
2019-06-27 14:01 UTC, lorisnardo
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Benjamin Redelings 2019-06-23 20:42:36 UTC
When trying to link a large program, the linker fails with "Too many open files".
This occurs with binutils 2.32, so the previous similar bugreport with 2.31 may not apply (#23573).

/usr/bin/x86_64-w64-mingw32-g++  -o rb.exe 'rb@exe/src_revlanguage_main.cpp.obj' -L/home/bredelings/win_root/mingw64/lib -Wl,-O1 -Wl,--start-group librb-core.a librb-revlanguage.a librb-lib.a -lboost_regex-mt -lboost_program_options-mt -lboost_thread-mt -lboost_system-mt -lboost_filesystem-mt -lboost_date_time-mt -lboost_serialization-mt -pthread -mconsole -lkernel32 -luser32 -lgdi32 -lwinspool -lshell32 -lole32 -loleaut32 -luuid -lcomdlg32 -ladvapi32 -Wl,--end-group
/usr/bin/x86_64-w64-mingw32-ld: cannot find -lstdc++
/usr/bin/x86_64-w64-mingw32-ld: cannot find -lmingw32
/usr/bin/x86_64-w64-mingw32-ld: cannot find -lgcc_s
/usr/bin/x86_64-w64-mingw32-ld: cannot find -lgcc
/usr/bin/x86_64-w64-mingw32-ld: cannot find -lmoldname
/usr/bin/x86_64-w64-mingw32-ld: cannot find -lmingwex
/usr/bin/x86_64-w64-mingw32-ld: cannot find -lmsvcrt
/usr/bin/x86_64-w64-mingw32-ld: cannot find -lpthread
/usr/bin/x86_64-w64-mingw32-ld: cannot find -ladvapi32
/usr/bin/x86_64-w64-mingw32-ld: cannot find -lshell32
/usr/bin/x86_64-w64-mingw32-ld: cannot find -luser32
/usr/bin/x86_64-w64-mingw32-ld: cannot find -lkernel32
/usr/bin/x86_64-w64-mingw32-ld: cannot find -lmingw32
/usr/bin/x86_64-w64-mingw32-ld: cannot find -lgcc_s
/usr/bin/x86_64-w64-mingw32-ld: cannot find -lgcc
/usr/bin/x86_64-w64-mingw32-ld: cannot find -lmoldname
/usr/bin/x86_64-w64-mingw32-ld: cannot find -lmingwex
/usr/bin/x86_64-w64-mingw32-ld: cannot find -lmsvcrt
/usr/bin/x86_64-w64-mingw32-ld: cannot find /usr/lib/gcc/x86_64-w64-mingw32/8.3-win32/crtend.o: Too many open files
collect2: error: ld returned 1 exit status

Versions look like this:

$ uname -a
Linux name 5.0.0-trunk-amd64 #1 SMP Debian 5.0.2-1~exp1 (2019-03-18) x86_64 GNU/Linux

$ /usr/bin/x86_64-w64-mingw32-g++ -v
Using built-in specs.
COLLECT_GCC=/usr/bin/x86_64-w64-mingw32-g++
COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-w64-mingw32/8.3-win32/lto-wrapper
Target: x86_64-w64-mingw32
Configured with: ../../src/configure --build=x86_64-linux-gnu --prefix=/usr --includedir='/usr/include' --mandir='/usr/share/man' --infodir='/usr/share/info' --sysconfdir=/etc --localstatedir=/var --disable-silent-rules --libdir='/usr/lib/x86_64-linux-gnu' --libexecdir='/usr/lib/x86_64-linux-gnu' --disable-maintainer-mode --disable-dependency-tracking --prefix=/usr --enable-shared --enable-static --disable-multilib --with-system-zlib --libexecdir=/usr/lib --without-included-gettext --libdir=/usr/lib --enable-libstdcxx-time=yes --with-tune=generic --with-headers=/usr/x86_64-w64-mingw32/include --enable-version-specific-runtime-libs --enable-fully-dynamic-string --enable-libgomp --enable-languages=c,c++,fortran,objc,obj-c++,ada --enable-lto --enable-threads=win32 --program-suffix=-win32 --program-prefix=x86_64-w64-mingw32- --target=x86_64-w64-mingw32 --with-as=/usr/bin/x86_64-w64-mingw32-as --with-ld=/usr/bin/x86_64-w64-mingw32-ld --enable-libatomic --enable-libstdcxx-filesystem-ts=yes
Thread model: win32
gcc version 8.3-win32 20190428 (GCC) 

$ /usr/bin/x86_64-w64-mingw32-ld --version
GNU ld (GNU Binutils) 2.32
Copyright (C) 2019 Free Software Foundation, Inc.
This program is free software; you may redistribute it under the terms of
the GNU General Public License version 3 or (at your option) a later version.
This program has absolutely no warranty.

The ld command-line that is failing looks like this:
/usr/bin/x86_64-w64-mingw32-ld -plugin /usr/lib/gcc/x86_64-w64-mingw32/8.3-win32/liblto_plugin.so -plugin-opt=/usr/lib/gcc/x86_64-w64-mingw32/8.3-win32/lto-wrapper -plugin-opt=-fresolution=/tmp/ccXUMZS3.res -plugin-opt=-pass-through=-lmingw32 -plugin-opt=-pass-through=-lgcc_s -plugin-opt=-pass-through=-lgcc -plugin-opt=-pass-through=-lmoldname -plugin-opt=-pass-through=-lmingwex -plugin-opt=-pass-through=-lmsvcrt -plugin-opt=-pass-through=-lpthread -plugin-opt=-pass-through=-ladvapi32 -plugin-opt=-pass-through=-lshell32 -plugin-opt=-pass-through=-luser32 -plugin-opt=-pass-through=-lkernel32 -plugin-opt=-pass-through=-lmingw32 -plugin-opt=-pass-through=-lgcc_s -plugin-opt=-pass-through=-lgcc -plugin-opt=-pass-through=-lmoldname -plugin-opt=-pass-through=-lmingwex -plugin-opt=-pass-through=-lmsvcrt -m i386pep --subsystem console -Bdynamic -o rb.exe /usr/lib/gcc/x86_64-w64-mingw32/8.3-win32/../../../../x86_64-w64-mingw32/lib/crt2.o /usr/lib/gcc/x86_64-w64-mingw32/8.3-win32/crtbegin.o -L/home/bredelings/win_root/mingw64/lib -L/usr/lib/gcc/x86_64-w64-mingw32/8.3-win32 -L/usr/lib/gcc/x86_64-w64-mingw32/8.3-win32/../../../../x86_64-w64-mingw32/lib rb@exe/src_revlanguage_main.cpp.obj -O1 --start-group librb-core.a librb-revlanguage.a librb-lib.a -lboost_regex-mt -lboost_program_options-mt -lboost_thread-mt -lboost_system-mt -lboost_filesystem-mt -lboost_date_time-mt -lboost_serialization-mt -lkernel32 -luser32 -lgdi32 -lwinspool -lshell32 -lole32 -loleaut32 -luuid -lcomdlg32 -ladvapi32 --end-group -lstdc++ -lmingw32 -lgcc_s -lgcc -lmoldname -lmingwex -lmsvcrt -lpthread -ladvapi32 -lshell32 -luser32 -lkernel32 -lmingw32 -lgcc_s -lgcc -lmoldname -lmingwex -lmsvcrt /usr/lib/gcc/x86_64-w64-mingw32/8.3-win32/crtend.o

$ dpkg -s binutils-mingw-w64-x86-64
Package: binutils-mingw-w64-x86-64
Status: install ok installed
Priority: optional
Section: devel
Installed-Size: 23637
Maintainer: Stephen Kitt <skitt@debian.org>
Architecture: amd64
Source: binutils-mingw-w64 (8.4~exp1)
Version: 2.32-8+8.4~exp1
Replaces: binutils-mingw-w64 (<< 2.22-1+1)
Depends: libc6 (>= 2.14), zlib1g (>= 1:1.2.0), binutils-common (>= 2.32)
Breaks: binutils-mingw-w64 (<< 2.22-1+1)
Description: Cross-binutils for Win64 (x64) using MinGW-w64
 MinGW-w64 provides a development and runtime environment for 32- and
 64-bit (x86 and x64) Windows applications using the Windows API and
 the GNU Compiler Collection (gcc).
 .
 This package contains the toolchain binutils targeting 64-bit Windows.
Built-Using: binutils (= 2.32-8)
Homepage: https://www.gnu.org/software/binutils/

$ dpkg -s g++-mingw-w64-x86-64
Package: g++-mingw-w64-x86-64
Status: install ok installed
Priority: optional
Section: devel
Installed-Size: 131359
Maintainer: Stephen Kitt <skitt@debian.org>
Architecture: amd64
Source: gcc-mingw-w64 (21.3)
Version: 8.3.0-7+21.3
Replaces: gcc-mingw-w64 (<< 4.6.3-3+4)
Depends: gcc-mingw-w64-x86-64 (= 8.3.0-7+21.3), gcc-mingw-w64-base (= 8.3.0-7+21.3), libc6 (>= 2.14), libgcc1 (>= 1:4.2), libgmp10, libisl19 (>= 0.15), libmpc3, libmpfr6 (>= 3.1.3), libstdc++6 (>= 5.2), zlib1g (>= 1:1.1.4)
Suggests: gcc-8-locales (>= 8.3.0)
Breaks: gcc-mingw-w64 (<< 4.6.3-3+4)
Description: GNU C++ compiler for MinGW-w64 targeting Win64
 MinGW-w64 provides a development and runtime environment for 32- and
 64-bit (x86 and x64) Windows applications using the Windows API and
 the GNU Compiler Collection (gcc).
 .
 This package contains the C++ compiler, supporting
 cross-compiling to 64-bit MinGW-w64 targets.
Built-Using: gcc-8 (= 8.3.0-7)
Homepage: https://www.gnu.org/software/gcc/
Comment 1 Benjamin Redelings 2019-06-23 20:44:43 UTC
Created attachment 11860 [details]
An strace log of the linker running out of open files.
Comment 2 Alan Modra 2019-06-24 07:12:41 UTC
Thousands of files are handled fine by the linker when plugins aren't involved.  See bfd/cache.c.  This comment from bfd/plugin.c explains why there is a problem when using plugins:

  /* The plugin API expects that the file descriptor won't be closed
     and reused as done by the bfd file cache.  So open it again.
     dup isn't good enough.  plugin IO uses lseek/read while BFD uses
     fseek/fread.  It isn't wise to mix the unistd and stdio calls on
     the same underlying file descriptor.  */

Until the gcc plugin API is fixed there isn't much the linker can do about running out of file descriptors.
Comment 3 lorisnardo 2019-06-27 14:01:23 UTC
Created attachment 11870 [details]
Example containing also the boot.o object file
Comment 4 lorisnardo 2019-06-27 14:03:58 UTC
Comment on attachment 11870 [details]
Example containing also the boot.o object file

Please do not consider/ remove this attachment it was inteded for another bug report(#23839)
Comment 6 Christoph Reiter 2021-08-15 11:45:41 UTC
We are seeing a similar error with 2.37: https://github.com/msys2/MINGW-packages/pull/9374

Linking CXX shared library ../../bin/Qt5WebKit.dll[0m
2021-08-15T11:12:27.6836529Z D:/a/_temp/msys/msys64/mingw64/lib/gcc/x86_64-w64-mingw32/10.3.0/../../../../x86_64-w64-mingw32/bin/ld.exe: cannot find D:/a/_temp/msys/msys64/mingw64/lib/libQt5Quick.dll.a: Too many open files
2021-08-15T11:12:27.6838502Z D:/a/_temp/msys/msys64/mingw64/lib/gcc/x86_64-w64-mingw32/10.3.0/../../../../x86_64-w64-mingw32/bin/ld.exe: cannot find D:/a/_temp/msys/msys64/mingw64/lib/libQt5WebChannel.dll.a: Too many open files
2021-08-15T11:12:27.6845233Z D:/a/_temp/msys/msys64/mingw64/lib/gcc/x86_64-w64-mingw32/10.3.0/../../../../x86_64-w64-mingw32/bin/ld.exe: cannot find -licuuc
....
Comment 7 Christoph Reiter 2021-08-15 11:47:19 UTC
There are also a lot of "plugin framework: out of file descriptors. Try using fewer objects/archives" before that.

Any ideas how to work around this welcome.
Comment 8 Alan Modra 2021-08-16 23:21:27 UTC
I believe this has been fixed on the git binutils-2_37-branch
Comment 9 Christoph Reiter 2021-08-17 18:42:31 UTC
Thanks, I can confirm that https://sourceware.org/git/?p=binutils-gdb.git;a=commit;h=1c611b40e6bfc8029bff7696814330b5bc0ee5c0 fixes my issue.

Sorry for the noise.
Comment 10 H.J. Lu 2021-08-18 13:17:10 UTC
Dup

*** This bug has been marked as a duplicate of bug 28138 ***