I'm getting "Too many open files" when building LLVM/clang (release_60 branch) with the most recent version of binutils (2.31.1-1). Running "ulimit -n 3200" did not help. Downgrading to 2.30-2 solved the problem (without even needing to tweak the limit, which defaults to 256). PS: I stumbled upon this issue with MinGW, but apparently it is likely the case for non-MinGW systems as well; see the original discussion thread below: https://github.com/Alexpux/MINGW-packages/issues/4317
"building LLVM/clang" involves quite a large number of steps. What command gave you "Too many open files"? This bug may well be a dup of https://sourceware.org/bugzilla/show_bug.cgi?id=23460
IIRC, it was during the final linking stages of the build, so it could have been due to 'ar', as pointed in #23460. I can confirm that tomorrow by noon (PST).
Actually, in my case, the bug seems to come from 'ld': T:/Tools/MSYS2/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/8.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: cannot find ../../../../lib/libclangBasic.a: Too many open files T:/Tools/MSYS2/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/8.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: cannot find ../../../../lib/libLLVMCore.a: Too many open files T:/Tools/MSYS2/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/8.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: cannot find ../../../../lib/libLLVMMC.a: Too many open files T:/Tools/MSYS2/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/8.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: cannot find ../../../../lib/libLLVMBinaryFormat.a: Too many open files T:/Tools/MSYS2/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/8.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: cannot find ../../../../lib/libLLVMDebugInfoCodeView.a: Too many open files T:/Tools/MSYS2/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/8.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: cannot find ../../../../lib/libLLVMDebugInfoMSF.a: Too many open files T:/Tools/MSYS2/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/8.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: cannot find ../../../../lib/libLLVMSupport.a: Too many open files
In addition, reverting to 2.30-4 also works.
Please check if it is caused by commit 27b0767593284f97384b3597ebd211164f8c8b47 Author: Alan Modra <amodra@gmail.com> Date: Tue Jun 5 21:04:00 2018 +0930 PR23254, ld.bfd mishandles file pointers while scanning archive Best practice is to not mix lseek/read with fseek/fread on the same underlying file descriptor, as not all stdio implementations will cope. Since the plugin uses lseek/read while bfd uses fseek/fread this patch reopens the file for exclusive use by the plugin rather than trying to restore the file descriptor. That allows the plugin to read the file after plugin_call_claim_file too.
I suspect this bug has already been fixed on master.
In that case, it should be landing on the upcoming 2.32 release, correct?
Yes 2.32 should have this bug fixed.
I got this bug with binutils 2.32, so I do not think this is fixed. /usr/bin/x86_64-w64-mingw32-g++ -o rb.exe 'rb@exe/src_revlanguage_main.cpp.obj' -L/home/bredelings/win_root/mingw64/lib -Wl,-O1 -Wl,--start-group librb-core.a librb-revlanguage.a librb-lib.a -lboost_regex-mt -lboost_program_options-mt -lboost_thread-mt -lboost_system-mt -lboost_filesystem-mt -lboost_date_time-mt -lboost_serialization-mt -pthread -mconsole -lkernel32 -luser32 -lgdi32 -lwinspool -lshell32 -lole32 -loleaut32 -luuid -lcomdlg32 -ladvapi32 -Wl,--end-group /usr/bin/x86_64-w64-mingw32-ld: cannot find -lstdc++ /usr/bin/x86_64-w64-mingw32-ld: cannot find -lmingw32 /usr/bin/x86_64-w64-mingw32-ld: cannot find -lgcc_s /usr/bin/x86_64-w64-mingw32-ld: cannot find -lgcc /usr/bin/x86_64-w64-mingw32-ld: cannot find -lmoldname /usr/bin/x86_64-w64-mingw32-ld: cannot find -lmingwex /usr/bin/x86_64-w64-mingw32-ld: cannot find -lmsvcrt /usr/bin/x86_64-w64-mingw32-ld: cannot find -lpthread /usr/bin/x86_64-w64-mingw32-ld: cannot find -ladvapi32 /usr/bin/x86_64-w64-mingw32-ld: cannot find -lshell32 /usr/bin/x86_64-w64-mingw32-ld: cannot find -luser32 /usr/bin/x86_64-w64-mingw32-ld: cannot find -lkernel32 /usr/bin/x86_64-w64-mingw32-ld: cannot find -lmingw32 /usr/bin/x86_64-w64-mingw32-ld: cannot find -lgcc_s /usr/bin/x86_64-w64-mingw32-ld: cannot find -lgcc /usr/bin/x86_64-w64-mingw32-ld: cannot find -lmoldname /usr/bin/x86_64-w64-mingw32-ld: cannot find -lmingwex /usr/bin/x86_64-w64-mingw32-ld: cannot find -lmsvcrt /usr/bin/x86_64-w64-mingw32-ld: cannot find /usr/lib/gcc/x86_64-w64-mingw32/8.3-win32/crtend.o: Too many open files collect2: error: ld returned 1 exit status Versions look like this: $ uname -a Linux name 5.0.0-trunk-amd64 #1 SMP Debian 5.0.2-1~exp1 (2019-03-18) x86_64 GNU/Linux $ /usr/bin/x86_64-w64-mingw32-g++ -v Using built-in specs. COLLECT_GCC=/usr/bin/x86_64-w64-mingw32-g++ COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-w64-mingw32/8.3-win32/lto-wrapper Target: x86_64-w64-mingw32 Configured with: ../../src/configure --build=x86_64-linux-gnu --prefix=/usr --includedir='/usr/include' --mandir='/usr/share/man' --infodir='/usr/share/info' --sysconfdir=/etc --localstatedir=/var --disable-silent-rules --libdir='/usr/lib/x86_64-linux-gnu' --libexecdir='/usr/lib/x86_64-linux-gnu' --disable-maintainer-mode --disable-dependency-tracking --prefix=/usr --enable-shared --enable-static --disable-multilib --with-system-zlib --libexecdir=/usr/lib --without-included-gettext --libdir=/usr/lib --enable-libstdcxx-time=yes --with-tune=generic --with-headers=/usr/x86_64-w64-mingw32/include --enable-version-specific-runtime-libs --enable-fully-dynamic-string --enable-libgomp --enable-languages=c,c++,fortran,objc,obj-c++,ada --enable-lto --enable-threads=win32 --program-suffix=-win32 --program-prefix=x86_64-w64-mingw32- --target=x86_64-w64-mingw32 --with-as=/usr/bin/x86_64-w64-mingw32-as --with-ld=/usr/bin/x86_64-w64-mingw32-ld --enable-libatomic --enable-libstdcxx-filesystem-ts=yes Thread model: win32 gcc version 8.3-win32 20190428 (GCC) $ /usr/bin/x86_64-w64-mingw32-ld --version GNU ld (GNU Binutils) 2.32 Copyright (C) 2019 Free Software Foundation, Inc. This program is free software; you may redistribute it under the terms of the GNU General Public License version 3 or (at your option) a later version. This program has absolutely no warranty. The ld command-line that is failing looks like this: /usr/bin/x86_64-w64-mingw32-ld -plugin /usr/lib/gcc/x86_64-w64-mingw32/8.3-win32/liblto_plugin.so -plugin-opt=/usr/lib/gcc/x86_64-w64-mingw32/8.3-win32/lto-wrapper -plugin-opt=-fresolution=/tmp/ccXUMZS3.res -plugin-opt=-pass-through=-lmingw32 -plugin-opt=-pass-through=-lgcc_s -plugin-opt=-pass-through=-lgcc -plugin-opt=-pass-through=-lmoldname -plugin-opt=-pass-through=-lmingwex -plugin-opt=-pass-through=-lmsvcrt -plugin-opt=-pass-through=-lpthread -plugin-opt=-pass-through=-ladvapi32 -plugin-opt=-pass-through=-lshell32 -plugin-opt=-pass-through=-luser32 -plugin-opt=-pass-through=-lkernel32 -plugin-opt=-pass-through=-lmingw32 -plugin-opt=-pass-through=-lgcc_s -plugin-opt=-pass-through=-lgcc -plugin-opt=-pass-through=-lmoldname -plugin-opt=-pass-through=-lmingwex -plugin-opt=-pass-through=-lmsvcrt -m i386pep --subsystem console -Bdynamic -o rb.exe /usr/lib/gcc/x86_64-w64-mingw32/8.3-win32/../../../../x86_64-w64-mingw32/lib/crt2.o /usr/lib/gcc/x86_64-w64-mingw32/8.3-win32/crtbegin.o -L/home/bredelings/win_root/mingw64/lib -L/usr/lib/gcc/x86_64-w64-mingw32/8.3-win32 -L/usr/lib/gcc/x86_64-w64-mingw32/8.3-win32/../../../../x86_64-w64-mingw32/lib rb@exe/src_revlanguage_main.cpp.obj -O1 --start-group librb-core.a librb-revlanguage.a librb-lib.a -lboost_regex-mt -lboost_program_options-mt -lboost_thread-mt -lboost_system-mt -lboost_filesystem-mt -lboost_date_time-mt -lboost_serialization-mt -lkernel32 -luser32 -lgdi32 -lwinspool -lshell32 -lole32 -loleaut32 -luuid -lcomdlg32 -ladvapi32 --end-group -lstdc++ -lmingw32 -lgcc_s -lgcc -lmoldname -lmingwex -lmsvcrt -lpthread -ladvapi32 -lshell32 -luser32 -lkernel32 -lmingw32 -lgcc_s -lgcc -lmoldname -lmingwex -lmsvcrt /usr/lib/gcc/x86_64-w64-mingw32/8.3-win32/crtend.o
Created attachment 11859 [details] strace of ld while failing Here's an strace of the command line /usr/bin/x86_64-w64-mingw32-ld -plugin /usr/lib/gcc/x86_64-w64-mingw32/8.3-win32/liblto_plugin.so -plugin-opt=/usr/lib/gcc/x86_64-w64-mingw32/8.3-win32/lto-wrapper -plugin-opt=-fresolution=/tmp/ccXUMZS3.res -plugin-opt=-pass-through=-lmingw32 -plugin-opt=-pass-through=-lgcc_s -plugin-opt=-pass-through=-lgcc -plugin-opt=-pass-through=-lmoldname -plugin-opt=-pass-through=-lmingwex -plugin-opt=-pass-through=-lmsvcrt -plugin-opt=-pass-through=-lpthread -plugin-opt=-pass-through=-ladvapi32 -plugin-opt=-pass-through=-lshell32 -plugin-opt=-pass-through=-luser32 -plugin-opt=-pass-through=-lkernel32 -plugin-opt=-pass-through=-lmingw32 -plugin-opt=-pass-through=-lgcc_s -plugin-opt=-pass-through=-lgcc -plugin-opt=-pass-through=-lmoldname -plugin-opt=-pass-through=-lmingwex -plugin-opt=-pass-through=-lmsvcrt -m i386pep --subsystem console -Bdynamic -o rb.exe /usr/lib/gcc/x86_64-w64-mingw32/8.3-win32/../../../../x86_64-w64-mingw32/lib/crt2.o /usr/lib/gcc/x86_64-w64-mingw32/8.3-win32/crtbegin.o -L/home/bredelings/win_root/mingw64/lib -L/usr/lib/gcc/x86_64-w64-mingw32/8.3-win32 -L/usr/lib/gcc/x86_64-w64-mingw32/8.3-win32/../../../../x86_64-w64-mingw32/lib rb@exe/src_revlanguage_main.cpp.obj -O1 --start-group librb-core.a librb-revlanguage.a librb-lib.a -lboost_regex-mt -lboost_program_options-mt -lboost_thread-mt -lboost_system-mt -lboost_filesystem-mt -lboost_date_time-mt -lboost_serialization-mt -lkernel32 -luser32 -lgdi32 -lwinspool -lshell32 -lole32 -loleaut32 -luuid -lcomdlg32 -ladvapi32 --end-group -lstdc++ -lmingw32 -lgcc_s -lgcc -lmoldname -lmingwex -lmsvcrt -lpthread -ladvapi32 -lshell32 -luser32 -lkernel32 -lmingw32 -lgcc_s -lgcc -lmoldname -lmingwex -lmsvcrt /usr/lib/gcc/x86_64-w64-mingw32/8.3-win32/crtend.o
Hi Benjamin, > /usr/bin/x86_64-w64-mingw32-ld -plugin > /usr/lib/gcc/x86_64-w64-mingw32/8.3-win32/liblto_plugin.so Is there any chance that you could provide a compressed tarball of the object files and libraries involved in this command line ? Without a test case to reproduce the problem, solving it is going to be really hard. I suspect that the problem is contained within some PE specific code in the linker, as otherwise we would have seen this issue reported elsewhere as well. My guess is that an archive is being scanned repeatedly for some reason. If you are able to run the linker inside a debugger you could put a break point on open() or bfd_openr() and find out where the kajority of the calls are coming from. Cheers Nick
Hi Nick, Thanks for the input. So far I've just looked at the strace log that I uploaded, and you are right that archives are getting opened repeatedly. There are three archive filesand they get opened a lot of times: * librb-core.a: opened 787 times * librb-revlanguage.a: opened 581 times * librb-lib.a: opened 20 times This seems to be related to the number of objects in each archive: * librb-core.a: 633 objects * librb-revlanguage.a: 594 objects * librb-lib: 19 objects However, its possible the number of opens is incomplete, since the later opens started failing due to too many open files.
The build directory is about 74Mb compressed, or 38M if I strip out unneeded files. Would there be a good place to upload this?
I tried running the linker under gdb, but apparently the debian package has stripped out all the debug symbols. Usually you can unstall them in a separate *-dbg package, but I don't see one for the mingw binutils. So, I can investigate how to compile the mingw ld from source, but if there is a good place to upload the archive files, I might wait on that.
Hi Benjamin, (In reply to Benjamin Redelings from comment #13) > The build directory is about 74Mb compressed, or 38M if I strip out unneeded > files. Would there be a good place to upload this? Ah yes, that might be difficult. Do you have a Fedora Project account ? If so you could upload the zipfile/tarball there. If you are using zip to create the archive you could use the -s option to split the output and then create multiple attachments to this BZ. Otherwise I think you will need to choose a file sharing site of some kind and try uploading there. Cheers Nick
Hi Nick, I was able to upload the files to here: https://www.dropbox.com/s/jyaapql2gj1r7br/gcc-win64.tar.gz?dl=0 Can you let me know if this works? $ cd gcc-win64 $ /usr/bin/x86_64-w64-mingw32-ld -plugin /usr/lib/gcc/x86_64-w64-mingw32/8.3-win32/liblto_plugin.so -plugin-opt=/usr/lib/gcc/x86_64-w64-mingw32/8.3-win32/lto-wrapper -plugin-opt=-fresolution=/tmp/ccXUMZS3.res -plugin-opt=-pass-through=-lmingw32 -plugin-opt=-pass-through=-lgcc_s -plugin-opt=-pass-through=-lgcc -plugin-opt=-pass-through=-lmoldname -plugin-opt=-pass-through=-lmingwex -plugin-opt=-pass-through=-lmsvcrt -plugin-opt=-pass-through=-lpthread -plugin-opt=-pass-through=-ladvapi32 -plugin-opt=-pass-through=-lshell32 -plugin-opt=-pass-through=-luser32 -plugin-opt=-pass-through=-lkernel32 -plugin-opt=-pass-through=-lmingw32 -plugin-opt=-pass-through=-lgcc_s -plugin-opt=-pass-through=-lgcc -plugin-opt=-pass-through=-lmoldname -plugin-opt=-pass-through=-lmingwex -plugin-opt=-pass-through=-lmsvcrt -m i386pep --subsystem console -Bdynamic -o rb.exe /usr/lib/gcc/x86_64-w64-mingw32/8.3-win32/../../../../x86_64-w64-mingw32/lib/crt2.o /usr/lib/gcc/x86_64-w64-mingw32/8.3-win32/crtbegin.o -L/home/bredelings/win_root/mingw64/lib -L/usr/lib/gcc/x86_64-w64-mingw32/8.3-win32 -L/usr/lib/gcc/x86_64-w64-mingw32/8.3-win32/../../../../x86_64-w64-mingw32/lib rb@exe/src_revlanguage_main.cpp.obj -O1 --start-group librb-core.a librb-revlanguage.a librb-lib.a -lboost_regex-mt -lboost_program_options-mt -lboost_thread-mt -lboost_system-mt -lboost_filesystem-mt -lboost_date_time-mt -lboost_serialization-mt -lkernel32 -luser32 -lgdi32 -lwinspool -lshell32 -lole32 -loleaut32 -luuid -lcomdlg32 -ladvapi32 --end-group -lstdc++ -lmingw32 -lgcc_s -lgcc -lmoldname -lmingwex -lmsvcrt -lpthread -ladvapi32 -lshell32 -luser32 -lkernel32 -lmingw32 -lgcc_s -lgcc -lmoldname -lmingwex -lmsvcrt /usr/lib/gcc/x86_64-w64-mingw32/8.3-win32/crtend.o /usr/bin/x86_64-w64-mingw32-ld: cannot find -lstdc++ /usr/bin/x86_64-w64-mingw32-ld: cannot find -lmingw32 /usr/bin/x86_64-w64-mingw32-ld: cannot find -lgcc_s /usr/bin/x86_64-w64-mingw32-ld: cannot find -lgcc /usr/bin/x86_64-w64-mingw32-ld: cannot find -lmoldname /usr/bin/x86_64-w64-mingw32-ld: cannot find -lmingwex /usr/bin/x86_64-w64-mingw32-ld: cannot find -lmsvcrt /usr/bin/x86_64-w64-mingw32-ld: cannot find -lpthread /usr/bin/x86_64-w64-mingw32-ld: cannot find -ladvapi32 /usr/bin/x86_64-w64-mingw32-ld: cannot find -lshell32 /usr/bin/x86_64-w64-mingw32-ld: cannot find -luser32 /usr/bin/x86_64-w64-mingw32-ld: cannot find -lkernel32 /usr/bin/x86_64-w64-mingw32-ld: cannot find -lmingw32 /usr/bin/x86_64-w64-mingw32-ld: cannot find -lgcc_s /usr/bin/x86_64-w64-mingw32-ld: cannot find -lgcc /usr/bin/x86_64-w64-mingw32-ld: cannot find -lmoldname /usr/bin/x86_64-w64-mingw32-ld: cannot find -lmingwex /usr/bin/x86_64-w64-mingw32-ld: cannot find -lmsvcrt /usr/bin/x86_64-w64-mingw32-ld: cannot find /usr/lib/gcc/x86_64-w64-mingw32/8.3-win32/crtend.o: Too many open files $ /usr/bin/x86_64-w64-mingw32-ld -m i386pep --subsystem console -Bdynamic -o rb.exe /usr/lib/gcc/x86_64-w64-mingw32/8.3-win32/../../../../x86_64-w64-mingw32/lib/crt2.o /usr/lib/gcc/x86_64-w64-mingw32/8.3-win32/crtbegin.o -L/home/bredelings/win_root/mingw64/lib -L/usr/lib/gcc/x86_64-w64-mingw32/8.3-win32 -L/usr/lib/gcc/x86_64-w64-mingw32/8.3-win32/../../../../x86_64-w64-mingw32/lib rb@exe/src_revlanguage_main.cpp.obj -O1 --start-group librb-core.a librb-revlanguage.a librb-lib.a -lboost_regex-mt -lboost_program_options-mt -lboost_thread-mt -lboost_system-mt -lboost_filesystem-mt -lboost_date_time-mt -lboost_serialization-mt -lkernel32 -luser32 -lgdi32 -lwinspool -lshell32 -lole32 -loleaut32 -luuid -lcomdlg32 -ladvapi32 --end-group -lstdc++ -lmingw32 -lgcc_s -lgcc -lmoldname -lmingwex -lmsvcrt -lpthread -ladvapi32 -lshell32 -luser32 -lkernel32 -lmingw32 -lgcc_s -lgcc -lmoldname -lmingwex -lmsvcrt /usr/lib/gcc/x86_64-w64-mingw32/8.3-win32/crtend.o $ So removing the linker plugin arguments makes the command work. I can put the linked binary from the no-linker-plugin version of the command in the dropbox folder if that's helpful. Its about 47Mb, 9Mb compressed. -BenRI
Hi Benjamin, > https://www.dropbox.com/s/jyaapql2gj1r7br/gcc-win64.tar.gz?dl=0 Thanks - I was able to download the tarball from there. Unfortunately I do not have a copy of the /usr/lib/x86_64-w64-mingw32/lib/crt2.o object file. (I am using an installed x86_64-mingw32 cross compiler running under Linux). When I commented that part out from the build command line however I was able to run the entire command, including using the plugin, and not get an error message about too many open files. (I did get the messages about the missing libraries however). I suspect therefore that part of the problem is that a native mingw32 build system has a lower limit on the maximum number of files that can be open at any one time. > So removing the linker plugin arguments makes the command work. OK, so this is significant. The implication therefore is that it is the plugin itself which is opening files and then not closing them. (Or maybe the plugin calls a linker function which opens a file, but then the plugin does call the corresponding linker function to close the file). It also means that compiling without -flto is an option for a workaround. Unfortunately the libtlo plugin is actually part of the gcc project, rather than the binutils project, so you may need to refile this bug report there, sorry. In fact it may be that there is no simple solution. It may well be that linking large projects with LTO enabled on a system with a low limit on the number of open files is just not going to work. :-( Not what you want to hear, I know, but it certainly is starting to look like this is the case. Cheers Nick
Hi Nick, Thank you so much for looking into this. I reran the exact same command and now it works with the linker plugin just fine. I also check using /proc, and the linker process has about 30 open files. I have done some package upgrades, and it looks like a patch for Debug bug 928214 ("Configure LTO support correctly", https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=928214) may be the one that fixed the problem. In any case, for the benefit of any future triggerers of this bug: 1. The problem is probably with GCC, not binutils. 2. I was not using native mingw binutils, I was using Debian's linux-native cross compiler. Referring to the bug report above, one possible explanation is that * the LTO plugin was misconfigured in autoconf * this caused gcc to fall back to the lto-wrapper helper and NOT use the linker plugin * the lto-wrapper help creates N open file descriptors for an archive containing N objects. * the actual lto-plugin works fine. This is somewhat speculative, but might hopefully give someone a path to explore if they run into this again.
It looks like this bug has already been fixed, so closing this PR.
For reference, the commit which fixes this is 999d6dff80fab12d22c2a8d91923db6bde7fb3e5