Bug 12658 - ld: potential infinite loop and memory leaks when link many object files
Summary: ld: potential infinite loop and memory leaks when link many object files
Status: UNCONFIRMED
Alias: None
Product: binutils
Classification: Unclassified
Component: binutils (show other bugs)
Version: 2.22
: P2 critical
Target Milestone: ---
Assignee: unassigned
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2011-04-09 13:39 UTC by Dongsheng Song
Modified: 2016-09-17 06:12 UTC (History)
7 users (show)

See Also:
Host:
Target:
Build:
Last reconfirmed:


Attachments
Compile gcc 4.6 for mingw-w64 target (977 bytes, text/plain)
2011-04-11 15:36 UTC, Dongsheng Song
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Dongsheng Song 2011-04-09 13:39:18 UTC
When I compile wxWidgets 2.8.12 into single library, ld use huge memory, I have to use 64bit Linux to do cross compile:

CFLAGS='-pipe' CXXFLAGS='-pipe' LDFLAGS='-pipe -s -static' \
${HOME}/vcs/svn/wxWidgets/branches/WX_2_8_BRANCH/configure --prefix=/tmp/w32 \
  --host=i686-w64-mingw32 --with-odbc --with-opengl --enable-shared \
  --enable-unicode --enable-mediactrl --enable-official_build \
  --disable-compat26 --enable-abi-incompatible-features --enable-monolithic

When make finished, 454 object files generate one 12M file wxmsw28u_gcc.dll.

During the building, after ld running 25 minutes, the VM physical memory (3GB) was exhausted, the host have 4 Xeon 3.20GHz CPU, and the other VMs were idle.

Maybe there have memory leaks in ld ?

*) 4 minutes
Cpu(s): 18.2%us,  5.9%sy,  0.0%ni, 75.1%id,  0.8%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:   3095296k total,  1354524k used,  1740772k free,      420k buffers
Swap:  7811064k total,        0k used,  7811064k free,   284988k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 1110 dongshen  20   0  989m 983m  892 R   96 32.5   3:50.05 ld

*) 8 minutes
Cpu(s): 20.8%us,  4.5%sy,  0.0%ni, 73.9%id,  0.7%wa,  0.0%hi,  0.2%si,  0.0%st
Mem:   3095296k total,  2531320k used,   563976k free,      420k buffers
Swap:  7811064k total,        0k used,  7811064k free,   500396k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 1110 dongshen  20   0 1926m 1.9g  892 R   97 63.5   8:01.56 ld

*) 18 minutes
Cpu(s): 19.1%us,  7.1%sy,  0.0%ni, 73.5%id,  0.0%wa,  0.0%hi,  0.2%si,  0.0%st
Mem:   3095296k total,  3072352k used,    22944k free,       32k buffers
Swap:  7811064k total,        0k used,  7811064k free,   290136k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 1110 dongshen  20   0 2658m 2.6g  892 R   99 87.8  18:02.34 ld

*) 25 minutes
Cpu(s): 22.9%us, 16.6%sy,  0.0%ni, 28.7%id, 29.4%wa,  0.2%hi,  2.2%si,  0.0%st
Mem:   3095296k total,  3075624k used,    19672k free,       16k buffers
Swap:  7811064k total,   201056k used,  7610008k free,    11748k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 1110 dongshen  20   0 3086m 2.8g  444 R   99 95.9  25:00.40 ld
Comment 1 Dongsheng Song 2011-04-10 04:24:34 UTC
Now, ld running 6 hours cpu time, use 4.6GB memory:

VmPeak:  4776608 kB
VmSize:  4734656 kB
VmLck:         0 kB
VmHWM:   3020260 kB
VmRSS:   2982304 kB
VmData:  4728000 kB
VmStk:       100 kB
VmExe:       912 kB
VmLib:      1504 kB
VmPTE:      9232 kB
Threads:        1

Cpu(s):  7.9%us, 11.5%sy,  0.0%ni, 44.4%id, 33.3%wa,  0.2%hi,  2.7%si,  0.0%st
Mem:   3095296k total,  3073516k used,    21780k free,       16k buffers
Swap:  7811064k total,  3905532k used,  3905532k free,     9484k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 1110 dongshen  20   0 4623m 2.8g  196 D   49 96.1 358:15.13 ld


root@x64-ora-01:/home/dongsheng/gcc-4.6-windows-linux# ./bin/i686-w64-mingw32-g++ -v
Using built-in specs.
COLLECT_GCC=./bin/i686-w64-mingw32-g++
COLLECT_LTO_WRAPPER=/home/dongsheng/gcc-4.6-windows-linux/libexec/gcc/i686-w64-mingw32/4.6.1/lto-wrapper
Target: i686-w64-mingw32
Configured with: /home/dongsheng/vcs/svn/gcc/branches/gcc-4_6-branch/configure --prefix=/home/dongsheng/gcc-4.6-windows-linux --with-sysroot=/home/dongsheng/gcc-4.6-windows-linux --target=i686-w64-mingw32 --disable-multilib --enable-languages=c,c++
Thread model: win32
gcc version 4.6.1 20110408 (prerelease) (GCC)

root@x64-ora-01:/home/dongsheng/gcc-4.6-windows-linux# ./i686-w64-mingw32/bin/ld --version
GNU ld (GNU Binutils) 2.21.51.20110408
Copyright 2011 Free Software Foundation, Inc.
This program is free software; you may redistribute it under the terms of
the GNU General Public License version 3 or (at your option) a later version.
This program has absolutely no warranty.
Comment 2 Nick Clifton 2011-04-11 11:56:07 UTC
Hi Dongsheng,

  Have you tried linking with the --reduce-memory-overheads option specified ?

  It is entirely possible that there are memory leaks in the linker, but without a lot of analysis it is hard to say where they might be.

  It does sound as though you are producing a really big DLL, so maybe it is just a case that you need lots of memory to perform the link.  Have you tried using earlier versions of the linker to see if this problem is something that has only recently been introduced ?

Cheers
  Nick
Comment 3 Dongsheng Song 2011-04-11 13:21:36 UTC
(In reply to comment #2)
> Hi Dongsheng,
> 
>   Have you tried linking with the --reduce-memory-overheads option specified ?
> 
>   It is entirely possible that there are memory leaks in the linker, but
> without a lot of analysis it is hard to say where they might be.
> 
>   It does sound as though you are producing a really big DLL, so maybe it is
> just a case that you need lots of memory to perform the link.  Have you tried
> using earlier versions of the linker to see if this problem is something that
> has only recently been introduced ?
> 
> Cheers
>   Nick

When I use '-Wl,--no-keep-memory,--reduce-memory-overheads', only reduce about 200MB memory usage.

I'm not very care about the memory usage, I'm more concern the infinite loop during link because it blocked from generate the final .DLL file.

*) 25 minutes
Cpu(s): 16.2%us,  8.0%sy,  0.0%ni, 75.0%id,  0.0%wa,  0.0%hi,  0.7%si,  0.0%st
Mem:   3095296k total,  3073056k used,    22240k free,       32k buffers
Swap:  7811064k total,        0k used,  7811064k free,   321800k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 1159 dongshen  20   0 2627m 2.6g  896 R  104 86.7  25:35.69 ld
Comment 4 Nick Clifton 2011-04-11 13:58:12 UTC
Hi Dongsheg,

  Ok - we are going to need a testcase that reproduced this problem then.  I appreciate that this might be difficult as I assume that you will not be able to reduce the size/number of object files involved.  If you can place a compressed tarball somewhere where I can download it and give me a full linker command line, I will try reproducing the problem myself.

Cheers
  Nick
Comment 5 Dongsheng Song 2011-04-11 15:36:47 UTC
Created attachment 5661 [details]
Compile gcc 4.6 for mingw-w64 target
Comment 6 Dongsheng Song 2011-04-11 15:37:39 UTC
(In reply to comment #5)
> Created attachment 5661 [details]
> Compile gcc 4.6 for mingw-w64 target

Sorry, the working directory size is 1.2GB, it's too hard to upload.

Maybe you can build yourself:
1) build mingw-w64 cross compiler
If you use Debian 6.0 i686 or amd64 box, I can upload the compiler.
See gcc-4.6-windows-x86.sh for more details.

2) Checkout wxWidgets 2.8 branch (http://svn.wxwidgets.org/svn/wx/wxWidgets/branches/WX_2_8_BRANCH)

3) Configure
CFLAGS='-pipe' CXXFLAGS='-pipe' LDFLAGS='-pipe -s -static' \
${HOME}/vcs/svn/wxWidgets/branches/WX_2_8_BRANCH/configure --prefix=/tmp/w32 --host=i686-w64-mingw32 \
  --with-odbc --with-opengl --enable-shared --enable-unicode --enable-mediactrl --enable-official_build --disable-compat26 --enable-abi-incompatible-features
...
Configured wxWidgets 2.8.12 for `i686-w64-mingw32'

  Which GUI toolkit should wxWidgets use?                 msw
  Should wxWidgets be compiled into single library?       no
  Should wxWidgets be compiled in debug mode?             no
  Should wxWidgets be linked as a shared library?         yes
  Should wxWidgets be compiled in Unicode mode?           yes
  What level of wxWidgets compatibility should be enabled?
                                       wxWidgets 2.4      no
                                       wxWidgets 2.6      no
  Which libraries should wxWidgets use?
                                       jpeg               builtin
                                       png                builtin
                                       regex              builtin
                                       tiff               builtin
                                       zlib               builtin
                                       odbc               yes
                                       expat              builtin
                                       libmspack          no
                                       sdl                no

4) build
make -j2
Comment 7 Nick Clifton 2011-04-12 10:18:29 UTC
Hi Dongsheng,

  I am sorry - I am currently unable to build the mingw-w64 cross compiler using the gcc-4.6-windows-x86.sh script.  It fails building the CRT with:

  configure: error: Please check if the mingw-w64 header set and the build/host option are set properly.
  make: *** No targets specified and no makefile found.  Stop.
  [2011-04-12 10:03:38] Build Mingw-w64 crt failed

I suspected that something like this might happen. :-(

Are you able to run the linker under GDB and maybe find out where the infinite loop is happening ?

Cheers
  Nick
Comment 8 Dongsheng Song 2011-04-13 06:17:27 UTC
(In reply to comment #7)
> Hi Dongsheng,
> 
>   I am sorry - I am currently unable to build the mingw-w64 cross compiler
> using the gcc-4.6-windows-x86.sh script.  It fails building the CRT with:
> 
>   configure: error: Please check if the mingw-w64 header set and the build/host
> option are set properly.
>   make: *** No targets specified and no makefile found.  Stop.
>   [2011-04-12 10:03:38] Build Mingw-w64 crt failed
> 
> I suspected that something like this might happen. :-(
> 
> Are you able to run the linker under GDB and maybe find out where the infinite
> loop is happening ?
> 
> Cheers
>   Nick

I have no enough to run the gdb to debug ld now.

I can not know why your build failed without the exact error messages, but I think this may be caused by your Linux not have the proper version of gmp, mpfr and mpc develop packages.

Packages necessary for building GCC 4.6:
http://gcc.gnu.org/install/prerequisites.html

GMP 4.3.2 (or later)
MPFR 2.4.2 (or later)
MPC 0.8.1 (or later)

FC14:
http://download.fedora.redhat.com/pub/fedora/linux/releases/14/Fedora/source/SRPMS/
gmp-4.3.1-7.fc14.src.rpm
mpfr-2.4.2-1.fc13.src.rpm
libmpc-0.8.1-1.fc13.src.rpm

FC15:
http://download.fedora.redhat.com/pub/fedora/linux/releases/test/15-Alpha/Fedora/source/SRPMS/
gmp-4.3.2-3.fc15.src.rpm
mpfr-3.0.0-4.fc15.src.rpm
libmpc-0.8.3-0.3.svn855.fc15.src.rpm

RHEL6:
http://ftp.scientificlinux.org/linux/scientific/6/SRPMS/vendor/
gmp-4.3.1-7.el6.src.rpm
mpfr-2.4.1-6.el6.src.rpm

If the answer is yes, then you must use FC15 or upgrade your gmp, mpfr and mpc develop packages yourself.
Comment 9 cvs-commit@gcc.gnu.org 2011-04-13 12:53:39 UTC
CVSROOT:	/cvs/src
Module name:	src
Changes by:	ktietz@sourceware.org	2011-04-13 12:53:36

Modified files:
	ld             : ChangeLog deffile.h deffilep.y pe-dll.c 

Log message:
	2011-04-13  Kai Tietz  <ktietz@redhat.com>
	
	PR binutils/12658
	* deffile.h (def_file_add_export): Add is_dup argument.
	(def_file_add_import): Likewise.
	* deffilep.y (are_names_equal): New helper.
	(cmp_export_elem): New helper.
	(find_export_in_list): Add search routine for exports.
	(def_file_add_export): Check for duplicates.
	(cmp_import_elem): New helper.
	(find_import_in_list): Add search routine for imports.
	(def_file_add_import): Check for duplicates.
	(def_exports): Handle duplicates.
	(def_imports): Likewise.
	* pe-dll.c (process_def_file_and_drectve): Likewise.
	(pe_implied_import_dll): Likewise.

Patches:
http://sourceware.org/cgi-bin/cvsweb.cgi/src/ld/ChangeLog.diff?cvsroot=src&r1=1.2310&r2=1.2311
http://sourceware.org/cgi-bin/cvsweb.cgi/src/ld/deffile.h.diff?cvsroot=src&r1=1.14&r2=1.15
http://sourceware.org/cgi-bin/cvsweb.cgi/src/ld/deffilep.y.diff?cvsroot=src&r1=1.33&r2=1.34
http://sourceware.org/cgi-bin/cvsweb.cgi/src/ld/pe-dll.c.diff?cvsroot=src&r1=1.135&r2=1.136
Comment 10 Nick Clifton 2011-04-13 14:01:37 UTC
Hi Dongsheng,

 Please could you try the latest FSF mainline sources with Kai's patch in them, and see if this has any affect on your problem ?

Cheers
  Nick
Comment 11 Dongsheng Song 2011-04-13 14:53:22 UTC
(In reply to comment #10)
> Hi Dongsheng,
> 
>  Please could you try the latest FSF mainline sources with Kai's patch in them,
> and see if this has any affect on your problem ?
> 
> Cheers
>   Nick

In 32bit Linux, still out of memory:

/home/dongsheng/gcc-4.6-windows-linux/lib/gcc/i686-w64-mingw32/4.6.1/../../../../i686-w64-mingw32/bin/ld: out of memory allocating 82956108 bytes after a total of 2268610560 bytes
collect2: ld returned 1 exit status
make: *** [/home/dongsheng/tmp/wx/lib/wxmsw28u_gcc.dll] Error 1

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 4039 oracle    20   0 3024m 2.9g  724 R  100 89.6  18:50.58 ld

VmPeak:  3097084 kB
VmSize:  3097084 kB
VmLck:         0 kB
VmHWM:   3041600 kB
VmRSS:   3022700 kB
VmData:  3094432 kB
VmStk:       100 kB
VmExe:      1036 kB
VmLib:      1472 kB
VmPTE:      3036 kB
Comment 12 Dongsheng Song 2011-04-14 01:32:17 UTC
(In reply to comment #11)
> (In reply to comment #10)
> > Hi Dongsheng,
> > 
> >  Please could you try the latest FSF mainline sources with Kai's patch in them,
> > and see if this has any affect on your problem ?
> > 
> > Cheers
> >   Nick
> 
> In 32bit Linux, still out of memory:

It seems that Kai's patch save about 300M+ memory on 64 bit Linux box (My new Scientific Linux 6 VM, 1G physical memory).

But it seems that still do a infinite loop:

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  DATA COMMAND
15602 dongshen  20   0 4246m 768m  144 D  4.3 77.0  47:49.43 4.1g ld

VmPeak:  4348140 kB
VmSize:  4348140 kB
VmLck:         0 kB
VmHWM:    892440 kB
VmRSS:    787316 kB
VmData:  4341348 kB
VmStk:       100 kB
VmExe:       928 kB
VmLib:      1620 kB
VmPTE:      8508 kB
VmSwap:  3554152 kB

# pmap -d 15602
15602:   /home/dongsheng/gcc-4.6-windows-linux/lib/gcc/i686-w64-mingw32/4.6.1/../../../../i686-w64-mingw32/bin/ld --sysroot=/home/dongsheng/gcc-4.6-windows-linux -m i386pe --shared -Bstatic -e _DllMainCRTStartup@12 --enable-auto-image-base -o /home/dongsheng/tmp/wx/lib/wxmsw28u_gcc.dll -s -s /home/dongsheng/gcc-4.6-windows-linux/lib/gcc/i686-w64-mingw32/4.6.1/../../../../i686-w64-mingw32/lib/../lib/dllcrt2.o /home/dongsheng/gcc-4.6-windows-linux/lib/gcc/i686-w64-mingw32/4.6.1/crtbegin.o -L/home/dongsheng/tmp/wx/lib
Address           Kbytes Mode  Offset           Device    Mapping
0000000000400000     928 r-x-- 0000000000000000 0fd:00000 ld
00000000006e7000      12 rw--- 00000000000e7000 0fd:00000 ld
00000000006ea000      20 rw--- 0000000000000000 000:00000   [ anon ]
0000000000801000 3494196 rw--- 0000000000000000 000:00000   [ anon ]
00007f016d265000    1052 rw--- 0000000000000000 000:00000   [ anon ]
00007f016d38c000  835028 rw--- 0000000000000000 000:00000   [ anon ]
00007f01a0302000    1832 rw--- 0000000000000000 000:00000   [ anon ]
00007f01a04cd000    1816 rw--- 0000000000000000 000:00000   [ anon ]
00007f01a0694000    1848 rw--- 0000000000000000 000:00000   [ anon ]
00007f01a0863000    1848 rw--- 0000000000000000 000:00000   [ anon ]
00007f01a0a32000    1908 rw--- 0000000000000000 000:00000   [ anon ]
00007f01a0c10000     420 rw--- 0000000000000000 000:00000   [ anon ]
00007f01a0c79000    1492 r-x-- 0000000000000000 0fd:00000 libc-2.12.so
00007f01a0dee000    2048 ----- 0000000000175000 0fd:00000 libc-2.12.so
00007f01a0fee000      16 r---- 0000000000175000 0fd:00000 libc-2.12.so
00007f01a0ff2000       4 rw--- 0000000000179000 0fd:00000 libc-2.12.so
00007f01a0ff3000      20 rw--- 0000000000000000 000:00000   [ anon ]
00007f01a0ff8000       8 r-x-- 0000000000000000 0fd:00000 libdl-2.12.so
00007f01a0ffa000    2048 ----- 0000000000002000 0fd:00000 libdl-2.12.so
00007f01a11fa000       4 r---- 0000000000002000 0fd:00000 libdl-2.12.so
00007f01a11fb000       4 rw--- 0000000000003000 0fd:00000 libdl-2.12.so
00007f01a11fc000     120 r-x-- 0000000000000000 0fd:00000 ld-2.12.so
00007f01a1225000      20 rw--- 0000000000000000 000:00000   [ anon ]
00007f01a126a000    1724 rw--- 0000000000000000 000:00000   [ anon ]
00007f01a1419000       4 rw--- 0000000000000000 000:00000   [ anon ]
00007f01a141a000       4 r---- 000000000001e000 0fd:00000 ld-2.12.so
00007f01a141b000       4 rw--- 000000000001f000 0fd:00000 ld-2.12.so
00007f01a141c000       4 rw--- 0000000000000000 000:00000   [ anon ]
00007fffc12ad000      96 rw--- 0000000000000000 000:00000   [ stack ]
00007fffc13c4000       4 r-x-- 0000000000000000 000:00000   [ anon ]
ffffffffff600000       4 r-x-- 0000000000000000 000:00000   [ anon ]
mapped: 4348536K    writeable/private: 4341860K    shared: 0K
Comment 13 Dongsheng Song 2011-04-14 02:04:40 UTC
Here is the stack trace (After ld running 8 minutes, use 1.4GB memory):

# gdb /home/dongsheng/gcc-4.6-windows-linux/i686-w64-mingw32/bin/ld 28412
GNU gdb (GDB) Red Hat Enterprise Linux (7.1-29.el6)
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /home/dongsheng/gcc-4.6-windows-linux/i686-w64-mingw32/bin/ld...done.
Attaching to program: /home/dongsheng/gcc-4.6-windows-linux/i686-w64-mingw32/bin/ld, process 28412
Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
_bfd_pei_swap_sym_in (abfd=0x4c5eda20, ext1=0x5474af34, in1=0x7fff02619200) at peigen.c:115
115         in->n_type = H_GET_16 (abfd, ext->e_type);
Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.7.el6_0.5.x86_64
(gdb) bt
#0  _bfd_pei_swap_sym_in (abfd=0x4c5eda20, ext1=0x5474af34, in1=0x7fff02619200) at peigen.c:115
#1  0x0000000000446106 in handle_COMDAT (abfd=0x4c5eda20, hdr=<value optimized out>, name=<value optimized out>,
    section=0x55422a28, flags_ptr=0x7fff02619ebc) at /home/dongsheng/vcs/git/binutils/bfd/coffcode.h:897
#2  styp_to_sec_flags (abfd=0x4c5eda20, hdr=<value optimized out>, name=<value optimized out>, section=0x55422a28,
    flags_ptr=0x7fff02619ebc) at /home/dongsheng/vcs/git/binutils/bfd/coffcode.h:1253
#3  0x0000000000431c0a in make_a_section_from_file (abfd=0x4c5eda20) at /home/dongsheng/vcs/git/binutils/bfd/coffgen.c:132
#4  coff_real_object_p (abfd=0x4c5eda20) at /home/dongsheng/vcs/git/binutils/bfd/coffgen.c:222
#5  coff_object_p (abfd=0x4c5eda20) at /home/dongsheng/vcs/git/binutils/bfd/coffgen.c:301
#6  0x00000000004320ce in bfd_check_format_matches (abfd=0x4c5eda20, format=<value optimized out>, matching=0x0)
    at /home/dongsheng/vcs/git/binutils/bfd/format.c:211
#7  0x000000000041b9df in ldfile_try_open_bfd (attempt=0x7fff0261c97b "monodll_msw_notebook.o", entry=0x160ea90)
    at /home/dongsheng/vcs/git/binutils/ld/ldfile.c:178
#8  0x000000000041c10c in ldfile_open_file (entry=0x160ea90) at /home/dongsheng/vcs/git/binutils/ld/ldfile.c:443
#9  0x0000000000410465 in load_symbols (entry=0x160ea90, place=0x7fff0261a040)
    at /home/dongsheng/vcs/git/binutils/ld/ldlang.c:2678
#10 0x0000000000410e52 in open_input_bfds (s=0x160ea90, force=0) at /home/dongsheng/vcs/git/binutils/ld/ldlang.c:3239
#11 0x000000000041215a in lang_process () at /home/dongsheng/vcs/git/binutils/ld/ldlang.c:6471
#12 0x00000000004168e6 in main (argc=536, argv=0x7fff0261a288) at /home/dongsheng/vcs/git/binutils/ld/ldmain.c:461
(gdb) q
A debugging session is active.

        Inferior 1 [process 28412] will be detached.

Quit anyway? (y or n) y
Detaching from program: /home/dongsheng/gcc-4.6-windows-linux/i686-w64-mingw32/bin/ld, process 28412
Comment 14 Dongsheng Song 2011-04-14 02:37:12 UTC
Here is the updated stack trace (After ld running 25 minutes, use 3.6GB memory):


  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  DATA COMMAND
28412 dongshen  20   0 3685m 860m  228 D 22.9 86.3  25:03.09 3.6g ld

# gdb /home/dongsheng/gcc-4.6-windows-linux/i686-w64-mingw32/bin/ld 28412
GNU gdb (GDB) Red Hat Enterprise Linux (7.1-29.el6)
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /home/dongsheng/gcc-4.6-windows-linux/i686-w64-mingw32/bin/ld...done.
Attaching to program: /home/dongsheng/gcc-4.6-windows-linux/i686-w64-mingw32/bin/ld, process 28412
Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
0x00000000004092e3 in match_simple_wild (pattern=0x161aa90 ".gnu.lto_*",
    name=0xbbcc5538 ".text$_ZNK12wxWindowBase9IsEnabledEv") at /home/dongsheng/vcs/git/binutils/ld/ldlang.c:368
368       if (pattern[0] != name[0] || pattern[1] != name[1]
Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.7.el6_0.5.x86_64
(gdb) bt
#0  0x00000000004092e3 in match_simple_wild (pattern=0x161aa90 ".gnu.lto_*",
    name=0xbbcc5538 ".text$_ZNK12wxWindowBase9IsEnabledEv") at /home/dongsheng/vcs/git/binutils/ld/ldlang.c:368
#1  0x000000000040b2d5 in walk_wild_section_specs1_wild1 (ptr=0x161a7a0, file=0x1613260,
    callback=0x409960 <gc_section_callback>, data=0x0) at /home/dongsheng/vcs/git/binutils/ld/ldlang.c:591
#2  0x0000000000410a11 in walk_wild (s=0x161a7a0, callback=0x409960 <gc_section_callback>, data=0x0)
    at /home/dongsheng/vcs/git/binutils/ld/ldlang.c:908
#3  0x0000000000410b67 in lang_gc_sections_1 (s=0x161a7a0) at /home/dongsheng/vcs/git/binutils/ld/ldlang.c:6241
#4  0x0000000000410b1d in lang_gc_sections_1 (s=0x1605cb0) at /home/dongsheng/vcs/git/binutils/ld/ldlang.c:6250
#5  0x0000000000412348 in lang_gc_sections () at /home/dongsheng/vcs/git/binutils/ld/ldlang.c:6263
#6  lang_process () at /home/dongsheng/vcs/git/binutils/ld/ldlang.c:6549
#7  0x00000000004168e6 in main (argc=536, argv=0x7fff0261a288) at /home/dongsheng/vcs/git/binutils/ld/ldmain.c:461
(gdb) q
A debugging session is active.

        Inferior 1 [process 28412] will be detached.

Quit anyway? (y or n) y
Detaching from program: /home/dongsheng/gcc-4.6-windows-linux/i686-w64-mingw32/bin/ld, process 28412
Comment 15 Dongsheng Song 2011-04-14 05:14:09 UTC
Here is the updated stack trace (After ld running CPU time 36 minutes, use 3.7GB
memory):

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  DATA COMMAND
28412 dongshen  20   0 3714m 844m  248 R  7.0 84.7  36:20.98 3.6g ld

VmPeak:  3803368 kB
VmSize:  3803368 kB
VmLck:         0 kB
VmHWM:    892824 kB
VmRSS:    860116 kB
VmData:  3796576 kB
VmStk:       100 kB
VmExe:       928 kB
VmLib:      1620 kB
VmPTE:      7448 kB
VmSwap:  2936724 kB

# gdb /home/dongsheng/gcc-4.6-windows-linux/i686-w64-mingw32/bin/ld 28412
GNU gdb (GDB) Red Hat Enterprise Linux (7.1-29.el6)
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /home/dongsheng/gcc-4.6-windows-linux/i686-w64-mingw32/bin/ld...done.
Attaching to program: /home/dongsheng/gcc-4.6-windows-linux/i686-w64-mingw32/bin/ld, process 28412
Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
0x000000000040bf93 in compare_section (sort=<value optimized out>, asec=0x52e6b5e8, bsec=0x242e1758)
    at /home/dongsheng/vcs/git/binutils/ld/ldlang.c:454
454           ret = strcmp (bfd_get_section_name (asec->owner, asec),
Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.7.el6_0.5.x86_64
(gdb) bt
#0  0x000000000040bf93 in compare_section (sort=<value optimized out>, asec=0x52e6b5e8, bsec=0x242e1758)
    at /home/dongsheng/vcs/git/binutils/ld/ldlang.c:454
#1  0x000000000040c0c7 in wild_sort_fast (ptr=0x1618140, sec=0x1618410, section=0x52e6b5e8, file=<value optimized out>,
    output=<value optimized out>) at /home/dongsheng/vcs/git/binutils/ld/ldlang.c:499
#2  output_section_callback_fast (ptr=0x1618140, sec=0x1618410, section=0x52e6b5e8, file=<value optimized out>, output=<value optimized out>)
    at /home/dongsheng/vcs/git/binutils/ld/ldlang.c:531
#3  0x000000000040b2f0 in walk_wild_section_specs1_wild1 (ptr=0x1618140, file=0x160e8d0, callback=0x40bff0 <output_section_callback_fast>,
    data=0x16057e0) at /home/dongsheng/vcs/git/binutils/ld/ldlang.c:594
#4  0x0000000000410a11 in walk_wild (s=0x1618140, callback=0x40bff0 <output_section_callback_fast>, data=0x16057e0)
    at /home/dongsheng/vcs/git/binutils/ld/ldlang.c:908
#5  0x000000000041208d in wild (s=0x1618140, target=0x0, os=0x16057e0) at /home/dongsheng/vcs/git/binutils/ld/ldlang.c:2835
#6  map_input_to_output_sections (s=0x1618140, target=0x0, os=0x16057e0) at /home/dongsheng/vcs/git/binutils/ld/ldlang.c:3533
#7  0x0000000000412057 in map_input_to_output_sections (s=0x16057e0, target=0x0, os=0x0) at /home/dongsheng/vcs/git/binutils/ld/ldlang.c:3555
#8  0x0000000000412451 in lang_process () at /home/dongsheng/vcs/git/binutils/ld/ldlang.c:6560
#9  0x00000000004168e6 in main (argc=536, argv=0x7fff0261a288) at /home/dongsheng/vcs/git/binutils/ld/ldmain.c:461
(gdb) q
A debugging session is active.

        Inferior 1 [process 28412] will be detached.

Quit anyway? (y or n) y
Detaching from program: /home/dongsheng/gcc-4.6-windows-linux/i686-w64-mingw32/bin/ld, process 28412
Comment 16 Nick Clifton 2011-04-14 15:27:42 UTC
Ho Dongsheng,

  Hmm, I am not sure what else to suggest.  You could try turning off section garbage collection (--no-gc-sections) as this appears to be turned on.

  If you really have 1.2 Gb of object files then I am not sure if you can reasonably expect the linker to be anything but slow when you only 1G of real memory available.  I would strongly urge trying the link on a bigger machine.

Cheers
  Nick
Comment 17 Dongsheng Song 2011-04-15 01:49:32 UTC
(In reply to comment #16)
> Ho Dongsheng,
> 
>   Hmm, I am not sure what else to suggest.  You could try turning off section
> garbage collection (--no-gc-sections) as this appears to be turned on.
> 
>   If you really have 1.2 Gb of object files then I am not sure if you can
> reasonably expect the linker to be anything but slow when you only 1G of real
> memory available.  I would strongly urge trying the link on a bigger machine.
> 
> Cheers
>   Nick

I do not care about the real time, I'm only aware the CPU (user + sys) time.

I don not think when I link object files to multiple .DLL files only use no more than 10 minutes, but link these objects files to single .DLL file not finished after 6 hours CPU time is acceptable.
Comment 18 Dongsheng Song 2011-04-15 05:50:28 UTC
If I link wxWidgets 2.8 into multiple .DLL files, the link speed is very fast, here is the details (the last 2 columns were reported by ld '--stats'):

DLL SIZE       FILE NAME                LD TIME     LD DATA SIZE
   1991K   wxbase28u_gcc.dll              42.56s    524M
    214K   wxbase28u_net_gcc.dll           0.29s     49M
    194K   wxbase28u_odbc_gcc.dll          0.20s     32M
    244K   wxbase28u_xml_gcc.dll           0.15s     18M
    831K   wxmsw28u_adv_gcc.dll            1.06s    102M
    481K   wxmsw28u_aui_gcc.dll            0.62s     82M
   5028K   wxmsw28u_core_gcc.dll         349.60s   1694M
    109K   wxmsw28u_dbgrid_gcc.dll         0.18s     39M
    108K   wxmsw28u_gl_gcc.dll             0.22s     44M
    632K   wxmsw28u_html_gcc.dll           0.95s     78M
    190K   wxmsw28u_media_gcc.dll          0.29s     32M
    166K   wxmsw28u_qa_gcc.dll             0.24s     28M
   1226K   wxmsw28u_richtext_gcc.dll       1.64s    102M
    571K   wxmsw28u_xrc_gcc.dll            0.68s     60M

  11740K            -                    398.68s   2884M

The total time is about 400s, why when they link into single .DLL file use so much memory (4.6GB) ? and not finished after 6 hours CPU (sys + user) time ?
Comment 19 Dongsheng Song 2011-04-15 05:55:52 UTC
(In reply to comment #18)
> If I link wxWidgets 2.8 into multiple .DLL files, the link speed is very fast,
> here is the details (the last 2 columns were reported by ld '--stats'):
> 
> DLL SIZE       FILE NAME                LD TIME     LD DATA SIZE
>    1991K   wxbase28u_gcc.dll              42.56s    524M
>     214K   wxbase28u_net_gcc.dll           0.29s     49M
>     194K   wxbase28u_odbc_gcc.dll          0.20s     32M
>     244K   wxbase28u_xml_gcc.dll           0.15s     18M
>     831K   wxmsw28u_adv_gcc.dll            1.06s    102M
>     481K   wxmsw28u_aui_gcc.dll            0.62s     82M
>    5028K   wxmsw28u_core_gcc.dll         349.60s   1694M
>     109K   wxmsw28u_dbgrid_gcc.dll         0.18s     39M
>     108K   wxmsw28u_gl_gcc.dll             0.22s     44M
>     632K   wxmsw28u_html_gcc.dll           0.95s     78M
>     190K   wxmsw28u_media_gcc.dll          0.29s     32M
>     166K   wxmsw28u_qa_gcc.dll             0.24s     28M
>    1226K   wxmsw28u_richtext_gcc.dll       1.64s    102M
>     571K   wxmsw28u_xrc_gcc.dll            0.68s     60M
> 
>   11740K            -                    398.68s   2884M
> 
> The total time is about 400s, why when they link into single .DLL file use so
> much memory (4.6GB) ? and not finished after 6 hours CPU (sys + user) time ?

PS: I checked the total object files, they are 999,944KB.
Comment 20 sunworld 2011-11-04 05:13:02 UTC
One question: when memory exhausted during linking, can it use swap file instead? I means to use hardisk to compensate the lack of memory space. Here it doesn't care about the linking speed. It on earth the memory is not cheap.

Thanks

-Sunworld
Comment 21 Nick Clifton 2011-11-07 15:13:41 UTC
Hi Sunworld,

> One question: when memory exhausted during linking, can it use swap file
> instead?

No and yes ...  "no" as in, "no the linker cannot be changed to use the 
swap file when it runs out of memory", but "yes" as in "yes, the swap 
file will be used if there is insufficient memory available".

Ie using the swap file is not a feature that needs to be added to the 
linker.  Rather it is a feature that is already present in the operating 
system and one which is used automatically.

Cheers
   Nick
Comment 22 Nick Clifton 2011-11-07 15:24:24 UTC
Hi Dongshen

> The total time is about 400s, why when they link into single .DLL file use so
> much memory (4.6GB) ? and not finished after 6 hours CPU (sys + user) time ?

I do not know, and without a way to reproduce the problem myself, I cannot find
out.  Sorry, but all I can suggest is that you either track down where the
memory is being used up yourself, or else try using a machine with more
physical memory on which to link the DLL.

Cheers
  Nick
Comment 23 Kai Tietz 2012-02-13 11:53:15 UTC
Hmm, does the gcc option '-fno-keep-inline-dllexport' help here?
Comment 24 CauseLove 2013-02-20 13:46:54 UTC
Me helped flag --reduce-memory-overheads