RFC: Closing memory leaks to reduce the BFD linker's memory profile

[Nick Clifton]

Hi Guys,

  One of the problems with the BFD linker is that it uses a lot of
  memory.  Especially when compared to other linkers.  This issue came
  to light recently in an unexpected way when the Envoy Proxy project
  discovered that they could not switch from ld.gold to ld.bfd because
  the CI system used by GitHub uses instances with a restricted amount
  of memory and these instances were crashing during testing.

  So I have been investigating the memory usage of the linker, looking
  for easy ways to reduce the memory profile.  One thought that did come
  to mind is that the linker leaks a lot of memory.  If this memory
  could be released as soon as it was no longer needed then there is a
  good chance that the overall memory profile of the linker would be
  reduced.

  A quick check with valgrind showed that one of the largest leaks comes
  from the wildcard matching code in ldlang.c:

==1431789== 88,703,000 bytes in 88,703 blocks are still reachable in loss record 117 of 118
==1431789==    at 0x49C3B26: malloc (vg_replace_malloc.c:447)
==1431789==    by 0x4AD93C3: xmalloc (xmalloc.c:149)
==1431789==    by 0x4051E26: call_chunkfun (obstack.c:94)
==1431789==    by 0x4051E26: _obstack_newchunk (obstack.c:206)
==1431789==    by 0x400EF42: stat_alloc (ldlang.c:189)
==1431789==    by 0x4020D1D: new_statement (ldlang.c:1186)
==1431789==    by 0x4020D1D: add_matching_section (ldlang.c:460)
==1431789==    by 0x4020D1D: walk_wild_section_match (ldlang.c:543)
==1431789==    by 0x4020D1D: resolve_wild_sections.part.0.lto_priv.0 (ldlang.c:1024)
==1431789==    by 0x400D524: resolve_wild_sections (ldlang.c:1008)
==1431789==    by 0x400D524: resolve_wilds.lto_priv.0 (ldlang.c:1047)
==1431789==    by 0x401EA48: lang_process (ldlang.c:8555)
==1431789==    by 0x400382D: main (ldmain.c:882)

  (This result comes from linking Clang's llc executable.  A nice big
  test...)

  So I created a patch that frees the memory when it is no longer
  needed.  (See attached in case you are interested).  Since freeing the
  memory involves walking over lists of section patterns however, and
  the BFD linker is already the slowest of them all, I decided that the
  new code should only be enabled when --reduce-memory-overheads is in
  operation.

  With this patch in place the leak is closed and the memory use as
  reported by valgrind goes from:

       in use at exit: 481,834,373 bytes in 3,572,602 blocks
    to:      
       in use at exit: 336,141,893 bytes in 27,250 blocks

  However the total amount of memory allocations jumps by a huge amount:

      total heap usage: 10,491,826 allocs, 6,919,224 frees, 17,986,487,171 bytes allocated
    to:
      total heap usage: 13,225,597 allocs, 13,198,347 frees, 793,738,870,333 bytes allocated

  And the time taken for the compilation goes from:

      Time spent in user mode   (CPU seconds) : 323.386
    to:  
      Time spent in user mode   (CPU seconds) : 3887.720

  Some this is presumably because other effects caused by using the
  --reduce-memory-overheads command line option to enable the leak
  closure.  But overall I am wondering whether this is actually a useful
  exercise or if I should be looking elsewhere for memory savings.

Cheers
  Nick

-------------- next part --------------
A non-text attachment was scrubbed...
Name: ld.lang.memory.leak.patch
Type: text/x-patch
Size: 2662 bytes
Desc: not available
URL: <https://sourceware.org/pipermail/binutils/attachments/20260109/35af2b08/attachment-0001.bin>