PING: [PATCH v12 0/6] elf: Use mmap to map in section contents

[H.J. Lu]

On Sun, Mar 17, 2024 at 5:19 AM H.J. Lu <hjl.tools@gmail.com> wrote:
>
> Changes in v12:
>
> 1. Don't call _bfd_elf_link_keep_memory for x86 since it always returns
> false.
> 2. Make _bfd_elf_link_keep_memory static.
>
> Changes in v11:
>
> 1. Update _bfd_mmap_read_temporary to allocate buffer as needed.
> 2. Don't allocate buffer when _bfd_mmap_read_temporary is called.
>
> Changes in v10:
>
> 1. Malloc a 4 * page size buffer for external relocations for the final
> link to avoid mmap/munmap on small relocation sections.
> 2. Malloc a buffer for input section contents which are smaller than
> 4 * page size or can't be mapped for the final link.
>
> Changes in v9:
>
> 1. Use MAP_FAILED for mmap failure.
>
> Changes in v8:
>
> 1. Rebase against master branch.
> 2. Add _bfd_elf_link_mmap_section_contents and
> _bfd_elf_link_munmap_section_contents.
>
> Changes in v7:
>
> 1. Don't add the --keep-memory linker option.
>
> Changes in v6:
>
> 1. Add the --keep-memory linker option and always cache symbol and
> relocation tables for --keep-memory.
> 2. Always keep symbol table and relocation info for eh_frame to speedup
> the Rust binary build by ~300x:
>
> https://sourceware.org/bugzilla/show_bug.cgi?id=31466
>
> Changes in v5:
>
> 1. Drop 2 patches which have been merged onto master branch.
> 2. Rename _bfd_elf_mmap_section to _bfd_elf_mmap_section_contents.
> 3. Rename _bfd_mmap_readonly_tracked, _bfd_mmap_readonly_untracked,
> _bfd_munmap_readonly_untracked, _bfd_mmap_read_untracked to
> _bfd_mmap_readonly_persistent, _bfd_mmap_readonly_temporary,
> _bfd_munmap_readonly_temporary and _bfd_mmap_read_temporary.
> 4. Drop the setup_group change.
> 5. Fix a typo.
> 6. Update comments.
>
> Changes in v4:
>
> 1. Change don't cache symbol nor relocation tables with mmap to opt-in.
>
> Changes in v3:
>
> 1. Fix non-mmap build.
> 2. Change the argument name of bfd_mmap_local from flags to prot since
> its values are PROT_XXX.
>
> Changes in v2:
>
> 1. Don't hard-code BFD_JUMP_TABLE_COPY in bfd so that elf-bfd.h can be
> included in libbfd.c.
> 2. Change the --with-mmap default to true.
> 3. Check USE_MMAP instead of HAVE_MMAP.
> 4. Remove the asize parameter to _bfd_mmap_readonly_tracked.
> 5. Add contents_addr and contents_size to bfd_elf_section_data.
> 6. Rename _bfd_link_keep_memory to _bfd_elf_link_keep_memory.
>
> ---
> We can use mmap to map in ELF section contents, instead of copying them
> into memory by hand.  We don't need to cache symbol nor relocation tables
> if they are mapped in.  Data to link the 3.5GB clang executable in LLVM
> 17 debug build on Linux/x86-64 with 32GB RAM is:
>
>                 stdio           mmap            improvement
> user            86.73           87.02           -0.3%
> system          9.55            9.21            3.6%
> total           100.40          97.66           0.7%
> maximum set(GB) 17.34           13.14           24%
> page faults     4047667         3042877         25%
>
> and data to link the 275M cc1plus executable in GCC 14 stage 1 build is:
>
> user            5.41            5.44            -0.5%
> system          0.80            0.76            5%
> total           6.25            6.26            -0.2%
> maximum set(MB) 1323            968             27%
> page faults     323451          236371          27%
>
> Data shows that these won't improve the single copy linker performance.
> But they improve the overall system performance when linker is used by
> reducing linker memory usage and page faults.  They allow more parallel
> linker jobs on LLVM debug build.
>
> Here is a quote from Noah Goldstein: "on a large project they are an
> extremely large speedup".
>
> H.J. Lu (6):
>   elf: Use mmap to map in read-only sections
>   elf: Add _bfd_elf_m[un]map_section_contents
>   elf: Use mmap to map in symbol and relocation tables
>   elf: Don't cache symbol nor relocation tables with mmap
>   elf: Always keep symbol table and relocation info for eh_frame
>   elf: Add _bfd_elf_link_m[un]map_section_contents
>
>  bfd/bfd-in2.h      |  24 ++++-
>  bfd/bfd.c          |  17 +++
>  bfd/bfdwin.c       |   8 +-
>  bfd/cache.c        |   7 +-
>  bfd/compress.c     |   2 +-
>  bfd/elf-bfd.h      |  24 +++++
>  bfd/elf-eh-frame.c |   4 +-
>  bfd/elf-sframe.c   |   4 +-
>  bfd/elf.c          | 246 ++++++++++++++++++++++++++++++++++--------
>  bfd/elf32-i386.c   |   8 +-
>  bfd/elf64-x86-64.c |  12 +--
>  bfd/elfcode.h      |   7 +-
>  bfd/elflink.c      | 213 ++++++++++++++++++++++++------------
>  bfd/elfxx-target.h |   6 +-
>  bfd/elfxx-x86.c    |   8 +-
>  bfd/elfxx-x86.h    |   1 +
>  bfd/libbfd-in.h    |  33 +++++-
>  bfd/libbfd.c       | 262 ++++++++++++++++++++++++++++++++++++++++++++-
>  bfd/libbfd.h       |  33 +++++-
>  bfd/linker.c       |  35 ------
>  bfd/lynx-core.c    |   2 +-
>  bfd/opncls.c       |  21 ++++
>  bfd/section.c      |   9 +-
>  bfd/sysdep.h       |   4 +
>  24 files changed, 795 insertions(+), 195 deletions(-)
>
> --
> 2.44.0
>

PING.

-- 
H.J.