debugedit: switch to xxhash for buildid recomputation
When debugedit recomputes build-id (due to -i), it previously used md5
or sha1 (depending on length of incoming buildid). This patch
replaces both those (including the code that does all the hashing)
with a 128-bit xxhash. The 128-bit hash is truncated or padded to
whatever the incoming ELF note width was.
xxhash is much faster (8x over sha1) than either of those
crypto-flavoured hashes, and still produces fairly collision-free
values. This was confirmed informally with a bulk build-id
recomputation of a few million binaries in the debuginfod server
corpus, yielding zero unexpected collisions.
Mark Wielaard [Sat, 28 Sep 2024 21:39:29 +0000 (23:39 +0200)]
debugedit: Handle DWARF5 DW_UT_type (Unit type 2) in ET_EXE/DYN
Handle DW_UT_type as if it is a DW_UT_compile unit by skipping the
type id and offset in the header. Which are the only differences with
DW_UT_compile. And the id and offset don't need to be rewritten. This
doesn't handle debug types in objects or partial linked (ET_REL) files
that contain COMDAT sections (because debugedit doesn't handle more
than one debug_info section). Add a testcase for foobarbaz.exe.
Keith Seitz [Fri, 16 Aug 2024 18:54:20 +0000 (11:54 -0700)]
find-debuginfo.sh: Exit with real exit status in parallel jobs
Currently, when the script is executed in parallel (-jN), the
resulting exit status will always be 0.
The script execs an appropriate number of clones of itself, calling
run_job to run the actual workload. This then calls do_file(), saving
the exit status into "res.$jobid".
In do_file(), though, if an error occurs, exit is called. This causes
the entire exec'd shell to exit with status 0 (since there are almost
always echo calls as the last executed statement). The real exit
status is therefor never written to the "res.$jobid" files by run_job().
The simple solution is to use 'return' instead of 'exit'. A number
of minor adjustments are also made to propagate this properly so that
it is reported as the correct exit status.
While at it, I've incorporated a patch for find-debuginfo/30505.
Using this patch and another patch to the RPM package (submitted as
github issue #3215), failures of gdb-add-index.sh will now properly fail
the build instead of being swallowed. It should be much easier for
developers to figure out why their builds have failed should gdb crash.
Bug: https://sourceware.org/bugzilla/show_bug.cgi?id=30505 Signed-off-by: Keith Seitz <keiths@redhat.com>
Mark Wielaard [Sun, 19 May 2024 19:59:15 +0000 (21:59 +0200)]
debugedit: Handle DW_MACRO_{define,undef}_strx
For DW_MACRO_{define,undef}_strx we need to record the string index of
the macro. Since this is read through the .debug_str_offets we need to
know the relevant str_offsets_base of the CU associated with the macro
table. Add a macros_offs field to struct CU. Set this in when seeing a
DW_AT_macros. And make sure relocations against .debug_macros are
resolved when handling ET_REL files.
Now all (macro) tests pass with CC=clang even when clang defaults to
-gdwarf-5. The .debug_types tests are skipped because clang doesn't
emit that section.
Mark Wielaard [Sat, 18 May 2024 22:18:02 +0000 (00:18 +0200)]
debugedit: Track active CU
The current CU is tracked indirectly by having static variables for
ptr_size, cu_version and str_offsets_base. These are used to
process/skip forms and resolve strx forms. Put all three in a new
struct CU which must be passed to do_read_str_form_relocated,
skip_form, edit_strp and edit_attributes. The struct CU can then also
be used when parsing the line table string forms.
This should make .debug_line tables which use strx forms work (no
producers have been found that use those though).
Mark Wielaard [Sat, 18 May 2024 23:14:51 +0000 (01:14 +0200)]
Simplify update_rela_data by checking rel_updated
Now that we track whether relocations have been updated per section we
can just check the section rel_updated field in update_rela_data. This
also makes sure the relbuf is always freed.
Mark Wielaard [Sun, 5 May 2024 12:46:31 +0000 (14:46 +0200)]
Make relocation reading explicit
Relocation handling used to be linear per section. But since handling
.debug_str_offsets (which can have relocations itself) we had to
"reset" the relocation handling when switching between reading
sections. This caused a lnear search from the start of the relocation
table each time section reading was switched. Since this was slow
do_read_32_relocated needed to be optimized using a binary search
(commit fbad879eb).
Since we do a binary search now, resetting the relocation
datastructure isn't really needed. It is also a little confusing and
fragile. Rewrite the relocation code so it works per section and
doesn't need "switching".
Make the relocation related data structures part of struct
debug_section. And change do_read_32_relocated to take a struct
debug_section. This way we make explicit from which section we are
reading.
debugedit is currently very slow when processing DWARF 5 debuginfo
produced by clang. For some kernel modules, debugedit processing
takes hours.
The root cause of the issue is the loop for finding the correct
REL entry in do_read_32_relocated. This is currently a simple
linear scan. For large objects, it may loop for hundreds of
thousands of iterations.
As the relocations are sorted, we can use a binary search instead,
which is what this patch implements. The time to run debugedit on
a large kernel module (nouveau.ko) drops down to 3 seconds with
this change.
Mark Wielaard [Mon, 18 Mar 2024 22:37:47 +0000 (23:37 +0100)]
debugedit: Only write the ELF file when updating strings or build-id
Only open the ELF file read/write and write out the data if we
actually did any elf structure update or updating the build-id.
* tools/debugedit.c (fdopen_dso): Call elf_begin with
ELF_C_READ when not changing dest_dir or build_id,
otherwise use ELF_C_RDWR.
(main): Call open with O_RDONLY when not changing dest_dir
or build_id, otherwise use O_RDWR. Call elf_update with
ELF_C_WRITE when rewriting any string, updating any ELF
structure or build_id.
Mark Wielaard [Thu, 23 Mar 2023 17:07:40 +0000 (18:07 +0100)]
debugedit: Add support for .debug_str_offsets (DW_FORM_strx)
In theory supporting strx .debug_str_offsets is easy, the strings in
.debug_str are just read through an indirection table. When the
strings are updated in .debug_str we just need to rewrite the
indirection table.
The tricky part is the ET_REL (object files or kernel modules)
support. Relocation reading is "global" per section and we expect to
read a relocation only once. But we need to read the
DW_AT_str_offsets_base before reading any strx form attributes. So we
read that first, then reset the relptr. And when we read from the
.debug_str_offsets section we need to save and restore the .debug_info
relptr.
* tools/debugedit.c (do_read_24): New function.
(str_offsets_base): New static variable.
(buf_read_ule24): New function.
(buf_read_ube24): Likewise.
(setup_relbuf): Handle .debug_str_offsets.
(do_read_uleb128): New function.
(do_read_str_form_relocated): Likewise.
(read_abbrev): Handle DW_FORM_strx[1234].
(edit_strp): Take the actual string form as argument.
Use do_read_str_form_relocated.
(read_dwarf5_line_entries): Pass form to edit_strp.
(edit_attributes_str_comp_dir): Take the actual string
form as argument. Use do_read_str_form_relocated.
(edit_attributes): Handle DW_FORM_strx[1234].
(edit_info): Read DW_AT_str_offsets_base first.
(update_str_offsets): New function.
(edit_dwarf2): Setup do_read_24. Call update_str_offsets.
The LFS64 function calls have been gated behind the _LARGEFILE64_SOURCE
macro in musl 1.2.4, and will be removed altogether in musl 1.2.5. Since
configure.ac has the AC_SYS_LARGEFILE macro, which ensures that all
functions on 32-bit glibc systems will be 64-bit, just replace the
interfaces with their normal variants.
Denys Vlasenko [Thu, 22 Jun 2023 13:31:03 +0000 (15:31 +0200)]
find-debuginfo: Add -v,--verbose for per file messages
Only print messages what big steps we are at without --verbose.
For a reader of rpmbuild's log, it's rather unclear what find-debuginfo
is doing. It used to be too verbose, "extracting debug info from FILE"
for every file, and while this can be suppressed now, we still end up
with something semi-mysterious like this:
...
extracting debug info from /builddir/build/BUILDROOT/xyz
gdb-add-index: No index was created for /builddir/build/BUILDROOT/xyz
gdb-add-index: [Was there no debuginfo? Was there already an index?]
symlinked /usr/lib/debug/usr/lib64/libcpupower.so.0.0.1.debug to /usr/lib/debug/usr/lib64/libcpupower.so.debug
symlinked /usr/lib/debug/usr/lib64/libcpupower.so.0.0.1.debug to /usr/lib/debug/usr/lib64/libcpupower.so.0.debug
cpio: binutils-2.30/bfd: Cannot stat: No such file or directory
cpio: binutils-2.30/bfd/aout-target.h: Cannot stat: No such file or directory
cpio: binutils-2.30/bfd/aoutx.h: Cannot stat: No such file or directory
cpio: binutils-2.30/bfd/archive.c: Cannot stat: No such file or directory
cpio: binutils-2.30/bfd/archive64.c: Cannot stat: No such file or directory
...
775655 blocks
+ /usr/lib/rpm/check-buildroot
+ /usr/lib/rpm/redhat/brp-ldconfig
...
The reader is left confused. "What these cpio errors are about?
Why those sources are not found?" (Well, because not every source
name extracted by 'debugedit -l' has to exist, but this requires
considerable digging aroung to understand).
We can give a few messages explaining what general steps we go through:
Extracting debug info from N files
DWARF-compressing N files
Creating .debug symlinks for symlinks to ELF files
Copying sources found by 'debugedit -l'
This is also useful to get a feeling which steps are time consuming.
Kernel builds often need to investigate this aspect. To help a bit more,
add "find-debuginfo: starting" and "find-debuginfo: done" messages too.
This patch adds these messages.
The -q options suppress these messages too.
It also adds a --verbose flag to print per file messages.
Those per file messages are now suppressed by default and
only the general step messages are show. Unless -q is given,
which suppresses all non-error output.
Signed-off-by: Denys Vlasenko <dvlasenk@redhat.com> Signed-off-by: Mark Wielaard <mark@klomp.org>
Mark Wielaard [Thu, 29 Jun 2023 12:26:39 +0000 (14:26 +0200)]
find-debuginfo: Prefix install_dir to PATH
Some distros install a symlink to find-debuginfo[.sh] under the
old /usr/lib/rpm/ path. But don't have symlinks there for other
helper tools like debugedit and sepdebugcrcfix. So those tools
are then not under the install_dir as find-debuginfo calculates
it. So instead of invoking those helper tools with a full install
dir path, prefix the install dir to the PATH so the tools can be
found there (or anywhere else on the PATH).
Denys Vlasenko [Wed, 14 Jun 2023 14:56:38 +0000 (16:56 +0200)]
find-debuginfo: remove duplicate filenames when creating debugsources.list
We remove duplicate filenames when we _process_ debugsources.list.
However, this means that momentarily we may have a very large
(in the range of *giga*bytes) debugsources.list.
This is unnecessary, we can also remove dups when we *create* it.
Mark Wielaard [Fri, 24 Mar 2023 23:49:36 +0000 (00:49 +0100)]
debugedit: Simplify and extend .debug_line tests
The debugedit .debug_line tests were only checking the directory
table, skipping the file name table assuming those would not be
include a directory element. But some compilers output the file
names as absolute paths (so ignoring the directory table entries,
which is slightly inefficient). Check both tables now using sed
to replace the directory strings.
Prarit Bhargava [Thu, 26 Jan 2023 21:08:57 +0000 (16:08 -0500)]
scripts/find-debuginfo.in: Add -q|--quiet
Projects with a large number of compiled files end up with a large number
of 'extracting debug info from' messages in the build log. In the case of
the Fedora kernel these messages account for 8504 lines in the log, or 61%
of the entire log [1].
Removing these lines make the log easier to view and comprehend for some
projects, however, not all projects will want to silence these messages so
suppressing them must be optional.
Add a -q|--quiet which allows users to silence the non-error output from
the script.
Mark Wielaard [Fri, 27 Jan 2023 14:23:15 +0000 (15:23 +0100)]
configure.ac: Use AC_LINK_IFELSE for -gz=none check
Because of a bug in gcc 13.0 (pre-release) -gz=none does not work
correctly when linking the final binary in some situations (when
-gz=zstd isn't supported). Work around this by explicitly checking for
support with AC_LINK_IFELSE.
Kalev Lember [Thu, 19 Jan 2023 16:03:18 +0000 (17:03 +0100)]
find-debuginfo: Pass -j down to dwz
Now that dwz 0.15 supports parallel jobs, add a way to control it from
here. find-debuginfo already has a -j parameter so we can just extend it
and pass the value down to dwz as well.
This should fix building large packages on memory constrained builders,
such as webkitgtk on s390x in Fedora koji build system, where we can now
use the -j option to tune down parallelism to avoid running out of
memory during dwz run.
Add a configure check to make sure the installed dwz supports the
-j option.
Signed-off-by: Kalev Lember <klember@redhat.com> Signed-off-by: Mark Wielaard <mark@klomp.org>
Mark Wielaard [Wed, 28 Dec 2022 21:23:01 +0000 (22:23 +0100)]
debugedit: Skip calling edit_dwarf2 if not rewriting and/or listing source
We skipped calling edit_dwarf2 when no base and dest dir were given and
debugedit was invoked with -i -n. But we never need to call edit_dwarf2
if we aren't rewriting paths and we don't want to list the source files.
Mark Wielaard [Wed, 20 Apr 2022 22:05:38 +0000 (00:05 +0200)]
debugedit: Use standard libelf elf_strptr
The strptr function in debugedit.c does the same thing as libelf
elf_strptr. But elf_strptr handles bounds checks and invalid section
offsets better. And elf_strptr handles compressed sections.
* tools/debugedit.c (strptr): Just call elf_strptr.
Mark Wielaard [Fri, 30 Jul 2021 16:09:46 +0000 (18:09 +0200)]
tests: Handle zero directory entry in .debug_line DWARF5 debugedit.at
We were skipping the zero directory entry, because it was always
the same as the directory entry at position one. But that isn't
true anymore with gcc 11.2.1. There the zero dir entry is unique.
Fix the debugedit.at .debug_line testcases using DWARF5 to also
include dir entry zero.
Mark Wielaard [Mon, 26 Jul 2021 22:46:35 +0000 (00:46 +0200)]
debuginfo-find: simplify RPM env handling
As pointed out by Dmitry V. Levin the RPM env checking used redundant
trailing semicolons and wasn't somewhat verbose. Replace the three
checks with simpler for loop check over all env variables we are
expecting.
Mark Wielaard [Mon, 26 Jul 2021 22:01:15 +0000 (00:01 +0200)]
find-debuginfo: Check RPM environment variables are there
find-debuginfo relies on a couple of RPM environment variables.
Ideally we provide command line arguments to set them. But they are
somewhat tied to how rpm sets things up. So for now just warn and
exit if they aren't set.
See also https://sourceware.org/bugzilla/show_bug.cgi?id=27637
Mark Wielaard [Mon, 5 Jul 2021 16:35:04 +0000 (18:35 +0200)]
Record and write files in read_dwarf5_line_entries.
Without giving -d /foo/build/bar debugedit wouldn't actually output
any file from the line number table for DWARF5.
* tools/debugedit.c (read_dwarf5_line_entries): Don't make
collecting_dirs and writing_files depend on dest_dir. Always
read idx for strp or line_strp on phase 0. comp_dir_len is
zero when comp_dir is NULL.
* tests/debugedit.at: Add source list mode tests for DWARF4
and DWARF5.
Mark Wielaard [Thu, 1 Jul 2021 09:58:38 +0000 (11:58 +0200)]
debugedit: Skip relocations with missing symbol/section offset.
We tried to handle relocations that didn't have a symbol associated
with any section. The would cause a message like: "Unhandled
relocation 1 in .debug_info section". Which wasn't that helpful
either. So skip relocations without an associated symbol section index
and improve the error message a little.
* debugedit.c (setup_relbuf): Continue when sym.st_shndx == 0.
Add relocation index to error message.
Mark Wielaard [Thu, 1 Jul 2021 12:50:04 +0000 (14:50 +0200)]
debugedit: Handle realloc failure correctly.
Although we are already in trouble when realloc fails don't make it
worse. When realloc fails we have to free the original pointer we
wanted to extend to not cause an even bigger memory leak.
* tools/debugedit.c (read_abbrev): Free original t after realloc
fails.
Mark Wielaard [Wed, 19 May 2021 11:41:26 +0000 (13:41 +0200)]
find-debuginfo: Generate and install as top-level build file.
find-debuginfo relied on lib_rpm_dir to find other debugedit tools.
But the script itself was generated at configure time in a subdir.
This makes testing the (non-installed) build slightly inconvenient.
Add a automake rule to generate find-debuginfo at build time, instead
of configure time, in the top-level builddir next to the other tools.
* .gitignore: Move scripts/find-debuginfo to find-debuginfo.
* configure.ac: Explicitly check we have SED.
* Makefile.am (bin_SCRIPTS): Drop subdir scripts.
(CLEANFILES): New.
(do_subst): New.
(find-debuginfo): New generate using do_subst.
(find-debuginfo.1): Adjust location and protect against
parallel builds.
(EXTRA_DIST): Add .in suffix to scripts/find-debuginfo.
* scripts/find-debuginfo.in (lib_rpm_dir): Rename to...
(install_dir): ...this.
Mark Wielaard [Wed, 19 May 2021 10:39:05 +0000 (12:39 +0200)]
tests: Remove RPM strings
The debugedit.at still contained some references to RPM even though it
doesn't depend on RPM anymore. Remove RPM from the banner and from the
DEBUGEDIT_SETUP m4 macro.
* tests/debugedit.at (AT_BANNER): Remove RPM from the name.
(RPM_DEBUGEDIT_SETUP): Rename to...
(DEBUGEDIT_SETUP): ...this.
Mark Wielaard [Tue, 18 May 2021 17:03:31 +0000 (19:03 +0200)]
Rename find-debuginfo.sh to find-debuginfo
Even though it is currently a bash script we install it as a normal
program. So simply install it without the .sh suffix. To get the VERSION
right in the script make it an AC_CONFIG_FILE (from find-debuginfo.in).
This also fixes the man page name and version.
* .gitignore: Remove find-debuginfo.sh.1.
Add scripts/find-debuginfo and find-debuginfo.1
* configure.ac (AC_CONFIG_FILES): Add scripts/find-debuginfo.
* Makefile.am (bin_SCRIPTS): Rename find-debuginfo.sh to
find-debuginfo.
(dist_man_MANS): Rename find-debuginfo.sh.1 to find-debuginfo.1.
(find-debuginfo.sh.1): Rename to ...
(find-debuginfo.1): ...this and update invocation.
(EXTRA_DIST): Rename scripts/find-debuginfo.sh to
scripts/find-debuginfo.
* scripts/find-debuginfo.sh: Rename to ...
* scripts/find-debuginfo.in: ...this.
Change all find-debuginfo.sh to find-debuginfo.
Use @VERSION@ for --version.
Mark Wielaard [Wed, 5 May 2021 21:33:14 +0000 (23:33 +0200)]
Makefile.am: Don't try to recursively make binaries to run help2man
We try to avoid having to run help2man unnecessary for a build from
a source distribution (which include the generated manpages). We did
this by not depending on the actual executable, but recursively calling
make to generate it when necessary. This causes trouble with parallel
makes because the two make processes don't know which objects have,
have not or are being build. Fix this by depending on the executable
but only running help2man if one of the sources has changed, not when
the executable has been regenerated.
Mark Wielaard [Wed, 5 May 2021 19:22:29 +0000 (21:22 +0200)]
automake: Add std-options to check --version and --help work as intended.
Now that we rely on --version and --help to generate documentation make
sure that make distcheck checks those standard options work correctly.
Also adjust find-debuginfo.sh[.1] rules slightly to make sure they the
scripts are also checked (bin_SCRIPTS are, but dist_bin_SCRIPTS aren't).
* configure.ac (AM_INIT_AUTOMAKE): Add std-options.
* Makefile.am (dist_bin_SCRIPTS): Change into...
(bin_SCRIPTS): ... this, using top_srcdir.
(TOP_CONFIGURE_AC): New variable, with comment explaining when
.1 man pages are (re)generated.
(debugedit.1): Use TOP_CONFIGURE_AC as dependency.
(sepdebugcrcfix.1): Likewise.
(find-debuginfo.sh.1): Likewise and depend on top_srcdir script.
Mark Wielaard [Thu, 29 Apr 2021 20:44:41 +0000 (22:44 +0200)]
sepdebugcrcfix: Add --version, --help and man page.
Add --version and --help support to sepdebugcrcfix. Use this to generate
a manual page using help2man.
* .gitignore: Add sepdebugcrcfix.1
* Makefile.am (dist_man_MANS): Add sepdebugcrcfix.1
(sepdebugcrcfix.1): New make rule.
* tools/sepdebugcrcfix.c (version): New static functions.
(help): Likewise.
(main): Call version or help depending on argc and argv.
Mark Wielaard [Thu, 29 Apr 2021 16:11:06 +0000 (18:11 +0200)]
debugedit: Add manual using help2man
This generates a debugedit.1 man page using help2man based on the
debugedit --help output. The generated file is included in the dist
so help2man isn't normally needed to build debugedit.
debugedit: consistently use error() instead of fprintf(stderr)
The error() function is already used in debugedit for printing some
of diagnostic messages, extend this practice to all such messages.
The user-visible side of this change is that all diagnostic messages
are now consistently prefixed.
* tools/debugedit.c (edit_attributes, handle_build_id, main): Replace
fprintf(stderr) with error().
debugedit: strip extra newline characters from diagnostic messages
Since the output produced by error() is already terminated by a newline
character, no newline terminating characters are needed in diagnostic
messages printed using error().
* tools/debugedit.c (main): Remove terminating newline characters from
format strings passed to error().
Ivan A. Melnikov [Fri, 13 Nov 2020 11:51:08 +0000 (15:51 +0400)]
debugedit: add MIPS support
According to the specification[1], all MIPS .debug_* sections are
marked with ELF type SHT_MIPS_DWARF. The format of the section
data stays the same, so we have can handle e.g. .debug_info
section as we used to.
As SHT_MIPS_DWARF is from processor-specific range, we have to check
that we're actually dealing with MIPS ELF file before handling such
sections.
[1] MIPS Extensions to DWARF Version 2.0. -- Silicon Graphics
Computer Systems, rev 1.17, 29 Aug 2001
Refs: https://bugzilla.altlinux.org/39284 Signed-off-by: Ivan A. Melnikov <iv@altlinux.org> Signed-off-by: Dmitry V. Levin <ldv@altlinux.org>
Mark Wielaard [Thu, 25 Mar 2021 23:00:42 +0000 (00:00 +0100)]
tests: Use CC, CFLAGS, LD and LDFLAGS to create testcases
Currently gcc is hardcoded in the tests. Use the configured tools to
make testing for a specfic target or alternative build tools easier.
* configure.ac: Add AC_CHECK_TOOL for LD.
* tests/atlocal.in: Set CC, CFLAGS, LD and LDFLAGS.
* tests/debugedit.at (RPM_DEBUGEDIT_SETUP): Use CC, CFLAGS, LD
and LDFLAGS.
Mark Wielaard [Wed, 24 Mar 2021 23:23:43 +0000 (00:23 +0100)]
tests: Check gcc accepts -gz=none before usage
commit 4f138 "tests: fix for toolchains producing compressed debug
sections" used -gz=none unconditionally to compile the test
binaries. But not all gcc versions support -gz.
Add a configure check for -gz=none. Set GZ_NONE_FLAG in
atlocal.in. Use it instead of -gz=none in debugedit.at.
Dmitry V. Levin [Mon, 22 Mar 2021 20:00:00 +0000 (20:00 +0000)]
debugedit: fix exit status in case of wrong number of arguments
Print error diagnostics to stderr and exit with status EXIT_FAILURE
when the number of arguments is not equal to 1.
* tools/debugedit.c (usage): Add "error" argument, print usageText
to stderr and exit with status EXIT_FAILURE if error is set to true.
All callers updated.
* tests/debugedit.at: Check debugedit usage.
Fixes: 5200953d5a65 ("Initial build system. Remove rpm and popt dependencies.") Signed-off-by: Dmitry V. Levin <ldv@altlinux.org>
Mark Wielaard [Mon, 15 Mar 2021 09:14:49 +0000 (10:14 +0100)]
Initial build system. Remove rpm and popt dependencies.
- Add automake, autoconf and autotest files.
- Replace popt with getopt argument parsing.
- Use md5 or sha1 instead of rpmDigest algorithms for build-id updates.
Mark Wielaard [Sun, 14 Mar 2021 15:56:04 +0000 (16:56 +0100)]
Add COPYING files for imported code
Code for debugedit has been imported that was derived from binutils,
libiberty and rpm some use different licenses, but all are upward
compatible with GPLv3+.
COPYING (GPLv2+) covers scripts/find-debuginfo.sh, all files under
tests, tools/hashtab.h, tools/sha1.c, tools/md5.{c,h},
tools/ansidecl.h and tools/debugedit.c.
COPYING3 (GPLv3+) covers tools/sepdebugcrcfix.c and tools/sha1.h
Mark Wielaard [Mon, 15 Feb 2021 11:29:26 +0000 (12:29 +0100)]
debugedit: Add DWARF5 tests
Adjust some existing tests so they are run with an explicit -gdwarf-4
and add an -gdwarf-5 variant to make testing independent of the gcc
default DWARF version. The tests that might generate a DWARF5 line
table work for both version 4 and 5 and some also ignore stderr output
when using -p.debug_line_str because some combinations of gcc/binutils
generate DWARF5 debug info with DWARF4 debug line tables, even when
-gdwarf-5 is given.
Tested against GCC10, which defaults to DWARF4 and GCC11, which defaults
to DWARF5.
Mark Wielaard [Mon, 15 Feb 2021 11:29:25 +0000 (12:29 +0100)]
debugedit: Handle DWARF-5 debug_line and debug_line_str.
Handle the new DWARF5 .debug_line tables and the new DW_FORM_line_strp.
DWARF5 tables are handled separately from the earlier tables. They
will never change size, but they do need updates to the .debug_str
or .debug_line_str references.
Based on a patch from Jan Kratochvil <jan.kratochvil@redhat.com>
Jan Kratochvil [Mon, 15 Feb 2021 11:29:24 +0000 (12:29 +0100)]
debugedit: Implement DWARF-5 unit header and new forms parsing.
Recognize the various new DWARF5 .debug sections.
Parse and skip new DWARF5 forms in read_abbrev and skip_form.
Read DWARF5 unit headers for compile and partial units in edit_info.
This is enough to be able to process gcc -gdwarf-5 produced binaries
without the new DWARF5 .debug_line format (which isn't produced with
binutils < 2.36).
Patches slightly edited/merged by Mark Wielaard <mark@klomp.org>
Jan Kratochvil [Mon, 15 Feb 2021 11:29:23 +0000 (12:29 +0100)]
debugedit: Move code to separate functions.
New functions edit_strp, skip_form and edit_attributes_str_comp_dir
called by edit_attributes.
Split part of read_dwarf2_line into a read_dwarf4_line function.
New function edit_dwarf2_any_str called by edit_dwarf2 at the end of
phase 0 to rebuild .debug_str.
Vitaly Chikunov [Sat, 14 Nov 2020 19:32:44 +0000 (22:32 +0300)]
debugedit: Distinguish files from directories in src list file
Append '/' to directories in source file list (for `-l' option) to allow
quickly distinguish them from regular files (to avoid adding them raw
into %files section). This is needed for ALT for our debuginfo
processing to speed things up.
Co-authored-by: Dmitry V. Levin <ldv@altlinux.org>
Vitaly Chikunov [Sat, 14 Nov 2020 15:14:28 +0000 (18:14 +0300)]
debugedit: Do not 'edit_dwarf2' when just extracting build-id
No need to call edit_dwarf2() if debugedit is invoked just to extract
build-id (with `-i -n'). Otherwise, we will get `DWARF version 0
unhandled' warning for compressed .debug files:
/usr/lib/rpm/debugedit: ./usr/lib/debug/...e.ko.debug: DWARF version 0 unhandled
Context: We have kernel modules elfcompressed in ALT.
ggardet [Fri, 9 Oct 2020 11:10:09 +0000 (13:10 +0200)]
Do not fail if there is no "$temp"/res.* file
find-debuginfo is multi-threaded and rpm runs
scripts usually with "-e" to abort on error.
If the debug-splitting tool fails, that job will abort.
But if you have X files that are problematic in the tree,
and you have X or less jobs, find-debuginfo will abort
as no single res.$number has been written.
But if you have more than X jobs, the build will succeed,
which makes the whole process random.
This commit remove this randomness.
Panu Matilainen [Thu, 11 Jun 2020 10:52:28 +0000 (13:52 +0300)]
Execute tests in independent testing roots to enable parallel running
Up to now we've run tests in a mish-mash of an environment where bits
and pieces of previous tests might exist, and as state has been shared
it's been impossible to execute tests in parallel.
This makes the pre-populated testing-root read-only to the owner to force
all writers to perform an extra setup call to gain a private testing
environment. Most of the users needing this already had RPMDB_INIT calls
to ensure clean rpmdb state so that's where this is hooked onto.
There's a fair bit of gymnastics with the environment to make things match
on both sides of fakechroot, some of which can hopefully go away eventually
once the dust from this settles. This is also rather heavy as it is,
on my laptop serial execution goes down from ~1m15s to ~1m45s, but then
parallel execution with -j8 is down to ~50s. There should be a number of
optimizations that can be made, such as setting up links for writable
directories instead of copying the entire testing-tree around, but leaving
that as a future exercise. This is more of an enabler than the goal state.
Note that this does *not* enable test-suite parallel running for
`make -jN check`, because `make` does not export the -j argument in
a way that we could pass to `./rpmtests` from the makefile. To enable
that, one needs to pass a suitable -jN argument through TESTSUITEFLAGS,
such as `make -j8 check TESTSUITEFLAGS="-j8"`. Or run invoke `rpmtests`
manually.
Denys Vlasenko [Thu, 5 Dec 2019 12:30:17 +0000 (13:30 +0100)]
find-debuginfo.sh: speed up %dir generation
For kernel build, "${LISTFILE}.dirs.sed" is debugfiles.list.dirs.sed
and it contains 1782 lines of sed script.
It is applied to two files, both are about 4450 lines long.
This is slow (~30 seconds) because of ~16 million regex matches
performed by sed.
But we don't need or want regex match here
(and it's buggy, since dots in pattern match will be treated
as "any character", which is wrong here. For example,
/usr/lib/debug/lib/modules/5@4@0-0@rc7@0@fc31@test@x86_64/vdso/
would match
/usr/lib/debug/lib/modules/5.4.0-0.rc7.0.fc31.test.x86_64/vdso/
pattern, but it should not).
This change performs matching using shell string comparison ops.
For kernel build, this change results in run time of about one second.
Michal Čihař [Sat, 11 Nov 2017 13:27:10 +0000 (14:27 +0100)]
Fix sepdebugcrcfix compilation on platforms without MAP_POPULATE
Not all architectures offer MAP_POPULATE. As MAP_POPULATE is only an
optimization to improve performance, it is safe to drop it when it is
unavailable. Originally fixed in a Debian patch.
Tom de Vries [Mon, 26 Aug 2019 09:30:31 +0000 (11:30 +0200)]
Remove unused variable id in find-debuginfo.sh
There's code setting variable id in scripts/find-debuginfo.sh, but there's no
subsequent use. The only use of $id in the script is in do_file, where it is
a local variable.
The variable setting was introduced in commit fe280a20b "Add dwz debuginfo
compression support" in combination with a subsequent make_id_link using the
variable, but the make_id_link was removed in commit 62807f0df
"Add build-id links to rpm for all ELF files".
Eliminate use of ambiguous logical operators in script conditionals
Prefer '[] && []' to '[ -a ]' and '[] || []' to '[ -o ]' in tests.
-a and -o to mean AND and OR in a [ .. ] test expression is not well
defined, and can cause incorrect results when arguments start with
dashes or contain !. Moreover binary -a and -o are inherently
ambiguous. test(1) man page recommends to use
'test EXPR1 && test EXPR2' or 'test EXPR1 || test EXPR2' instead.
Mark Wielaard [Mon, 17 Jun 2019 09:23:26 +0000 (11:23 +0200)]
debugedit: Make sure .debug_line old/new idx start equal.
Found by running the debugedit tests under valgrind.
If the old and new .debug_line offset isn't changed then we might
write out an uninitialized new_idx.
Mark Wielaard [Mon, 17 Jun 2019 09:23:25 +0000 (11:23 +0200)]
Handle .debug_macro in debugedit.
When compiling with -g3 gcc will generate a .debug_macro section
which has pointers to the .debug_str section. Since we might rewrite
the .debug_str section, we also need to update any .debug_macro
pointers.
Updated the debugedit.at testcase by building everything with -g
and add various checks to see the .debug_macro section looks OK
after running debugedit. Added a new rpmbuild.at testcase to check
handing of .debug_macro in the whole rpmbuild debuginfo pipeline
to double check the separate .debug file also contains the macros.
Original patch by Michael Schroeder <mls@suse.de>. Extended by
Mark Wielaard <mark@klomp.org> to deal with relocations and possible
multiple COMDAT .debug_macro sections.
Mark Wielaard [Mon, 17 Jun 2019 09:23:24 +0000 (11:23 +0200)]
debugedit: Refactor reading/writing of relocated values.
This refactors the reading and writing of relocated values into seperate
helper functions (setup_relbuf and update_rela_data). It will be easier
to reuse this code in case we want to read/write relocated values in other
sections than DEBUG_INFO. The only functional change is that we explicitly
track whether the relocation data is updated, and only explicitly update
and write out the relocation data if so. In the case there were no strp
or stmt updates, there will also not be any relocation updates, even if
there is relocation data available.
All new debugedit testcases pass before and after this refactoring.
Mark Wielaard [Mon, 17 Jun 2019 09:23:23 +0000 (11:23 +0200)]
Add some debugedit tests.
debugedit didn't have any direct tests, it was only tested indirectly
through some other debuginfo testcases. So add some testcases that
just test debugedit functionality directly.
The tests create different kinds of ELF files (object files, partially
linked files and executables) and run debugedit path replacements on
them, checking that the resulting .debug_str, .debug_info and .debug_line
sections look sane.
Testcases verified against various different gcc and binutils versions.
gcc 4.8 and gcc 6 generate slightly different .debuginfo
Also add debugedit --version. Which makes it easy to see we are
testing the correct version/install of debugedit in rpmtests.log.
Adjust autotest paths to include rpmlibexecdir, where debuginfo
is installed. Note that rpmlibexecdir != libexecdir (see rpm.am).
nickclifton [Wed, 19 Jun 2019 11:45:09 +0000 (12:45 +0100)]
Add step to find-debuginfo.sh script to compress annobin notes.
With the introduction of the annobin gcc plugin to the build process,
built binary files have become larger. Sometimes significantly so.
This is a patch that adds a new step to the post-link process performed
by rpmbuild, to run the objcopy program with the --merge-notes option
specified. This will reduce the size of the annobin notes in binary
files, thus alleviating the size growth.
git://sourceware.org / debugedit.git / log