Previous versions of dwz-tests.exp emit the full path names to the
per-test .sh scripts into the .sum/.log files. This version replaces
those patn names with just the $basename, which is enough to identify
the tests uniquely, and allows proper regression/comparison analysis.
Frank Ch. Eigler [Sat, 18 Feb 2023 11:36:15 +0000 (12:36 +0100)]
make dejagnu logs more elaborate
dwz-tests.exp produces totally barebones dwz.log files, basically
duplicating the .sum file. If there are any errors, there's
basically no info to go on.
The patch runs the .sh subtests in -x trace mode, and logs
stdout/stderr to the .log file, so e.g. bunsen can safe-keep it.
Otherwise I believe no-op.
Mark Wielaard [Thu, 3 Nov 2022 22:08:50 +0000 (23:08 +0100)]
Check -DXXH_INLINE_ALL works, otherwise link against libxxhash
Because we don't have a real configure script this does a Makefile
shell trick trying to compile with XXH_INLINE_ALL defined when
including xxhash.h. If that fails (like on oldstable debian) it
doesn't add -DXXH_INLINE_ALL=1 to CFLAGS and does add -lxxhash
to LIBS.
Mark Wielaard [Fri, 1 Jul 2022 23:11:00 +0000 (01:11 +0200)]
Redirect stder in gdb-add-index.sh test
gdb-add-index might produce an error message on stderr when trying to
disable debuginfod support. Any message to stderr makes the testcase
fail. This looks like a gdb bug:
https://sourceware.org/bugzilla/show_bug.cgi?id=29316
But it is easy to workaround by redirecting stderr to stdout.
Mark Wielaard [Thu, 30 Jun 2022 12:17:32 +0000 (14:17 +0200)]
Always assign svalue when checking and reporting negative values
GCC12 warns:
In file included from /usr/include/error.h:59,
from dwz.c:23:
In function ‘error’,
inlined from ‘checksum_die’ at dwz.c:3364:7:
/usr/include/bits/error.h:42:5: warning: ‘svalue’ may be used
uninitialized [-Wmaybe-uninitialized]
42 | __error_alias (__status, __errnum, __format, __va_arg_pack ());
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
dwz.c: In function ‘checksum_die’:
dwz.c:3354:25: note: ‘svalue’ was declared here
3354 | int64_t svalue = read_sleb128 (ptr);
| ^~~~~~
It is correct. We jump to the common error reporting label negative
where we expect svalue to be set so we can print it in the error
message. But it isn't in all places. Fix this by explicitly assigning
svalue first in all places we goto negative on failure.
* dwz.c (checksum_die): Always assign svalue when
checking and reporting on unexpected negative values.
Mark Wielaard [Sun, 21 Feb 2021 15:55:17 +0000 (16:55 +0100)]
Print abbrev or DIE offset for Unknown DWARF error message.
* dwz.c (read_abbrev): Add .debug_abbrev offset to error message.
(read_exprloc): Print DIE offset that referenced the unknown
operand in error message.
(read_expr_low_mem_phase1): Likewise.
(read_debug_info): Add die_offset to error messages for unknown
forms, attributes extending beyond end of CU or unknown block
form attributes.
Mark Wielaard [Thu, 30 Jun 2022 12:03:31 +0000 (14:03 +0200)]
Workaround binutils readelf following and printing alt file.
Even with -wN readelf 2.38-15.fc37 follows and prints the
contents of the alt file. Workaround that by removing the
alt file in tests when we are done with it. This should be
fine even when binutils readelf gets fixed to not do that
with -wN (but maybe this is by design?) This fixes the
testsuite on rawhide.
Mark Wielaard [Thu, 30 Jun 2022 11:28:26 +0000 (13:28 +0200)]
Mark and init shstrtab[_len] and const in optimize_multifile
Old gcc (4.8.5) warn that shstrtab and shstrtab_len can be clobbered
by the longjump. Work around that by marking shstrtab_gnu and
shstrtab_dwarf5 as static and shstrtab and shstrtab_len as const.
Init shstrtab and shstrtab_len at declaration.
* dwz.c (optimize_multifile): Make shstrtab_[gnu|dwarf5],
shstrtab[_len] static. shstrtab[_len] init and make const.
Lv Ying via Dwz [Wed, 12 Jan 2022 21:29:44 +0000 (05:29 +0800)]
Fix handling readelf following links by default
The two testcases still failed when using newer readelf
(binutils commit c46b706620e):
testsuite/dwz.tests/odr-struct-multifile.sh
testsuite/dwz.tests/cycle.sh
Tom de Vries [Mon, 12 Apr 2021 19:50:23 +0000 (21:50 +0200)]
Add -p native and -e native
Add option parameter native to options -p and -e.
We determine native as the result of readelf output on an object generated
by using CC without CFLAGS, such that if we build dwz with -m32 on
x86_64 like so:
...
$ make CFLAGS="-m32 -O2 -g" LDFLAGS=-m32
...
and we have:
...
$ file ./dwz
dwz: ELF 32-bit LSB executable, Intel 80386 <SNIP>
...
we still have:
...
$ ./dwz -?
...
-p, --multifile-pointer-size <SIZE|auto|native>
Set pointer size of multifile, in number of bytes.
Native pointer size is 8.
Default value: auto.
...
2021-04-12 Tom de Vries <tdevries@suse.de>
* Makefile (args.o): Add dependency on native.o.
(CFLAGS_FOR_SOURCE, NATIVE_ENDIAN, NATIVE_ENDIAN_LITTLE)
(NATIVE_ENDIAN_BIG, NATIVE_ENDIAN_VAL): New var.
(%.o: %.c): Add pattern rule.
(clean): Update.
(native.o): Add pattern rule.
* util.h (XSTR, STR): New macro.
* args.c (NATIVE_ENDIAN): New macro.
(dwz_multi_file_options_help, usage): Mention -p native and -e native.
(parse_args): Handle -p native and -e native.
* dwz.1 (-p, -e): Mention native.
* native.c: New file.
Tom de Vries [Fri, 9 Apr 2021 15:06:24 +0000 (17:06 +0200)]
Eliminate sed usage in building dwz-for-test
We started out building dwz-for-test using -U__GNUC__. That resulted
in problems with system headers on fedora, so instead a solution using
sed was implemented. This works well and ensures that all __GNUC__ uses
are addressed, but it makes refactoring more complicated.
Instead, use the slightly more error-prone but less cumbersome method
of replacing uses of __GNUC__ with uses of a new variable USE_GNUC,
and define USE_GNUC=0 for dwz-for-test.
2021-04-09 Tom de Vries <tdevries@suse.de>
* Makefile: Compile dwz-for-test using -DUSE_GNUC=0.
* dwz.c (USE_GNUC): Weakly define if __GNUC__ is defined, and
use instead of __GNUC__.
Tom de Vries [Fri, 9 Apr 2021 12:00:16 +0000 (14:00 +0200)]
Ensure help message lines end with dot
I noticed a missing dot in the help message, after "parallel":
...
./dwz -?
...
-j, --jobs <n> Process <n> files in parallel
Default value: number of processors / 2.
...
Add an assert that catches this, and add the missing dot.
2021-04-09 Tom de Vries <tdevries@suse.de>
* args.c (IMPLIES): New macro, copied from dwz.c.
(dwz_multi_file_options_help): Add missing dot.
(print_options_help): Assert that help messages end with dot.
Tom de Vries [Wed, 7 Apr 2021 04:16:01 +0000 (06:16 +0200)]
[testsuite] Fix handling of missing section in smaller-than.sh
On openSUSE Tumbleweed I run into:
...
Running testsuite/dwz.tests/dwz-tests.exp ...
testsuite/scripts/smaller-than.sh: line 18: 16#: \
invalid integer constant (error token is "16#")
...
FAIL: testsuite/dwz.tests/dwz.sh
...
The relevant line in smaller-than.sh is:
...
s=$(printf "%d" $((16#$s)))
...
This seems to be due bash (version 5.1.4), which gives:
...
$ echo $((16#))
bash: 16#: invalid integer constant (error token is "16#")
...
while bash on Leap 15.2 (version 4.4.23) gives:
...
$ echo $((16#))
0
...
Fix this by explicitly handling s == "" in proc section_size.
2021-04-07 Tom de Vries <tdevries@suse.de>
PR dwz/27694
* testsuite/scripts/smaller-than.sh (section_size): Handle s == "".
Tom de Vries [Wed, 31 Mar 2021 07:09:58 +0000 (09:09 +0200)]
Enable parallel multifile with -e -p
Currently, parallel dwz is disabled when multifile is used:
...
$ dwz -m 5 3 1 2 4 -j 4
...
Enable this when the multifile parameter characteristics are specified using
-p and -e:
...
$ dwz -m 5 3 1 2 4 -j 4 -p 8 -e l
...
This works around the child processes having to communicate back to the parent
the found pointer size and endiannes, and doing the -j auto and -e auto
consistency checking.
The problem of the different child processes writing to the same temporary
multifile is solved by writing in file order to the temporary multifile.
In principle, the problem could be solved by writing into per-file temporary
multifiles and concatenating them afterwards. However, the temporary
multifile contains DW_FORM_ref_addr references, and after concatenation
those references in the temporary multifile require relocation (i.e., add the
offset of the start of the file contribution). This requires a lot of
changes in various parts of the code, so for now we choose instead this
cleaner solution.
The enforcing of writing in file order is done by passing a token to each
child process using pipes.
Tom de Vries [Mon, 29 Mar 2021 08:14:05 +0000 (10:14 +0200)]
Apply multifile_force_{ptr_size,endian} ASAP
Rather than waiting for multi_ptr_size/multi_endian to be set in
write_multifile, apply multifile_force_{ptr_size,endian} ASAP, in
dwz_files_1. This is a preparation for running write_multifile
in parallel.
2021-03-29 Tom de Vries <tdevries@suse.de>
* dwz.c (dwz_files_1): Apply multifile_force_ptr_size and
multifile_force_endian.
Tom de Vries [Mon, 29 Mar 2021 06:24:33 +0000 (08:24 +0200)]
Add ret to struct file_result
Further simplify the serial/parallel code in dwz_files_1 by factoring out the
return status handling. Add a ret field to struct file_result to store the
per-file return status.
2021-03-29 Tom de Vries <tdevries@suse.de>
* dwz.c (struct file_result): Add ret field.
(wait_child_exit): Return void. Set ret.
(wait_children_exit): Return void. Update call to wait_child_exit.
(dwz_files_1): Move return status handling out of serial/parallel code.
Tom de Vries [Mon, 29 Mar 2021 06:24:33 +0000 (08:24 +0200)]
Fix incorrect res arg for wait_child_exit
In dwz_files_1 we have:
...
file = files[i];
struct file_result *res = &resa[i];
if (nr_forks == max_forks)
{
int thisret = wait_child_exit (-1, pids, i, res);
...
The -1 pid argument to wait_child_exit means that any child status will be
returned, while the res argument passes the result array element for
the file we want to start processing, which is incorrect.
Fix this by passing the result array instead, and letting wait_child_exit
figure out which element to use.
2021-03-29 Tom de Vries <tdevries@suse.de>
* dwz.c (wait_child_exit): Find appropriate element in result array.
(dwz_files_1): Use resa when using pid -1.
Tom de Vries [Sat, 27 Mar 2021 19:13:35 +0000 (20:13 +0100)]
Calculate workset before serial/parallel code in dwz_files_1
The function dwz_files_1 has code that can execute either serially or in
parallel. Evidently, there's code duplication between the serial and
parallel code, which creates the burden of keeping this in sync. The
burden can be kept minimal by keeping the duplicate code minimal.
Part of the duplicate code is the selection which files to handle.
Move this selection out of the serial/parallel code, and use it to compute
an array workset, which is then used in both the serial and parallel code.
2021-03-27 Tom de Vries <tdevries@suse.de>
* dwz.c (dwz_files_1): Calculate workset before serial/parallel code.
Tom de Vries [Sat, 27 Mar 2021 12:53:58 +0000 (13:53 +0100)]
Use goto in dwz_files_1
Use goto in dwz_files_1 to remove indentation, replacing:
...
if (...)
ret = 1;
else
{
/* Lots of code. */
}
/* Cleanup. */
...
with:
...
if (...)
{
ret = 1;
goto cleanup;
}
Tom de Vries [Sat, 27 Mar 2021 12:53:58 +0000 (13:53 +0100)]
Initialize pids in dwz_files_1
The loop dealing with waiting on the remaining children contains:
...
if (res->res == -2)
/* Skip hard links. */
continue;
...
The reason we need this is because the pids array isn't initialized in this
case, so the following code:
...
if (pids[i] == 0)
continue;
...
would otherwise read an uninitialized variable.
Fix this by:
- initializing the pids array, and
- removing the res->res == -2 code.
Tom de Vries [Fri, 26 Mar 2021 13:28:19 +0000 (14:28 +0100)]
Run two-files-low-mem-die-limit-0.sh with -j1
On the buildbot we have for dwz-debian-arm64:
...
child process exited abnormally
FAIL: build/testsuite/dwz.tests/two-files-low-mem-die-limit-0.sh
...
AFAIU, the problem is that the test-case is grepping like this:
...
egrep -q "Compressing (1|2)$" dwz.err
...
expecting the grep to fail on these messages:
...
Compressing 1 in low-mem mode
Compressing 2 in low-mem mode
...
But with -j 2 we can have the following interleaving of the messages:
...
Compressing 1 in low-mem modeCompressing 2
in low-mem mode
...
which makes the grep succeed.
Fix this by running the test with -j1.
2021-03-26 Tom de Vries <tdevries@suse.de>
* testsuite/dwz.tests/two-files-low-mem-die-limit-0.sh: Run dwz
with -j1.
Tom de Vries [Fri, 26 Mar 2021 10:45:17 +0000 (11:45 +0100)]
Process files in parallel
Add an option -j <n> / --jobs <n> that allows multiple files to be processed
in parallel. Note that this does not yet parallelize when multifile is used.
Consider the experiment do.sh:
...
ns=$(seq 1 10)
for n in $ns; do
cp debug/cc1 $n
done
time.sh ./dwz $@ -lnone $ns
...
On a 4 smt-thread, dual core system we get:
...
$ for n in $(seq 1 4); do echo "N: $n"; ./do.sh -j $n; done
N: 1
maxmem: 1261744
real: 56.53
user: 53.24
system: 3.17
N: 2
maxmem: 1260216
real: 31.74
user: 58.84
system: 4.39
N: 3
maxmem: 1261868
real: 28.26
user: 75.65
system: 5.00
N: 4
maxmem: 1262036
real: 26.80
user: 87.31
system: 5.69
...
The sweet spot of real time reduction vs. extra user/system time seems to be
around 2, so we set the default -j to processors / 2.
2021-03-23 Tom de Vries <tdevries@suse.de>
PR dwz/25951
* args.c (max_forks): New var.
(dwz_options, dwz_multi_file_options_help, usage): Add entries for
-j <n>.
(parse_args): Handle -j <n>. Assign default value for max_forks.
* args.h (max_forks): Declare.
* dwz.1: Add entries for -j <n> / --jobs <n> entry.
* dwz.c (dwz_files_1): Handle max_forks.
Tom de Vries [Fri, 26 Mar 2021 09:11:54 +0000 (10:11 +0100)]
Add --multifile-pointer-size <n> and --multifile-endian <l|L|b|B>
Consider binaries:
- hello.64 generated using -m64, with pointer size 8,
- hello.32 generated using -m32, with pointer size 4.
When trying to generate a multifile using files with different
pointer sizes, we get:
...
$ cp ../hello.64 1; cp 1 2; \
cp ../hello.32 3; cp 3 4; \
dwz -m 5 1 2 3 4; echo $?
dwz: Multi-file optimization not allowed for different pointer sizes \
or endianity
0
...
and the multi-file optimization has not been applied for any file:
...
$ for f in 1 2 3 4; do \
echo -n "$f: "; \
readelf -S -W $f \
| grep -c gnu_debugaltlink; \
done
1: 0
2: 0
3: 0
4: 0
...
Add an option --multifile-pointer-size <n> / -p <n> that sets the pointer size
of the multifile, such that we have instead with say -p 8:
...
$ cp ../hello.64 1; cp 1 2; \
cp ../hello.32 3; cp 3 4; \
./dwz -m 5 -p 8 1 2 3 4; echo $?
./dwz: File 3 skipped for multi-file optimization, different pointer size
./dwz: File 4 skipped for multi-file optimization, different pointer size
0
...
and:
...
$ for f in 1 2 3 4; do \
echo -n "$f: "; \
readelf -S -W $f \
| grep -c gnu_debugaltlink; \
done
1: 1
2: 1
3: 0
4: 0
...
Tom de Vries [Thu, 25 Mar 2021 13:52:04 +0000 (14:52 +0100)]
Fix uninitialized var in dwz_one_file
The commit dbfecee "Move hardlink handling out of dwz function" added
initialization of res->res in dwz_files_1, but failed to do the same
in dwz_one_file.
Add the missing initialization by factoring out the initialization code from
dwz_files_1, and using it in dwz_one_file.
2021-03-25 Tom de Vries <tdevries@suse.de>
* dwz.c (init_file_result): Factor out of ...
(dwz_files_1): ... here.
(dwz_one_file): Use init_file_result.
Tom de Vries [Thu, 25 Mar 2021 08:38:27 +0000 (09:38 +0100)]
Move hardlink handling out of dwz function
Currently hardlink handling is done in the dwz function, on per-file basis,
with the analysis having a scope of previously processed files.
Move hardlink handling out of dwz into dedicated functions detect_hardlinks
and update_hardlinks, both called from dwz_files_1.
The detect_hardlinks is called before any file is processed, the
update_hardlinks is called after all files are processed.
This allows parallelization of the processing of the files.
2021-03-23 Tom de Vries <tdevries@suse.de>
* dwz.c (struct file_result): Add comment about res == -3 as
uninitialized. Add hardlink_to field.
(detect_hardlinks, update_hardlinks): New function, factored out of ...
(dwz): ... here. Drop resa and files parameters.
(dwz_with_low_mem): Drop resa and files parameters.
(dwz_one_file): Update call to dwz_with_low_mem.
(dwz_files_1): Update call to dwz_with_low_mem. Add calls to
detect_hardlinks and update_hardlinks.
Tom de Vries [Wed, 24 Mar 2021 17:05:20 +0000 (18:05 +0100)]
Fix memory leak in write_multifile
When building dwz with -fsanitize=address, we run into some memory leaks:
...
$ cp hello 1; ./dwz 1; cp 1 2; ./dwz -m 3 1 2
./dwz: 1: DWARF compression not beneficial - old size 3372 new size 3372
./dwz: 2: DWARF compression not beneficial - old size 3372 new size 3372
Direct leak of 432 byte(s) in 6 object(s) allocated from:
#0 0x7f7ab92cc6d8 in __interceptor_calloc \
(/usr/lib64/libasan.so.4+0xdc6d8)
#1 0x475dc1 in htab_try_create hashtab.c:164
#2 0x44c610 in build_abbrevs dwz.c:11230
#3 0x44ee00 in compute_abbrevs dwz.c:11510
#4 0x46f07c in dwz dwz.c:15396
#5 0x474e0d in dwz_files_1 dwz.c:16327
#6 0x475699 in dwz_files dwz.c:16417
#7 0x4759fc in main dwz.c:16458
#8 0x7f7ab8c41349 in __libc_start_main (/lib64/libc.so.6+0x24349)
Indirect leak of 2928 byte(s) in 6 object(s) allocated from:
#0 0x7f7ab92cc6d8 in __interceptor_calloc \
(/usr/lib64/libasan.so.4+0xdc6d8)
#1 0x475de7 in htab_try_create hashtab.c:168
#2 0x44c610 in build_abbrevs dwz.c:11230
#3 0x44ee00 in compute_abbrevs dwz.c:11510
#4 0x46f07c in dwz dwz.c:15396
#5 0x474e0d in dwz_files_1 dwz.c:16327
#6 0x475699 in dwz_files dwz.c:16417
#7 0x4759fc in main dwz.c:16458
#8 0x7f7ab8c41349 in __libc_start_main (/lib64/libc.so.6+0x24349)
SUMMARY: AddressSanitizer: 3360 byte(s) leaked in 12 allocation(s).
...
A more concrete way to show some of the leaks is by using this patch on
build_abbrevs:
...
+ assert (cu->cu_new_abbrev == NULL);
cu->cu_new_abbrev = h;
return 0;
}
...
which triggers here:
...
#4 0x00000000004211b1 in build_abbrevs (cu=0x7ffff65f0e38, t=0x64e640,
ndies=0x7fffffffd81c, vec=0x649280 <ob2>) at dwz.c:11239
#5 0x0000000000421db2 in compute_abbrevs (dso=0x0) at dwz.c:11511
#6 0x000000000042fa2c in write_multifile (dso=0x64c210)
at dwz.c:15104
...
We could just do a htab_delete in build_abbrevs, which takes care of all the
leaks detected by the assert.
But write_multifile drops CUs from the cu list if
cu->cu_die->die_no_multifile == 1, and if a dropped CU has cu->cu_new_abbrev
!= NULL then cu_new_abbrev is still leaked.
Fix this by calling htab_delete in the loop that drops the CUs.
2021-03-24 Tom de Vries <tdevries@suse.de>
PR dwz/27643
* dwz.c (write_multifile): Clean up cu->cu_new_abbrev to fix memory
leak.
Direct leak of 432 byte(s) in 6 object(s) allocated from:
#0 0x7f560af846d8 in __interceptor_calloc (/usr/lib64/libasan.so.4+0xdc6d8)
#1 0x475db1 in htab_try_create hashtab.c:164
#2 0x406ed6 in read_abbrev dwz.c:1296
#3 0x42bca9 in read_debug_info dwz.c:6818
#4 0x4608cb in read_dwarf dwz.c:13706
#5 0x46ef9a in dwz dwz.c:15383
#6 0x474a27 in dwz_one_file dwz.c:16279
#7 0x475951 in main dwz.c:16450
#8 0x7f560a8f9349 in __libc_start_main (/lib64/libc.so.6+0x24349)
Direct leak of 72 byte(s) in 1 object(s) allocated from:
#0 0x7f560af846d8 in __interceptor_calloc (/usr/lib64/libasan.so.4+0xdc6d8)
#1 0x475db1 in htab_try_create hashtab.c:164
#2 0x42a8fd in read_debug_info dwz.c:6634
#3 0x4608cb in read_dwarf dwz.c:13706
#4 0x46ef9a in dwz dwz.c:15383
#5 0x474a27 in dwz_one_file dwz.c:16279
#6 0x475951 in main dwz.c:16450
#7 0x7f560a8f9349 in __libc_start_main (/lib64/libc.so.6+0x24349)
Indirect leak of 4072 byte(s) in 1 object(s) allocated from:
#0 0x7f560af846d8 in __interceptor_calloc (/usr/lib64/libasan.so.4+0xdc6d8)
#1 0x475dd7 in htab_try_create hashtab.c:168
#2 0x42a8fd in read_debug_info dwz.c:6634
#3 0x4608cb in read_dwarf dwz.c:13706
#4 0x46ef9a in dwz dwz.c:15383
#5 0x474a27 in dwz_one_file dwz.c:16279
#6 0x475951 in main dwz.c:16450
#7 0x7f560a8f9349 in __libc_start_main (/lib64/libc.so.6+0x24349)
Indirect leak of 2928 byte(s) in 6 object(s) allocated from:
#0 0x7f560af846d8 in __interceptor_calloc (/usr/lib64/libasan.so.4+0xdc6d8)
#1 0x475dd7 in htab_try_create hashtab.c:168
#2 0x406ed6 in read_abbrev dwz.c:1296
#3 0x42bca9 in read_debug_info dwz.c:6818
#4 0x4608cb in read_dwarf dwz.c:13706
#5 0x46ef9a in dwz dwz.c:15383
#6 0x474a27 in dwz_one_file dwz.c:16279
#7 0x475951 in main dwz.c:16450
#8 0x7f560a8f9349 in __libc_start_main (/lib64/libc.so.6+0x24349)
SUMMARY: AddressSanitizer: 7504 byte(s) leaked in 14 allocation(s).
...
The leaks are related to the meta_abbrev_htab, which is allocated in
the initial read_debug_info call for .debug_info, and then allocated once more
in a second call to read_debug_info for .debug_types.
The second allocation overwrites the first one, and consequently the first one
is leaked.
Fix this by only allocating meta_abbrev_htab if not already allocated.
Tom de Vries [Wed, 24 Mar 2021 09:10:07 +0000 (10:10 +0100)]
[testsuite] Show error output in twice-multifile.sh
When building dwz with -fsanitize=address, we get:
...
$ make check RUNTESTFLAGS=dwz-tests.exp=twice-multifile.sh
Running src/testsuite/dwz.tests/dwz-tests.exp ...
child process exited abnormally
FAIL: src/testsuite/dwz.tests/twice-multifile.sh
...
The test-case fails because "dwz -m 3 1 2" returns 1, but we don't see
why because the error output is redirected to dwz.err.
Fix this by dumping dwz.err if "dwz -m 3 1 2" status is not zero.
2021-03-24 Tom de Vries <tdevries@suse.de>
* testsuite/dwz.tests/twice-multifile.sh: Dump dwz.err if dwz status
is not 0.
Tom de Vries [Tue, 23 Mar 2021 10:25:01 +0000 (11:25 +0100)]
Factor out dwz_files_1
In dwz_files, a variable resa is malloced at the start, and freed at the end.
This prevents (early) returns, which makes the code more complicated than
necessary. Also, the function already contains an early return, which means
it leaks resa.
Fix this by factoring out dwz_files_1 out of dwz_files, where dwz_files is a
wrapper tasked with alloc/free of variable resa.
2021-03-23 Tom de Vries <tdevries@suse.de>
* dwz.c (dwz_files_1): New function, factored out of ...
(dwz_files): ... here.
Tom de Vries [Mon, 22 Mar 2021 05:13:41 +0000 (06:13 +0100)]
[testsuite] Remove smaller-than.sh test in pr24747.sh
As reported in PR27603, when using clang-11 we run into:
...
Running testsuite/dwz.tests/dwz-tests.exp ...
+ exec=start-gold
+ cp start-gold 1
+ dwz 1
+ smaller-than.sh 1 start-gold
FAIL: testsuite/dwz.tests/pr24747.sh
...
The FAIL as such has been fixed by commit b3c99ab "Make smaller-than.sh match
dwz heuristic", but the only reason that dwz manages to optimize something is
because the dwarf producer is lazy and doesn't generate minimal encodings.
We cannot assume that the dwarf producer is lazy, so remove the
smaller-than.sh test.
Tom de Vries [Mon, 22 Mar 2021 04:49:54 +0000 (05:49 +0100)]
Make smaller-than.sh match dwz heuristic
As reported in PR27603, when using clang-11 we run into:
...
Running testsuite/dwz.tests/dwz-tests.exp ...
+ exec=start-gold
+ cp start-gold 1
+ dwz 1
+ smaller-than.sh 1 start-gold
FAIL: testsuite/dwz.tests/pr24747.sh
...
The files are the same size:
...
$ ls -l start-gold tmp.pr24747.sh/1
-rwxr-xr-x 1 vries users 2024 Mar 18 11:58 start-gold
-rwxr-xr-x 1 vries users 2024 Mar 18 11:58 tmp.pr24747.sh/1
...
but not the same:
...
$ diff -q start-gold tmp.pr24747.sh/1
Files start-gold and tmp.pr24747.sh/1 differ
...
The .debug_info size is smaller, by 6 bytes:
...
[Nr] Name Type Address Off Size ES Flg Lk Inf Al
-[ 6] .debug_info PROGBITS 00000000 0001cd 000040 00 0 0 1
+[ 6] .debug_info PROGBITS 00000000 0001cd 00003a 00 0 0 1
...
The size reduction in the .debug_info section happens not to result in a
smaller file because subsequent sections .note.gnu.gold-version and .symtab
have a bigger-than-one align.
We could try to fix this by adapting the heuristic to not compress if the
eventual file size is equal.
Instead, fix this by making the smaller-than.sh test match the heuristic,
which only looks at the size of some .debug sections.
2021-03-22 Tom de Vries <tdevries@suse.de>
* testsuite/scripts/smaller-than.sh: Calculate size as sum of
.debug sections.
Tom de Vries [Sat, 20 Mar 2021 08:37:31 +0000 (09:37 +0100)]
[testsuite] Handle readelf following links by default
Since binutils commit c46b706620e "Change the readelf and objdump programs so
that they will automatically follow links to separate debug info files" there
are a few FAILs in the dwz testsuite.
This is just due to changing the default behaviour for readelf.
Fix this by testing whether the new -Wn readelf flag is supported,
and if so, adding it to the readelf call.
2021-03-20 Tom de Vries <tdevries@suse.de>
PR dwz/27592
* testsuite/dwz.tests/odr-struct-multifile.sh: Call readelf with -wN
if supported.
* testsuite/dwz.tests/cycle.sh: Same.
Tom de Vries [Wed, 17 Mar 2021 16:33:17 +0000 (17:33 +0100)]
[testsuite] Fix pr27463.sh on riscv64
On riscv64, I run into:
...
cc main.c unavailable-dwarf-piece-dw.S -o unavailable-dwarf-piece
unavailable-dwarf-piece-dw.S: Assembler messages:
unavailable-dwarf-piece-dw.S:24: Error: non-constant .uleb128 is not supported
...
make: *** [Makefile:100: unavailable-dwarf-piece] Error 1
...
Fix this similar to commit 53c0488 "[testsuite] Fix pr25109.sh on riscv64".
2021-03-17 Tom de Vries <tdevries@suse.de>
* Makefile (no-multifile-prop): Allow target rule to fail.
Tom de Vries [Wed, 17 Mar 2021 08:38:41 +0000 (09:38 +0100)]
Add --devel-skip-producer
In PR27588 a test-case was reported where dwz did not optimize due
to illegal DWARF produced by nasm:
...
$ dwz libxul.so -o libxul.so.z
dwz: libxul.so: loclistptr attribute, yet no .debug_loc section
...
Add an option --devel-skip-producer <producer> such that we can filter out
CUs coming from a particular producer, such that we have instead:
...
$ readelf -wi libxul.so | grep DW_AT_producer \
| sed 's/.*: //' | sort | uniq -c
176 clang LLVM (rustc version 1.50.0)
2 GNU AS 2.36.1
2 GNU C11 11.0.1 20210315 (experimental) <SNIP>
16 NASM 2.15.05
76 yasm 1.3.0
$ dwz libxul.so -o libxul.so.z --devel-skip-producer NASM
$ readelf -wi libxul.so.z | grep DW_AT_producer \
| sed 's/.*: //' | sort | uniq -c
176 clang LLVM (rustc version 1.50.0)
2 GNU AS 2.36.1
2 GNU C11 11.0.1 20210315 (experimental) <SNIP>
76 yasm 1.3.0
...
2021-03-16 Tom de Vries <tdevries@suse.de>
* dwz.c (add_skip_producer, skip_producer): New function.
(read_debug_info): Skip CUs fi DW_AT_producer in skip_producers list.
(dwz_options, usage): Add --devel-skip-producer entries.
(parse_args): Handle skip_producer_parsed.
Tom de Vries [Tue, 16 Mar 2021 16:03:54 +0000 (17:03 +0100)]
Print cu_offset for --devel-dump-pus
While working on PR27578 I used --devel-dump-edges to identify a problem:
...
idx: 48
cu: 0xbc
incoming: 200
incoming: 201
incoming: 201
incoming: 203
incoming: 204
incoming: 208
...
but I had difficulty relating back the PU with idx 48 to a PU as printed by
--devel-dump-pus.
By adding the printing of the cu_offset for --devel-dump-pus, we have:
...
-Partial unit (phase two):
+Partial unit (phase two) @ 0xbc:
611f5 O 2de8b1f3(deb4b00f) 2de8b1f3 Visual structure_type
...
and it's clear what PU is meant.
2021-03-16 Tom de Vries <tdevries@suse.de>
* dwz.c (partition_dups_1): Print cu_offset for --devel-dump-pus.
Tom de Vries [Tue, 16 Mar 2021 15:45:11 +0000 (16:45 +0100)]
Handle reordered dup chains in create_import_tree
With the test-case from PR27578, we run into:
...
$ dwz libvclplug_genlo.so.debug -o libvclplug_genlo.so.debug.z \
-lnone --odr --devel-progress
create_import_tree phase 2
dwz: dwz.c:8833: remove_import_edges: Assertion `i == cucount' failed.
Aborted (core dumped)
...
Using --devel-verify-edges, we can trigger an assert earlier:
...
create_import_tree phase 1
dwz: dwz.c:8923: verify_edges_1: \
Assertion `count == 0 || e1->icu->idx > last_idx' failed.
Aborted (core dumped)
...
where e1->icu->idx == 201 and last_idx == 201.
The problem is (as we can see using --devel-dump-edges) that there's a
duplicate edge from CU 201 to PU 48:
...
idx: 48
cu: 0xbc
incoming: 200
incoming: 201
incoming: 201
incoming: 203
incoming: 204
incoming: 208
...
Since it starts with a DECL, it will be reordered such that the DEF is at
the start. However, that breaks the code in create_import_tree that checks
for duplicate chain members from the same CU, which assumes that those are
adjacent.
Tom de Vries [Tue, 16 Mar 2021 15:45:11 +0000 (16:45 +0100)]
Fix another reference from PU to CU for odr
When using the test-case from PR27578 with --no-import-optimize, we run into:
...
$ dwz libvclplug_genlo.so.debug -o libvclplug_genlo.so.debug.z \
-lnone --odr --no-import-optimize
dwz: dwz.c:12700: write_die: \
Assertion `IMPLIES (cu->cu_kind == CU_PU, \
die_cu (refd)->cu_kind == CU_PU)' failed.
Aborted (core dumped)
...
The assert is triggered when trying to write out the reference to the DIE at
2f5e for DW_AT_type:
...
<1><2e8c>: Abbrev Number: 76 (DW_TAG_structure_type)
<2e8d> DW_AT_name : (indirect string, offset: 0x5f592): SalXLib
<2e91> DW_AT_byte_size : 320
...
<2><2ed9>: Abbrev Number: 22 (DW_TAG_member)
<2eda> DW_AT_name : (indirect string, offset: 0x56c9b): aReadFDS_
<2ede> DW_AT_decl_file : 33
<2edf> DW_AT_decl_line : 173
<2ee0> DW_AT_decl_column : 21
<2ee1> DW_AT_type : <0x2f5e>
<2ee5> DW_AT_data_member_location: 48
<2ee6> DW_AT_accessibility: 2 (protected)
...
The corresponding duplicate chain (with some annotations manually added) is:
...
duplicate chain:
Compilation Unit @ offset 0x44:
2d01 O 9f6a4268(29178f4e) 9f6a4268 SalXLib structure_type DECL
2e8c O 9f6a4268(87d059e8) 9f6a4268 SalXLib structure_type DEF
40a6 O 9f6a4268(87d059e8) 9f6a4268 SalXLib structure_type DEF
Compilation Unit @ offset 0xfaf6:
419be O 9f6a4268(29178f4e) 9f6a4268 SalXLib structure_type DECL
50e56 O 9f6a4268(29178f4e) 9f6a4268 SalXLib structure_type DECL
6063a O 9f6a4268(29178f4e) 9f6a4268 SalXLib structure_type DECL
Compilation Unit @ offset 0x1bb0f:
...
The duplicate chain is created by split_dups, which merged all the DECLs with
one duplicate chain containing only DEFs.
When doing the test for merged_singleton on this duplicate chain, it returns
NULL, because there are two DEFs. However, the DEFs are from the same CU, and
the type for aReadFDS_ is 0x2f5e for both DEFs. Consequently, there is no
duplicate chain for 0x2f5e, and we need to force a singleton duplicate chain.
Fix this by making merged_singleton return the first DEF.
2021-03-16 Tom de Vries <tdevries@suse.de>
PR dwz/27578
* dwz.c (merged_singleton): Handle DEFs in the same CU.
Tom de Vries [Mon, 15 Mar 2021 10:55:19 +0000 (11:55 +0100)]
Fix ubsan triggered in gdb-add-index.sh
When compiling with CFLAGS="-O0 -ggdb3 -fsanitize=undefined"
LDFLAGS="-fsanitize=undefined' and running the testsuite, we run into:
...
dwz.c:573:64: runtime error: left shift of 144 by 24 places cannot be \
represented in type 'int'
FAIL: src/testsuite/dwz.tests/gdb-add-index.sh
...
The problem is here in buf_read_ule32:
...
static inline uint32_t
buf_read_ule32 (unsigned char *data)
{
return data[0] | (data[1] << 8) | (data[2] << 16) | (data[3] << 24);
...
The data[3] is an unsigned char, which is promoted to int when used as shift
operand, which causes the shift to be a signed one, which can result in
undefined behaviour when shifting into the sign bit.
Fix this by casting to unsigned int. Likewise in buf_read_ube32.
2021-03-15 Tom de Vries <tdevries@suse.de>
* dwz.c (buf_read_ule32, buf_read_ube32): Avoid undefined signed
left-shift behaviour.
Martin Liska [Mon, 15 Mar 2021 10:17:12 +0000 (11:17 +0100)]
Fix UBSAN reported for Libreoffice so files
Use 1U in left shifts in bitvector_set_bit and bitvector_bit_p to avoid
undefined signed left-shift behaviour.
Fixes:
dwz.c:9041:28: runtime error: left shift of 1 by 31 places cannot be \
represented in type 'int'
#0 0x405ad3 in bitvector_bit_p dwz.c:9041
#1 0x405ad3 in create_import_tree dwz.c:9528
#2 0x405ad3 in dwz dwz.c:15445
#3 0x407517 in dwz_files dwz.c:16871
#4 0x407517 in main dwz.c:16985
#5 0x7ffff6a8eb24 in __libc_start_main ../csu/libc-start.c:332
#6 0x4086dd in _start (dwz+0x4086dd)
dwz.c:9032:21: runtime error: left shift of 1 by 31 places cannot be \
represented in type 'int'
#0 0x405a83 in bitvector_set_bit dwz.c:9032
#1 0x405a83 in create_import_tree dwz.c:9530
#2 0x405a83 in dwz dwz.c:15445
#3 0x407517 in dwz_files dwz.c:16871
#4 0x407517 in main dwz.c:16985
#5 0x7ffff6a8eb24 in __libc_start_main ../csu/libc-start.c:332
#6 0x4086dd in _start (dwz+0x4086dd)
2021-03-15 Martin Liska <mliska@suse.cz>
* dwz.c (bitvector_set_bit, bitvector_bit_p): Use 1U in
left shift to avoid undefined signed left-shift behaviour.
Tom de Vries [Wed, 10 Mar 2021 05:24:40 +0000 (06:24 +0100)]
Release process improvements
While doing the 0.14 release I ran into a few minor issues when running the
scripts in contrib/release:
- The script contrib/release/upload-release.sh requires a version argument,
but the version can just be read from the VERSION file.
- The contrib/release/do-release.sh does not do the git push. I used
my usual push script which also does a git rebase --ignore-date, which meant
the commit got updated and the tag no longer referred to the commit.
Also, I realized it could be useful to have a release checklist to go through.
So:
- Use $(cat VERSION) in contrib/release/upload-release.sh to get version
- Add git push commands to contrib/release/do-release.sh
- Add README.release-checklist
2021-03-08 Tom de Vries <tdevries@suse.de>
* README.release-checklist: New file.
* contrib/release/do-release.sh: Add git push commands.
* contrib/release/upload-release.sh: Use $(cat VERSION) to get version.
Tom de Vries [Mon, 8 Mar 2021 13:20:31 +0000 (14:20 +0100)]
Add --devel-no-checksum-cycle-opt
In checksum_ref_die, we need to determine a per-toplevel-die checksum based on
the inter-toplevel-die references. It has four modes:
- mode 1, a trivial one in which we walk towards toplevel-dies.
- mode 2, in which we optimistically assume that there are no cycles.
If that is the case, we're done. Otherwise, we use either mode 3 or mode 4.
- mode 3, dealing with cycles in a faster way
- mode 4, dealing with cycles
I did some testing using the testsuite and external testsuite and found that
mode 4 was not triggered once. I'm not quite sure how to construct a
test-case for that either.
Add an option --devel-no-checksum-cycle-opt that can be used to enforce using
mode 3 instead of mode 4, and add a test-case that uses it.
2021-02-23 Tom de Vries <tdevries@suse.de>
PR dwz/27429
* Makefile (TEST_EXECS): Add cycle.
(cycle): New target.
* dwz.c (checksum_cycle_opt): New var.
(checksum_ref_die): Handle checksum_cycle_opt.
(dwz_options, usage): Add --devel-checksum-cycle-opt entries.
* testsuite/dwz.tests/cycle.c: New test.
* testsuite/dwz.tests/cycle.sh: New test.
Tom de Vries [Fri, 5 Mar 2021 09:41:00 +0000 (10:41 +0100)]
[testsuite] Fix odr-loc.sh with -gdwarf-5
When running the test-suite like this:
...
$ make clean; make; make check CC="gcc -gdwarf-5" CXX="g++ -gdwarf-5"
...
we run into:
...
FAIL: src/testsuite/dwz.tests/odr-loc.sh
...
The test-case calls dwz twice, once without and once with
--devel-ignore-locus.
The first call is there to verify that all DIEs that can be deduplicated
without --devel-ignore-locus are indeed deduplicated, but not more.
The FAIL happens because some DIEs are not deduplicated. This is a regression
since commit 61c8d81 "Fix DW_AT_decl_file for odr". This makes sense: the
commit changed partitioning, and now the size heuristics are preventing some
deduplication.
Fix this by ignoring the size heuristics in the first dwz call using
--devel-ignore-size.
2021-03-05 Tom de Vries <tdevries@suse.de>
* testsuite/dwz.tests/odr-loc.sh: Call dwz first time using
--devel-ignore-size.
Tom de Vries [Fri, 5 Mar 2021 06:20:07 +0000 (07:20 +0100)]
Fix .debug_line reference above end of section
Consider the file test-import-repeatedly from PR26738, combined with a trivial
a.out:
...
$ cp test-import-repeatedly 1
$ gcc -m32 -g ~/hello.c
$ cp a.out 2
...
When doing multifile optimization, we run into:
...
$ dwz -m 3 2 1
dwz: 3: .debug_line reference above end of section
...
Using --devel-save-temps and src/contrib/gen-dwz-debug-all.sh we get the
unoptimized multifile, and find there a CU:
...
Compilation Unit @ offset 0x371:
Length: 0x6d (32-bit)
Version: 4
Abbrev Offset: 0x14d
Pointer Size: 4
<0><37c>: Abbrev Number: 1 (DW_TAG_compile_unit)
<37d> DW_AT_stmt_list : 0xda
<381> DW_AT_language : 0 (Unknown: 0)
<382> DW_AT_comp_dir : ./build-3.8
...
which refers to a .debug_line offset 0xda, but the debug_line section only
contains an entry at offset 0x0.
This can be explained as follows. The DIEs written into the CU do not
contain a single DW_AT_decl_file. Consequently, htab_line will be NULL once
we get here in write_multifile:
...
|| (line_htab != NULL && write_multifile_line ()))
...
and no .debug_line contribution will be added. However, the CU DIE
referencing the .debug_line contribution is written regardless.
We could fix this by not writing the DW_AT_stmt_list for the CU DIE in this
case, but that would require changing the order in which abbreviations are
computed.
Instead, fix this conservatively, by removing "line_htab != NULL &&", such we
get a minimal .debug_line contribution.
2021-03-02 Tom de Vries <tdevries@suse.de>
PR dwz/26738
* dwz.c (write_multifile): Also call write_multifile_line when
line_htab == NULL.
Tom de Vries [Thu, 4 Mar 2021 08:02:30 +0000 (09:02 +0100)]
Clean up temporary file in hardlink mode
Consider an executable file with hardlinks a.out and b.out.
When running dwz once:
...
$ dwz -h a.out b.out
...
a.out and b.out are updated, and remain hardlinks to the same file.
But when running dwz once more, a.out and b.out remain unchanged, and a
temporary file b.out.#dwz#.XXXXXX is left.
This is caused by the fact that the code in function dwz that is intended to
handle unchanged hardlinks is never triggered. It is guarded by a
"resa[n].res == 1" condition, but res->res is set to 0 at the end of function
dwz, irrespective of whether the file changed or not.
Fix this by only setting res->res to 0 if the file changed.
This makes test-case twice-multifile.sh fail. The test-case first
dwz-compresses files 1 and 2 in single-file mode. Then it attempts a
dwz -m 3 1 2.
This results in a res.res value of 1 (processed, unchanged) for both files.
This causes the files not to be counted as a success in main, and multifile
mode optimization is skipped.
Fix this by handling res.res == 1 in main.
2021-03-02 Tom de Vries <tdevries@suse.de>
PR dwz/24275
* dwz.c (dwz): Only set res->res to 0 if the file changed.
(main): Handle res.res == 1.
* testsuite/dwz.tests/twice-hardlink.sh: Remove PR24275 workaround.
Tom de Vries [Tue, 2 Mar 2021 07:26:52 +0000 (08:26 +0100)]
Fix assert after goto failure
Consider the following code in function dwz:
...
else if (write_aranges (dso))
{
cleanup ();
failure:
ret = 1;
}
...
char *p1 = realpath (file, NULL);
char *p2 = realpath (multifile, NULL);
...
if (p1 == NULL || p2 == NULL)
{
...
error (0, 0, "Could not compute relative multifile "
"pathname from %s to %s",
file, multifile);
goto failure;
...
I've managed to triggered this error in the following way.
First we do this setup:
...
$ mkdir tmpdir
$ ln -s tmpdir tmpdir-slink
$ cp hello 1
$ cp 1 2
...
Then we want to run dwz with a common file in tmpdir using the symlink
tmpdir-slink:
...
$ dwz -m ./tmpdir-slink/3 -r 1 2
...
but, stop just before calling realpath and remove the symlink to make the p2
realpath call return NULL.
We can do that using gdb:
...
$ gdb -batch \
-ex "b 15451" \
-ex run \
-ex "shell rm -f tmpdir-slink" \
-ex cont \
--args ./dwz -m ./tmpdir-slink/3 -r 1 2
Breakpoint 1 at 0x382d1: file dwz.c, line 15451.
Breakpoint 1, dwz (file=0x7fffffffe285 "1", outfile=0x0, res=0x5555557a5260,
resa=0x0, files=0x7fffffffdeb8) at dwz.c:15451
15451 if (!multifile_relative)
dwz: Could not compute relative multifile pathname from 1 to ./tmpdir-slink/3
dwz: dwz.c:2170: off_htab_add_die: Assertion `*slot == NULL' failed.
Program received signal SIGABRT, Aborted.
0x00007ffff75fffb7 in raise () from /lib/x86_64-linux-gnu/libc.so.6
...
but we hit an assertion.
This is because cleanup is not called.
Fix this by moving the failure label up one line.
2021-03-02 Tom de Vries <tdevries@suse.de>
* dwz.c (dwz): Move failure label to before cleanup.
Tom de Vries [Sat, 27 Feb 2021 10:34:55 +0000 (11:34 +0100)]
Fix reference of PU to CU for odr
When compiling dwz with the assert listed in the commit message for
"Call reorder_dups ASAP", and using test-case cc1.dwz-processed like so, we
run into:
...
$ dwz cc1.dwz-processed -o 1 -lnone --odr
dwz: dwz.c:12567: write_die: Assertion \
`IMPLIES (cu->cu_kind == CU_PU, die_cu (refd)->cu_kind == CU_PU)' failed.
Aborted (core dumped)
...
The assert is related to this duplicate chain:
... 28dd83f O b500801d621b6872 pointer_type \
(type: 28dd72f die_struct structure_type) 2903769 O b500801d621b6872 pointer_type \
(type: 28dd72f die_struct structure_type)
...
which contains two DIEs that are both in the same CU:
...
<0><28d281f>: Abbrev Number: 200 (DW_TAG_compile_unit)
...
<1><28dd72f>: Abbrev Number: 35 (DW_TAG_structure_type)
<28dd730> DW_AT_name : (indirect string, offset: 0x1b2ca6): die_struct
<28dd734> DW_AT_byte_size : 80
<28dd735> DW_AT_decl_file : 87
<28dd736> DW_AT_decl_line : 3069
<28dd738> DW_AT_decl_column : 63
<28dd739> DW_AT_sibling : <0x28dd83b>
<1><28dd83f>: Abbrev Number: 9 (DW_TAG_pointer_type)
<28dd840> DW_AT_byte_size : 8
<28dd841> DW_AT_type : <0x28dd72f>
...
<1><2903769>: Abbrev Number: 9 (DW_TAG_pointer_type)
<290376a> DW_AT_byte_size : 8
<290376b> DW_AT_type : <0x28dd72f>
...
<0><2911617>: Abbrev Number: 177 (DW_TAG_compile_unit)
...
The dup chain is forced to a partial unit, but the die_struct DIE is not,
because it's not part of a duplicate chain, and it's not marked as a
singleton. Fix this by marking the die_struct DIE as singleton.
Tom de Vries [Sat, 27 Feb 2021 10:34:55 +0000 (11:34 +0100)]
Call reorder_dups ASAP
Currently, we call reorder_dups just before partial unit creation, and only for
DIEs that will be copied.
This approach causes a problem with reachable DIE propagation (which is done
in partition_dups after phase 1): when a dup-chain that needs to be reordered
(starts with ODR_DECL but also contains ODR_DEF) is marked as reachable during
this propagation, propagation will stop at the ODR_DECL, while it should
continue at the first ODR_DEF instead.
Fix this by calling reorder_dups ASAP, just after computing the partitions.
The problem can be detected using this assert:
...
@@ -12563,6 +12563,8 @@ write_die
{
dw_cu_ref refdcu = die_cu (refd);
value = refd->u.p2.die_new_offset;
+ assert (IMPLIES (cu->cu_kind == CU_PU,
+ die_cu (refd)->cu_kind == CU_PU));
assert (value && refdcu->cu_kind != CU_ALT);
if (t->attr[j].form == DW_FORM_ref_addr)
{
...
We accept the small performance penalty because this patch is a prerequisite
for the PR25424 bug fix.
2021-02-25 Tom de Vries <tdevries@suse.de>
* dwz.c (calculate_partitions): New function, factored out of ...
(partition_dups_1): ... here. Drop vec_size parameter. Add
nr_partitions and partitions parameter. Iterate over partitions array.
(partition_dups): Call calculate_partitions. Update calls to
partition_dups_1.
Mark Wielaard [Fri, 26 Feb 2021 14:38:58 +0000 (15:38 +0100)]
PR27463 Accept DW_FORM_sdata for DW_AT_decl/call_file
Using DW_FORM_sdata for DW_AT_decl/call_file is somewhat inefficient
since file index numbers are always positive values. But it is a valid
form to encode a constant value. Accept DW_FORM_sdata as long as it
encodes a positive value. Extend the positive value check to
DW_FORM_implicit_const.
* dwz.c (checksum_die): Accept DW_FORM_sdata for
DW_AT_decl/call_file as long as the value is positive. Also
check DW_FORM_implicit_const value is positive for these
attributes.
(die_eq_1): Handle DW_FORM_sdata for DW_AT_decl/call_file.
(build_abbrevs_for_die): Likewise.
(write_die): Likewise.
* testsuite/dwz.tests/pr27463.sh: New test.
* testsuite/lib/unavailable-dwarf-piece.exp: New testfile
from gdb.
* testsuite/dwz.tests/dwz-tests.exp: Add unavailable-dwarf-piece
for pr25109.sh.
* testsuite/dwz.tests/main.c (foo): New function and labels.
(bar): Likewise.
* Makefile (TEST_EXECS_DWARF_ASM): Add unavailable-dwarf-piece.
Tom de Vries [Fri, 26 Feb 2021 11:47:34 +0000 (12:47 +0100)]
[testsuite] Fix pr25109.sh on riscv64
On riscv64, I run into:
...
cc main.c no-multifile-prop-dw.S -o no-multifile-prop
no-multifile-prop-dw.S: Assembler messages:
no-multifile-prop-dw.S:25: Error: non-constant .uleb128 is not supported
make: *** [Makefile:99: no-multifile-prop] Error 1
...
Fix this by allowing to fail to build the test-case, and marking it as
unsupported.
Tom de Vries [Thu, 25 Feb 2021 08:04:33 +0000 (09:04 +0100)]
Fix DW_AT_decl_file for odr
Consider odr-struct. It has two structs aaa (from different CUs), each with
members of type bbb and ccc, but in one case bbb is a decl, in the other case
ccc is a decl.
When doing odr, we end up with one struct aaa, and no decls:
...
$ dwz --odr odr-struct
$ readelf -wi odr-struct \
| egrep -A2 "DW_TAG_structure" \
| egrep "DW_TAG|DW_AT_name|DW_AT_decl"
<1><19>: Abbrev Number: 25 (DW_TAG_structure_type)
<1a> DW_AT_name : ccc
<1><2f>: Abbrev Number: 25 (DW_TAG_structure_type)
<30> DW_AT_name : aaa
<1><4b>: Abbrev Number: 25 (DW_TAG_structure_type)
<4c> DW_AT_name : bbb
...
Now consider using the same file for multifile optimization, combined with
odr:
...
$ cp odr-struct 1; cp 1 2; dwz -m 3 1 2 --odr
...
The desired outcome is that the structs aaa are unified (same as above) in
both 1 and 2, and then moved to multifile 3.
The problem is that the DW_AT_decl_file is different for struct bbb in 1 and
3, and that causes the struct types to linger in 1.
The problem can already be shown without multifile mode using:
....
$ dwz odr-struct --odr
$ llvm-dwarfdump odr-struct \
| grep -A3 struct \
| egrep -v "^--|DW_AT_byte_size" \
| sed 's%/.*/%%'
0x00000019: DW_TAG_structure_type
DW_AT_name ("ccc")
DW_AT_decl_file ("odr.cc")
0x0000002f: DW_TAG_structure_type
DW_AT_name ("aaa")
DW_AT_decl_file ("odr.h")
0x0000004b: DW_TAG_structure_type
DW_AT_name ("bbb")
DW_AT_decl_file ("odr.cc")
...
The DW_AT_decl_file for struct bbb should be odr-2.cc.
The problem is caused by by odr: odr allows defs and decls to be part of the
same duplicate chain, which breaks the invariant that DIEs in the duplicate
chain are isomorph. During write_die, the first DIE in the chain is written
out as the representative copy, which means having a decl as the first in the
chain is counterproductive. We have reorder_dups to fix this problem, which
detects if a duplicate chain starts with a decl and then moves the first def
before it. However, this breaks another variant: that for each partition, all
representative DIEs are from the same CU. Consequently, the file table of the
partition may not match with the DW_AT_decl_file number.
In other words, the reordered duplicate chain is in the wrong partition.
Fix this by:
- ignoring ODR_DECL DIEs at the start of a duplicate chain when partitioning
- moving the reorder_dups call to the start of partial unit creating, such
that we get the correct refcu.
This also breaks the invariant checked in create_import_tree that
partition_dups doesn't generate two seperate partial units with the same set
of referrer CUs, so we allow this for odr.
2021-02-19 Tom de Vries <tdevries@suse.de>
PR dwz/27438
* dwz.c (partition_cmp): Ignore ODR_DECL dies at the
start of a duplicate chain.
(partition_dups_1): Same. Move call to reorder_dups earlier.
(create_import_tree): Allow partial units with same set of referrers
for odr.
* testsuite/dwz.tests/odr-struct-multifile.sh: New test.
Tom de Vries [Tue, 23 Feb 2021 11:07:51 +0000 (12:07 +0100)]
Clean up die_odr_state interface
The die_odr_state function returns the odr state for a die, and if it hasn't
been calculated, it calculates it first using a call to set_die_odr_state:
...
static unsigned int UNUSED
die_odr_state (dw_cu_ref cu, dw_die_ref die)
{
if (die->die_odr_state != ODR_UNKNOWN)
return die->die_odr_state;
The call to set_die_odr_state needs a cu argument, and consequently
die_odr_state also has one. That means a call to die_odr_state needs a proper
CU argument if set_die_odr_state gets called, and for optimality a NULL CU
otherwise.
As it happens, there's only one place where the proper CU argument is
required, and it's easy to accidentally do:
...
die_odr_state (die_cu (die), die)
...
where a NULL CU would suffice.
Fix this by explicitly calling set_die_odr_state and dropping the cu parameter
from die_odr_state.
2021-02-23 Tom de Vries <tdevries@suse.de>
* dwz.c (set_die_odr_state): Assert die->die_odr_state == ODR_UNKNOWN.
(die_odr_state): Drop cu parameter. Assert
die->die_odr_state != ODR_UNKNOWN.
(checksum_die): Call set_die_odr_state. Update call to die_odr_state.
(read_debug_info, split_dups, reorder_dups, merged_singleton)
(partition_dups): Update call to die_odr_state.
Tom de Vries [Tue, 23 Feb 2021 09:24:21 +0000 (10:24 +0100)]
Don't print die_hash2 for ODR_UNKNOWN in dump_die_with_indent
With the following debugging session where we dwz exec hello using odr:
...
$ gdb -q --args dwz hello -o hello.z --odr
Reading symbols from dwz...
(gdb) b checksum_die
Breakpoint 1 at 0x409778: file dwz.c, line 3348.
(gdb) r
Starting program: dwz hello -o hello.z --odr
Breakpoint 1, checksum_die (...) at dwz.c:3348
3348 switch (die->die_ck_state)
(gdb) call dump_die (die)
b X 0(0) 0 ../sysdeps/x86_64/start.S compile_unit
(gdb) p die->die_odr_state
$1 = ODR_UNKNOWN
...
we see that we're printing the die_hash2 for a die with die_odr_state ==
ODR_UNKNOWN, while this is only supposed to trigger for ODR_DECL and ODR_DEF.
Fix this in dump_die_with_indent, by not printing die_hash2 when die_odr_state ==
ODR_UNKNOWN.
2021-02-23 Tom de Vries <tdevries@suse.de>
* dwz.c (dump_die_with_indent): Don't print die_hash2 when
die_odr_state == ODR_UNKNOWN.
Tom de Vries [Mon, 22 Feb 2021 15:38:26 +0000 (16:38 +0100)]
Don't call die_odr_state with unnecessarily defined cu arg
When compiling dwz with this patch:
...
die_odr_state (dw_cu_ref cu, dw_die_ref die)
{
if (die->die_odr_state != ODR_UNKNOWN)
- return die->die_odr_state;
+ {
+ assert (cu == NULL);
+ return die->die_odr_state;
+ }
...
and running f.i. odr-struct.sh, we run into the abort.
The recent commit 3312feb "Fix CK_BAD propagation for --odr" introduced this
code:
...
if (die_odr_state (die_cu (die), die) != ODR_NONE)
die->u.p1.die_ref_hash = die->u.p1.die_hash;
...
and there's no need to pass a CU argument, which makes the abort trigger.
Fix this by passing a NULL CU instead.
2021-02-22 Tom de Vries <tdevries@suse.de>
* dwz.c (read_debug_info): Pass NULL CU to die_odr_state call.
Tom de Vries [Mon, 22 Feb 2021 07:44:16 +0000 (08:44 +0100)]
Fix CK_BAD propagation for --odr
With the reproducer from PR26252 we get:
...
$ dwz 2 -o 2.z --odr -lnone
dwz: dwz.c:7396: partition_found_dups: \
Assertion `die->die_ck_state == CK_KNOWN' failed.
Aborted (core dumped)
...
The problem is caused by the odr code in checksum_ref_die, which skips
checksum calculation for the children of odr types, with as unintended
side-effect that it break the CK_BAD propagation to toplevel DIEs.
Fix this by making the skipping of the checksum calculation less intrusive.
Specially:
- undo all modifications related to odr in checksum_ref_die
- After calling checksum_ref_die in read_debug_info:
- override die_ref_hash for odr DIEs, and
- recalculate die_ref_hash for all other DIEs.
We still have the same amount of compression with cc1, that is: without odr
we have 42.30% reduction:
...
$ dwz cc1 -o cc1.z -lnone --no-odr
$ diff.sh cc1 cc1.z
.debug_info red: 44.84% 11152724861527733
.debug_abbrev red: 40.28% 17227261028968
.debug_str red: 0% 66093556609355
total red: 42.30% 11985932969166056
...
and with odr but without the bug fix we have 54.55%:
...
$ dwz cc1 -o cc1.z -lnone --odr
$ diff.sh cc1 cc1.z
.debug_info red: 57.46% 11152724847449258
.debug_abbrev red: 75.08% 1722726 429434
.debug_str red: 0% 66093556609355
total red: 54.55% 11985932954488047
...
and with odr and the bug fix still 54.55%:
...
$ dwz cc1 -o cc1.z -lnone --odr
$ diff.sh cc1 cc1.z
.debug_info red: 57.46% 11152724847446501
.debug_abbrev red: 75.51% 1722726 422027
.debug_str red: 0% 66093556609355
total red: 54.55% 11985932954477883
...
2021-02-18 Tom de Vries <tdevries@suse.de>
PR dwz/26252
* dwz.c (checksum_ref_die): Undo modifications related to odr.
(read_debug_info): After calling checksum_ref_die, override
die_ref_hash for odr DIEs, and recalculate die_ref_hash for all
other DIEs.
Tom Tromey [Sun, 21 Feb 2021 02:12:54 +0000 (19:12 -0700)]
Print --version and --help to stdout
Normall --version and --help output go to stdout and cause a program
to exit with status 0. The rationale for the exit status is that the
user asked for this behavior, and the program successfully complied.
Printing --help output to stdout is nicer for piping into a pager.
This patch changes dwz to follow this approach.
I kept the program invocation name in the Usage line. Different
programs seem to handle this differently.
git://sourceware.org / dwz.git / log