Sources Bugzilla – Bug 13443
test messages should be unique
Last modified: 2013-01-24 10:29:13 UTC
There are many tests in the test suite that output the same message as other tests in the same test file. We should make each output unique, to facilitate test identification, as well as helping with test case result analysis/comparison. $ cat testsuite/gdb.sum | grep "gdb..*:" | sed 's/[A-Z]\+: //g' | sort | uniq -c | sort -n | grep -v "^\( \)*1 " | wc -l 1263 That means there are currently 1263 outputs in gdb.sum that are not unique. That was on amd64-linux-linux. The top 20 of worse offenders is: $ cat testsuite/gdb.sum | grep "gdb..*:" | sed 's/[A-Z]\+: //g' | sort | uniq -c | sort -n | tail -n 20 7 gdb.python/py-value.exp: get value from history 8 gdb.python/py-template.exp: print foo 8 gdb.python/py-template.exp: python foo = gdb.history(0) 8 gdb.python/py-type.exp: get value from history 9 gdb.base/call-sc.exp: return foo; synchronize pc to main() 9 gdb.base/call-sc.exp: set print address off 9 gdb.base/call-sc.exp: set print sevenbit-strings 9 gdb.base/call-sc.exp: set width 0 9 gdb.python/py-breakpoint.exp: continue to breakpoint: Break at multiply. 10 gdb.base/checkpoint.exp: restart 1 three 10 gdb.base/relational.exp: set variable x 11 gdb.base/relational.exp: set variable z 11 gdb.python/py-type.exp: print value 15 gdb.base/dump.exp: print zero_all () 24 gdb.base/structs.exp: set print address off 24 gdb.base/structs.exp: set print elements 300 24 gdb.base/structs.exp: set print sevenbit-strings 24 gdb.base/structs.exp: set width 0 26 gdb.reverse/sigall-precsave.exp: reverse signal delivered 26 gdb.reverse/sigall-reverse.exp: reverse signal delivered The first column indicates how often is the output message repeated in gdb.sum. And here's the distribution: $ cat testsuite/gdb.sum | grep "gdb..*:" | sed 's/[A-Z]\+: //g' | sort | uniq -c | sort -n | awk ' { print $1 } ' | uniq -c 18636 1 1129 2 55 3 41 4 8 5 6 6 5 7 3 8 5 9 2 10 2 11 1 15 4 24 2 26 Unfortunately, by far, we have many cases of the same output appearing twice.
gcc's compare_tests script is one example of a tool that gets confused by the non-unique test messages. E.g., comparing the same .sum with itself, should naturally not find any regressions or new passes, but note: $ ~/src/gcc/gcc/contrib/compare_tests testsuite/gdb.sum testsuite/gdb.sum Tests that now fail, but worked before: gdb.base/jit.exp: one_jit_test-1: info function jit_function gdb.base/jit.exp: one_jit_test-2: info function jit_function gdb.mi/mi-break.exp: list of breakpoints gdb.mi/mi2-break.exp: list of breakpoints one_jit_test-1 info function jit_function one_jit_test-2 info function jit_function Tests that now work, but didn't before: gdb.base/jit.exp: one_jit_test-1: info function jit_function gdb.base/jit.exp: one_jit_test-2: info function jit_function gdb.mi/mi-break.exp: list of breakpoints gdb.mi/mi2-break.exp: list of breakpoints one_jit_test-1 info function jit_function one_jit_test-2 info function jit_function This is because each of these tests outputs the same message twice, once with a PASS and once with a FAIL.
CVSROOT: /cvs/src Module name: src Changes by: qiyao@sourceware.org 2012-12-04 03:55:52 Modified files: gdb/testsuite : ChangeLog gdb/testsuite/gdb.trace: actions.exp Log message: gdb/testsuite 2012-12-04 Yao Qi <yao@codesourcery.com> PR gdb/13443 * gdb.trace/actions.exp: Make test messages unique. Patches: http://sourceware.org/cgi-bin/cvsweb.cgi/src/gdb/testsuite/ChangeLog.diff?cvsroot=src&r1=1.3471&r2=1.3472 http://sourceware.org/cgi-bin/cvsweb.cgi/src/gdb/testsuite/gdb.trace/actions.exp.diff?cvsroot=src&r1=1.25&r2=1.26
CVSROOT: /cvs/src Module name: src Changes by: abidh@sourceware.org 2013-01-18 10:50:15 Modified files: gdb/testsuite : ChangeLog gdb/testsuite/gdb.base: checkpoint.exp Log message: 2013-01-18 Hafiz Abid Qadeer<abidh@codesourcery.com> PR gdb/13443 * gdb.base/checkpoint.exp: Update test messages to make them unique. Patches: http://sourceware.org/cgi-bin/cvsweb.cgi/src/gdb/testsuite/ChangeLog.diff?cvsroot=src&r1=1.3517&r2=1.3518 http://sourceware.org/cgi-bin/cvsweb.cgi/src/gdb/testsuite/gdb.base/checkpoint.exp.diff?cvsroot=src&r1=1.20&r2=1.21
CVSROOT: /cvs/src Module name: src Changes by: abidh@sourceware.org 2013-01-24 10:29:09 Modified files: gdb/testsuite : ChangeLog gdb/testsuite/gdb.mi: mi-var-block.exp Log message: 2013-01-24 Hafiz Abid Qadeer <abidh@codesourcery.com> PR gdb/13443 * gdb.mi/mi-var-block.exp: Make test messages unique. Patches: http://sourceware.org/cgi-bin/cvsweb.cgi/src/gdb/testsuite/ChangeLog.diff?cvsroot=src&r1=1.3530&r2=1.3531 http://sourceware.org/cgi-bin/cvsweb.cgi/src/gdb/testsuite/gdb.mi/mi-var-block.exp.diff?cvsroot=src&r1=1.29&r2=1.30