Bug 13443 - test messages should be unique
test messages should be unique
Status: NEW
Product: gdb
Classification: Unclassified
Component: testsuite
HEAD
: P2 normal
: ---
Assigned To: Not yet assigned to anyone
:
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2011-11-27 11:35 UTC by Pedro Alves
Modified: 2013-01-24 10:29 UTC (History)
0 users

See Also:
Host:
Target:
Build:
Last reconfirmed:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Pedro Alves 2011-11-27 11:35:50 UTC
There are many tests in the test suite that output the same message as other tests in the same test file.

We should make each output unique, to facilitate test identification, as well
as helping with test case result analysis/comparison.

$ cat testsuite/gdb.sum | grep "gdb..*:" | sed 's/[A-Z]\+: //g' | sort | uniq -c | sort -n | grep -v "^\( \)*1 " | wc -l
1263

That means there are currently 1263 outputs in gdb.sum that are not unique.  That was on amd64-linux-linux.

The top 20 of worse offenders is:

$ cat testsuite/gdb.sum | grep "gdb..*:" | sed 's/[A-Z]\+: //g' | sort | uniq -c | sort -n | tail -n 20

      7 gdb.python/py-value.exp: get value from history
      8 gdb.python/py-template.exp: print foo
      8 gdb.python/py-template.exp: python foo = gdb.history(0)
      8 gdb.python/py-type.exp: get value from history
      9 gdb.base/call-sc.exp: return foo; synchronize pc to main()
      9 gdb.base/call-sc.exp: set print address off
      9 gdb.base/call-sc.exp: set print sevenbit-strings
      9 gdb.base/call-sc.exp: set width 0
      9 gdb.python/py-breakpoint.exp: continue to breakpoint: Break at multiply.
     10 gdb.base/checkpoint.exp: restart 1 three
     10 gdb.base/relational.exp: set variable x
     11 gdb.base/relational.exp: set variable z
     11 gdb.python/py-type.exp: print value
     15 gdb.base/dump.exp: print zero_all ()
     24 gdb.base/structs.exp: set print address off
     24 gdb.base/structs.exp: set print elements 300
     24 gdb.base/structs.exp: set print sevenbit-strings
     24 gdb.base/structs.exp: set width 0
     26 gdb.reverse/sigall-precsave.exp: reverse signal delivered
     26 gdb.reverse/sigall-reverse.exp: reverse signal delivered

The first column indicates how often is the output message repeated in gdb.sum.

And here's the distribution:

$ cat testsuite/gdb.sum | grep "gdb..*:" | sed 's/[A-Z]\+: //g' | sort | uniq -c | sort -n | awk ' { print $1 } ' | uniq -c
  18636 1
   1129 2
     55 3
     41 4
      8 5
      6 6
      5 7
      3 8
      5 9
      2 10
      2 11
      1 15
      4 24
      2 26

Unfortunately, by far, we have many cases of the same output appearing twice.
Comment 1 Pedro Alves 2011-11-27 11:42:57 UTC
gcc's compare_tests script is one example of a tool that gets confused by the non-unique test messages.  

E.g., comparing the same .sum with itself, should naturally not find any regressions or new passes, but note:

$ ~/src/gcc/gcc/contrib/compare_tests testsuite/gdb.sum testsuite/gdb.sum 
Tests that now fail, but worked before:

gdb.base/jit.exp: one_jit_test-1: info function jit_function
gdb.base/jit.exp: one_jit_test-2: info function jit_function
gdb.mi/mi-break.exp: list of breakpoints
gdb.mi/mi2-break.exp: list of breakpoints
one_jit_test-1 info function jit_function
one_jit_test-2 info function jit_function

Tests that now work, but didn't before:

gdb.base/jit.exp: one_jit_test-1: info function jit_function
gdb.base/jit.exp: one_jit_test-2: info function jit_function
gdb.mi/mi-break.exp: list of breakpoints
gdb.mi/mi2-break.exp: list of breakpoints
one_jit_test-1 info function jit_function
one_jit_test-2 info function jit_function

This is because each of these tests outputs the same message twice, once with a PASS and once with a FAIL.
Comment 2 cvs-commit@gcc.gnu.org 2012-12-04 03:56:03 UTC
CVSROOT:	/cvs/src
Module name:	src
Changes by:	qiyao@sourceware.org	2012-12-04 03:55:52

Modified files:
	gdb/testsuite  : ChangeLog 
	gdb/testsuite/gdb.trace: actions.exp 

Log message:
	gdb/testsuite
	2012-12-04  Yao Qi  <yao@codesourcery.com>
	
	PR gdb/13443
	* gdb.trace/actions.exp: Make test messages unique.

Patches:
http://sourceware.org/cgi-bin/cvsweb.cgi/src/gdb/testsuite/ChangeLog.diff?cvsroot=src&r1=1.3471&r2=1.3472
http://sourceware.org/cgi-bin/cvsweb.cgi/src/gdb/testsuite/gdb.trace/actions.exp.diff?cvsroot=src&r1=1.25&r2=1.26
Comment 3 cvs-commit@gcc.gnu.org 2013-01-18 10:50:21 UTC
CVSROOT:	/cvs/src
Module name:	src
Changes by:	abidh@sourceware.org	2013-01-18 10:50:15

Modified files:
	gdb/testsuite  : ChangeLog 
	gdb/testsuite/gdb.base: checkpoint.exp 

Log message:
	2013-01-18  Hafiz Abid Qadeer<abidh@codesourcery.com>
	
	PR gdb/13443
	* gdb.base/checkpoint.exp: Update test messages to make them
	unique.

Patches:
http://sourceware.org/cgi-bin/cvsweb.cgi/src/gdb/testsuite/ChangeLog.diff?cvsroot=src&r1=1.3517&r2=1.3518
http://sourceware.org/cgi-bin/cvsweb.cgi/src/gdb/testsuite/gdb.base/checkpoint.exp.diff?cvsroot=src&r1=1.20&r2=1.21
Comment 4 cvs-commit@gcc.gnu.org 2013-01-24 10:29:13 UTC
CVSROOT:	/cvs/src
Module name:	src
Changes by:	abidh@sourceware.org	2013-01-24 10:29:09

Modified files:
	gdb/testsuite  : ChangeLog 
	gdb/testsuite/gdb.mi: mi-var-block.exp 

Log message:
	2013-01-24  Hafiz Abid Qadeer  <abidh@codesourcery.com>
	
	PR gdb/13443
	* gdb.mi/mi-var-block.exp: Make test messages unique.

Patches:
http://sourceware.org/cgi-bin/cvsweb.cgi/src/gdb/testsuite/ChangeLog.diff?cvsroot=src&r1=1.3530&r2=1.3531
http://sourceware.org/cgi-bin/cvsweb.cgi/src/gdb/testsuite/gdb.mi/mi-var-block.exp.diff?cvsroot=src&r1=1.29&r2=1.30