Bug 13443 - test messages should be unique
Summary: test messages should be unique
Status: NEW
Alias: None
Product: gdb
Classification: Unclassified
Component: testsuite (show other bugs)
Version: HEAD
: P2 normal
Target Milestone: ---
Assignee: Not yet assigned to anyone
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2011-11-27 11:35 UTC by Pedro Alves
Modified: 2014-08-09 00:46 UTC (History)
0 users

See Also:
Host:
Target:
Build:
Last reconfirmed:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Pedro Alves 2011-11-27 11:35:50 UTC
There are many tests in the test suite that output the same message as other tests in the same test file.

We should make each output unique, to facilitate test identification, as well
as helping with test case result analysis/comparison.

$ cat testsuite/gdb.sum | grep "gdb..*:" | sed 's/[A-Z]\+: //g' | sort | uniq -c | sort -n | grep -v "^\( \)*1 " | wc -l
1263

That means there are currently 1263 outputs in gdb.sum that are not unique.  That was on amd64-linux-linux.

The top 20 of worse offenders is:

$ cat testsuite/gdb.sum | grep "gdb..*:" | sed 's/[A-Z]\+: //g' | sort | uniq -c | sort -n | tail -n 20

      7 gdb.python/py-value.exp: get value from history
      8 gdb.python/py-template.exp: print foo
      8 gdb.python/py-template.exp: python foo = gdb.history(0)
      8 gdb.python/py-type.exp: get value from history
      9 gdb.base/call-sc.exp: return foo; synchronize pc to main()
      9 gdb.base/call-sc.exp: set print address off
      9 gdb.base/call-sc.exp: set print sevenbit-strings
      9 gdb.base/call-sc.exp: set width 0
      9 gdb.python/py-breakpoint.exp: continue to breakpoint: Break at multiply.
     10 gdb.base/checkpoint.exp: restart 1 three
     10 gdb.base/relational.exp: set variable x
     11 gdb.base/relational.exp: set variable z
     11 gdb.python/py-type.exp: print value
     15 gdb.base/dump.exp: print zero_all ()
     24 gdb.base/structs.exp: set print address off
     24 gdb.base/structs.exp: set print elements 300
     24 gdb.base/structs.exp: set print sevenbit-strings
     24 gdb.base/structs.exp: set width 0
     26 gdb.reverse/sigall-precsave.exp: reverse signal delivered
     26 gdb.reverse/sigall-reverse.exp: reverse signal delivered

The first column indicates how often is the output message repeated in gdb.sum.

And here's the distribution:

$ cat testsuite/gdb.sum | grep "gdb..*:" | sed 's/[A-Z]\+: //g' | sort | uniq -c | sort -n | awk ' { print $1 } ' | uniq -c
  18636 1
   1129 2
     55 3
     41 4
      8 5
      6 6
      5 7
      3 8
      5 9
      2 10
      2 11
      1 15
      4 24
      2 26

Unfortunately, by far, we have many cases of the same output appearing twice.
Comment 1 Pedro Alves 2011-11-27 11:42:57 UTC
gcc's compare_tests script is one example of a tool that gets confused by the non-unique test messages.  

E.g., comparing the same .sum with itself, should naturally not find any regressions or new passes, but note:

$ ~/src/gcc/gcc/contrib/compare_tests testsuite/gdb.sum testsuite/gdb.sum 
Tests that now fail, but worked before:

gdb.base/jit.exp: one_jit_test-1: info function jit_function
gdb.base/jit.exp: one_jit_test-2: info function jit_function
gdb.mi/mi-break.exp: list of breakpoints
gdb.mi/mi2-break.exp: list of breakpoints
one_jit_test-1 info function jit_function
one_jit_test-2 info function jit_function

Tests that now work, but didn't before:

gdb.base/jit.exp: one_jit_test-1: info function jit_function
gdb.base/jit.exp: one_jit_test-2: info function jit_function
gdb.mi/mi-break.exp: list of breakpoints
gdb.mi/mi2-break.exp: list of breakpoints
one_jit_test-1 info function jit_function
one_jit_test-2 info function jit_function

This is because each of these tests outputs the same message twice, once with a PASS and once with a FAIL.
Comment 2 Sourceware Commits 2012-12-04 03:56:03 UTC
CVSROOT:	/cvs/src
Module name:	src
Changes by:	qiyao@sourceware.org	2012-12-04 03:55:52

Modified files:
	gdb/testsuite  : ChangeLog 
	gdb/testsuite/gdb.trace: actions.exp 

Log message:
	gdb/testsuite
	2012-12-04  Yao Qi  <yao@codesourcery.com>
	
	PR gdb/13443
	* gdb.trace/actions.exp: Make test messages unique.

Patches:
http://sourceware.org/cgi-bin/cvsweb.cgi/src/gdb/testsuite/ChangeLog.diff?cvsroot=src&r1=1.3471&r2=1.3472
http://sourceware.org/cgi-bin/cvsweb.cgi/src/gdb/testsuite/gdb.trace/actions.exp.diff?cvsroot=src&r1=1.25&r2=1.26
Comment 3 Sourceware Commits 2013-01-18 10:50:21 UTC
CVSROOT:	/cvs/src
Module name:	src
Changes by:	abidh@sourceware.org	2013-01-18 10:50:15

Modified files:
	gdb/testsuite  : ChangeLog 
	gdb/testsuite/gdb.base: checkpoint.exp 

Log message:
	2013-01-18  Hafiz Abid Qadeer<abidh@codesourcery.com>
	
	PR gdb/13443
	* gdb.base/checkpoint.exp: Update test messages to make them
	unique.

Patches:
http://sourceware.org/cgi-bin/cvsweb.cgi/src/gdb/testsuite/ChangeLog.diff?cvsroot=src&r1=1.3517&r2=1.3518
http://sourceware.org/cgi-bin/cvsweb.cgi/src/gdb/testsuite/gdb.base/checkpoint.exp.diff?cvsroot=src&r1=1.20&r2=1.21
Comment 4 Sourceware Commits 2013-01-24 10:29:13 UTC
CVSROOT:	/cvs/src
Module name:	src
Changes by:	abidh@sourceware.org	2013-01-24 10:29:09

Modified files:
	gdb/testsuite  : ChangeLog 
	gdb/testsuite/gdb.mi: mi-var-block.exp 

Log message:
	2013-01-24  Hafiz Abid Qadeer  <abidh@codesourcery.com>
	
	PR gdb/13443
	* gdb.mi/mi-var-block.exp: Make test messages unique.

Patches:
http://sourceware.org/cgi-bin/cvsweb.cgi/src/gdb/testsuite/ChangeLog.diff?cvsroot=src&r1=1.3530&r2=1.3531
http://sourceware.org/cgi-bin/cvsweb.cgi/src/gdb/testsuite/gdb.mi/mi-var-block.exp.diff?cvsroot=src&r1=1.29&r2=1.30
Comment 5 Sourceware Commits 2013-07-31 00:38:42 UTC
CVSROOT:	/cvs/src
Module name:	src
Changes by:	qiyao@sourceware.org	2013-07-31 00:38:40

Modified files:
	gdb/testsuite  : ChangeLog 
	gdb/testsuite/gdb.trace: backtrace.exp 

Log message:
	gdb/testsuite/
	
	PR gdb/13443
	* gdb.trace/backtrace.exp (gdb_backtrace_tdp_3): Add parameter
	'traceframe'.  Wrap test with with_test_prefix.
	(top level): Update.

Patches:
http://sourceware.org/cgi-bin/cvsweb.cgi/src/gdb/testsuite/ChangeLog.diff?cvsroot=src&r1=1.3752&r2=1.3753
http://sourceware.org/cgi-bin/cvsweb.cgi/src/gdb/testsuite/gdb.trace/backtrace.exp.diff?cvsroot=src&r1=1.28&r2=1.29
Comment 6 Sourceware Commits 2014-08-09 00:46:59 UTC
This is an automated email from the git hooks/post-receive script. It was
generated because a ref change was pushed to the repository containing
the project "gdb and binutils".

The branch, master has been updated
       via  5792e8e37be2f848152115fd4783b0e28b5253ad (commit)
      from  26278bb8717a16139b3a4da9ba1cd2aed435f1ee (commit)

Those revisions listed above that are new to this repository have
not appeared on any other notification email; so we list those
revisions in full, below.

- Log -----------------------------------------------------------------
https://sourceware.org/git/gitweb.cgi?p=binutils-gdb.git;h=5792e8e37be2f848152115fd4783b0e28b5253ad

commit 5792e8e37be2f848152115fd4783b0e28b5253ad
Author: Yao Qi <yao@codesourcery.com>
Date:   Thu Aug 7 15:01:22 2014 +0800

    Make test messages in gdb.mi/mi-var-display.exp unique
    
    When I fix a bug in gdb.mi/mi-var-display.exp, I find its test
    messages aren't unique, which makes some confusions for me.
    
    $ cat testsuite/gdb.sum | grep "PASS" | sort | uniq -c | sort -n
    ...
     2 PASS: gdb.mi/mi-var-display.exp: set format variable bar
     2 PASS: gdb.mi/mi-var-display.exp: set format variable foo
     2 PASS: gdb.mi/mi-var-display.exp: set format variable weird.func_ptr
     2 PASS: gdb.mi/mi-var-display.exp: set format variable weird.func_ptr_ptr
     2 PASS: gdb.mi/mi-var-display.exp: show format variable foo
     3 PASS: gdb.mi/mi-var-display.exp: eval variable foo
    
    This patch is to make test messages in mi-var-display.exp unique.
    
    gdb/testsuite:
    
    2014-08-09  Yao Qi  <yao@codesourcery.com>
    
    	PR testsuite/13443
    	* gdb.mi/mi-var-display.exp: Make test messages unique.

-----------------------------------------------------------------------

Summary of changes:
 gdb/testsuite/ChangeLog                 |    5 ++++
 gdb/testsuite/gdb.mi/mi-var-display.exp |   32 +++++++++++++++---------------
 2 files changed, 21 insertions(+), 16 deletions(-)