Re: [2.20] [2/6] Generate .test-result files for tests with special rules

["Carlos O'Donell" <carlos at redhat dot com>]

On 01/09/2014 09:11 PM, Joseph S. Myers wrote:
> This patch extends the generation of PASS and FAIL status in
> .test-result files for individual tests to cover tests with their own
> custom makefile rules.
> 
> In general it's just adding $(evaluate-test) calls to all such rules.
> Where a rule had multiple commands, they need linking in the form
> 
> 	command1 && command2; $(evaluate-test)
> 
> so that $(evaluate-test) still gets run even if the first command
> fails and so the second isn't run at all.
> 
> Note that the tests the makefiles expect to fail (posix/annexc and
> conformtest) currently get FAIL listed in the .test-result file,
> rather than XFAIL; patch 3 will introduce a better XFAIL mechanism.
> 
> There are a few stylistic questions here:
> 
> * Should $(evaluate-test) go on the same line as the previous part of
>   the command if it fits, or always on a separate line (or on a
>   separate line unless the whole thing fits on a single line)?  (Patch
>   1 uses a single line; this patch generally puts it on the same line
>   as the previous part of the command if it fits.)

Always on a separate line, though separate line unless the whole thing
fits is also OK.

> * Where && is used between commands, should it go at the start of a
>   line (normal GNU style, and used in this patch) or at the end of a
>   line (used in various places in existing makefiles), and if at the
>   start, should the continuation line be indented?

GNU style.
 
> * Should all tests that generate a .out file and then further examine
>   it (e.g. comparing with an expected file) have their makefile rules
>   split (in a separate patch) into part that generates the file and
>   part that examines its contents (the two parts each having their own
>   test results)?  This would reduce the need for use of && in test
>   commands.

I think effectively splitting them into two tests is best.

> * Should all the tests generating output files not named
>   <something>.out be changed (in a separate patch) to use the .out
>   naming convention?

Yes.

> Tested x86_64.
> 
> 2014-01-10  Joseph Myers  <joseph@codesourcery.com>
> 
> 	* Makefile ($(objpfx)c++-types-check.out): Use $(evaluate-test).
> 	($(objpfx)check-local-headers.out): Likewise.
> 	($(objpfx)begin-end-check.out): Likewise.
> 	* Makerules (check-abi-%): Likewise.
> 	* catgets/Makefile ($(objpfx)test1.cat): Likewise.
> 	($(objpfx)test2.cat): Likewise.
> 	($(objpfx)de/libc.cat): Likewise.
> 	($(objpfx)test-gencat.out): Likewise.
> 	* conform/Makefile ($(objpfx)run-conformtest.out): Likewise.
> 	* elf/Makefile ($(objpfx)order.out): Likewise.
> 	($(objpfx)noload-mem): Likewise.
> 	($(objpfx)tst-pathopt.out): Likewise.
> 	($(objpfx)tst-rtld-load-self.out): Likewise.
> 	($(objpfx)tst-array1.out): Likewise.
> 	($(objpfx)tst-array1-static.out): Likewise.
> 	($(objpfx)tst-array2.out): Likewise.
> 	($(objpfx)tst-array3.out): Likewise.
> 	($(objpfx)tst-array4.out): Likewise.
> 	($(objpfx)tst-array5.out): Likewise.
> 	($(objpfx)tst-array5-static.out): Likewise.
> 	($(objpfx)check-textrel.out:): Likewise.
> 	($(objpfx)check-execstack.out): Likewise.
> 	($(objpfx)check-localplt.out): Likewise.
> 	($(objpfx)order2.out): Likewise.
> 	($(objpfx)tst-leaks1-mem): Likewise.
> 	($(objpfx)tst-leaks1-static-mem): Likewise.
> 	($(objpfx)tst-initorder.out): Likewise.
> 	($(objpfx)tst-initorder2.out): Likewise.
> 	($(objpfx)tst-unused-dep.out): Likewise.
> 	* grp/Makefile ($(objpfx)tst_fgetgrent.out): Likewise.
> 	* iconv/Makefile (test-iconvconfig): Likewise.
> 	* iconvdata/Makefile ($(objpfx)mtrace-tst-loading): Likewise.
> 	($(objpfx)iconv-test.out): Likewise.
> 	($(objpfx)tst-tables.out): Likewise.
> 	* intl/Makefile ($(objpfx)mtrace-tst-gettext): Likewise.
> 	($(objpfx)tst-gettext.out): Likewise.
> 	($(objpfx)tst-translit.out): Likewise.
> 	($(objpfx)tst-gettext2.out): Likewise.
> 	($(objpfx)tst-gettext4.out): Likewise.
> 	($(objpfx)tst-gettext6.out): Likewise.
> 	* io/Makefile ($(objpfx)ftwtest.out): Likewise.
> 	* libio/Makefile ($(objpfx)test-freopen.out): Likewise.
> 	($(objpfx)tst-fopenloc.check): Likewise.
> 	* malloc/Makefile ($(objpfx)tst-mtrace.out): Likewise.
> 	* misc/Makefile ($(objpfx)tst-error1-mem): Likewise.
> 	* posix/Makefile ($(objpfx)globtest.out): Likewise.
> 	($(objpfx)wordexp-tst.out): Likewise.
> 	($(objpfx)annexc.out): Likewise.
> 	($(objpfx)tst-fnmatch-mem): Likewise.
> 	($(objpfx)bug-regex2-mem): Likewise.
> 	($(objpfx)bug-regex14-mem): Likewise.
> 	($(objpfx)bug-regex21-mem): Likewise.
> 	($(objpfx)bug-regex31-mem): Likewise.
> 	($(objpfx)tst-vfork3-mem): Likewise.
> 	($(objpfx)tst-rxspencer-mem): Likewise.
> 	($(objpfx)tst-pcre-mem): Likewise.
> 	($(objpfx)tst-boost-mem): Likewise.
> 	($(objpfx)tst-getconf.out): Likewise.
> 	($(objpfx)bug-ga2-mem): Likewise.
> 	($(objpfx)bug-glob2-mem): Likewise.
> 	* resolv/Makefile ($(objpfx)mtrace-tst-leaks): Likewise.
> 	($(objpfx)mtrace-tst-leaks2): Likewise.
> 	* stdio-common/Makefile ($(objpfx)tst-unbputc.out): Likewise.
> 	($(objpfx)tst-printf.out): Likewise.
> 	($(objpfx)tst-setvbuf1.out): Likewise.
> 	* stdlib/Makefile ($(objpfx)isomac.out): Likewise.
> 	($(objpfx)tst-fmtmsg.out): Likewise.
> 	* string/Makefile ($(objpfx)tst-svc.out): Likewise.
> 	* sysdeps/x86/Makefile ($(objpfx)tst-xmmymm.out): Likewise.
> 
> localedata:
> 2014-01-10  Joseph Myers  <joseph@codesourcery.com>
> 
> 	* Makefile ($(objpfx)sort-test.out): Use $(evaluate-test).
> 	($(objpfx)tst-fmon.out): Likewise.
> 	($(objpfx)tst-numeric.out): Likewise.
> 	($(objpfx)tst-locale.out): Likewise.
> 	($(objpfx)tst-rpmatch.out): Likewise.
> 	($(objpfx)tst-trans.out): Likewise.
> 	($(objpfx)tst-mbswcs.out): Likewise.
> 	($(objpfx)tst-ctype.out): Likewise.
> 	($(objpfx)tst-wctype.out): Likewise.
> 	($(objpfx)tst-langinfo.out): Likewise.
> 	($(objpfx)mtrace-tst-leaks): Likewise.
> nptl:
> 2014-01-10  Joseph Myers  <joseph@codesourcery.com>
> 
> 	* nptl/Makefile ($(objpfx)tst-stack3-mem): Use $(evaluate-test).
> 	($(objpfx)tst-tls6.out): Likewise.
> 	($(objpfx)tst-cleanup0.out): Likewise.
> 	($(objpfx)tst-cancel-wrappers.out): Likewise.
> 	($(objpfx)tst-oddstacklimit.out): Likewise.
> 
> nptl_db:
> 2014-01-10  Joseph Myers  <joseph@codesourcery.com>
> 
> 	* nptl_db/Makefile ($(objpfx)db-symbols.out): Use
> 	$(evaluate-test).

This looks good to me and is pretty mechanical.

Cheers,
Carlos.