Re: [2.20] [4/6] Enumerate tests with special rules in tests-special variable

["Carlos O'Donell" <carlos at redhat dot com>]

Joseph,

This patch is OK to checkin.

I have one nit, the use of XXX in Makefile, I'd rather see TODO:
and rewording to describe what should be fixed.

On 01/09/2014 09:13 PM, Joseph S. Myers wrote:
> In order to generate overall summaries of the results of all tests in
> the glibc testsuite, we need to identify and concatenate the files
> with the results of individual tests.
> 
> Tomas Dohnalek's patch used $(common-objpfx)*/*.test-result for this.
> However, the normal glibc approach is explicit enumeration of the
> expected set of files with a given property, rather than all files
> matching some pattern like that.  Furthermore, we would like to be
> able to mark tests as UNRESOLVED if the file with their results is for
> some reason missing (patch 5 does this), and in future we would like
> to be able to mark tests as UNSUPPORTED if they are disabled for a
> particular configuration (rather than simply having them missing from
> the list of tests as at present).  Such handling of tests that were
> not run or did not record results requires an explicit enumeration of
> tests.
> 
> For the tests following the default makefile rules, $(tests) (and
> $(xtests)) provides such an enumeration.  Others, however, are added
> directly as dependencies of the "tests" and "xtests" makefile
> targets.  This patch changes the makefiles to put them in variables
> tests-special and xtests-special, with appropriate dependencies on the
> tests listed there then being added centrally.
> 
> Those variables are used in Rules and so need to be set before Rules
> is included in a subdirectory makefile, which is often earlier in the
> makefile than the dependencies were present before.  The choice of
> whether to include a given test is also often based on variables set
> in Makeconfig, so includes of Makeconfig need to be added to some
> subdirectory makefiles that didn't have them before (and that
> previously just got it via the inclusion of Rules).
> 
> We previously discussed the question of where to include Rules; see
> the question at
> <https://sourceware.org/ml/libc-alpha/2012-11/msg00798.html>, and a
> discussion in
> <https://sourceware.org/ml/libc-alpha/2013-01/msg00337.html> of why
> Rules is included early rather than late in subdirectory makefiles.
> 
> I saw no reason why stdio-common had an indirection through targets
> do-tst-unbputc and do-tst-printf rather than having the tests target
> depend directly on the .out files.  Such indirections don't work for
> generating test summaries - each test generating a .test-result file
> needs listing in tests or tests-special - and so I removed that
> indirection.  It was similarly necessary to avoid an indirection
> through the check-abi target and get the check-abi-* targets for
> individual libraries into the tests-special variable.  The intl/ test
> $(objpfx)tst-gettext.out, previously built only because of
> dependencies from other tests, was also added to tests-special for the
> same reason.
> 
> The entries in tests-special are the full makefile targets, complete
> with $(objpfx) and .out.  If tests were named consistently with a .out
> suffix (I raised the question in path 2 of whether they should be
> whenever output files are generated; there are also a few that don't
> currently generate output files at all), then it would not be
> necessary to include those elements in this variable.
> 
> Some questions arising from this patch:
> 
> * Should we define a rule that Makeconfig is always included by
>   subdirectory makefiles immediately after they define subdir, whether
>   or not they actually need an inclusion of Makeconfig, so they do not
>   need to worry about whether conditionals on variables from
>   Makeconfig can be used in a particular part of the file?  (If we do,
>   then Rules would require that Makeconfig has already been included,
>   rather than including it.  And various makefiles wouldn't need dummy
>   "all:" targets because that in Makeconfig would always suffice.)

IMO we should always include Makeconfig in subdirectory makefiles.
It's simply the easiest solution for maintenance and homogenizes
the makefiles.

> * If a test can only be run conditionally, the additions to the tests
>   variable, or to dependencies of the tests target / the tests-special
>   variable (before / after this patch), are conditional in the
>   makefiles, but whether the makefile targets that say how to run such
>   a test are also conditional varies.  This patch doesn't change such
>   conditionals, but should we define a rule for whether such things
>   are conditional (probably saying that the makefile rules aren't,
>   just the additions to the relevant variable)?

My opinion is that no test should be conditional at the makefile level.
Rather all information about the configuration should make it into the
test and the test should then output that it was skipped and why. This
would move conditional information down into the specific tests and
keep the information in one place. Where multiple tests have to be skipped
those test should use a common test source to skip the tests as a group.
This way also the number of tests run is constant for all configurations,
only the results change.

> * Should tests that don't generate output files be changed (in a
>   separate patch or patches) to do so?

Yes. All tests should generate output files. I also think all tests
should be named in such a way that makes it easy to identify the test
e.g. starting with tst-, and not bug-. As it makes it easier for
developers to know exact what is and is not a test, grep, sed, and
do other manipulations on them for whatever purposes.

> Tested x86_64.
> 
> 2014-01-10  Joseph Myers  <joseph@codesourcery.com>
> 
> 	* Makefile (tests): Change dependencies to ....
> 	(tests-special): ... additions to this variable.
> 	(tests): Depend on $(tests-special).
> 	* Makerules (check-abi-list): New variable.
> 	(check-abi): Depend on $(check-abi-list).
> 	[$(subdir) = elf] (tests-special): Add check-abi-libc.
> 	[$(build-shared) = yes && subdir] (tests-special): Add
> 	$(check-abi-list).
> 	* Rules (tests): Depend on $(tests-special).
> 	(xtests): Depend on $(xtests-special).
> 	* catgets/Makefile: Include ../Makeconfig.
> 	(tests): Change dependencies to ....
> 	(tests-special): ... additions to this variable.
> 	* conform/Makefile: Include ../Makeconfig.
> 	(tests): Change dependencies to ....
> 	(tests-special): ... additions to this variable.
> 	* elf/Makefile (tests): Change dependencies to ....
> 	(tests-special): ... additions to this variable.
> 	* grp/Makefile (tests): Change dependencies to ....
> 	(tests-special): ... additions to this variable.
> 	* iconv/Makefile (xtests): Change dependencies to ....
> 	(xtests-special): ... additions to this variable.
> 	* iconvdata/Makefile (tests): Change dependencies to ....
> 	(tests-special): ... additions to this variable.
> 	* intl/Makefile (tests): Change dependencies to ....
> 	(tests-special): ... additions to this variable.  Also add
> 	$(objpfx)tst-gettext.out.
> 	* io/Makefile: Include ../Makeconfig.
> 	(tests): Change dependencies to ....
> 	(tests-special): ... additions to this variable.
> 	* libio/Makefile (tests): Change dependencies to ....
> 	(tests-special): ... additions to this variable.
> 	* malloc/Makefile (tests): Change dependencies to ....
> 	(tests-special): ... additions to this variable.
> 	* misc/Makefile (tests): Change dependencies to ....
> 	(tests-special): ... additions to this variable.
> 	* posix/Makefile (tests): Change dependencies to ....
> 	(tests-special): ... additions to this variable.
> 	(xtests): Change dependencies to ....
> 	(xtests-special): ... additions to this variable.
> 	* resolv/Makefile (tests): Change dependencies to ....
> 	(tests-special): ... additions to this variable.
> 	(xtests): Change dependencies to ....
> 	(xtests-special): ... additions to this variable.
> 	* stdio-common/Makefile (tests): Change dependencies to ....
> 	(tests-special): ... additions to this variable.
> 	(do-tst-unbputc): Remove target.
> 	(do-tst-printf): Likewise.
> 	* stdlib/Makefile (tests): Change dependencies to ....
> 	(tests-special): ... additions to this variable.
> 	* string/Makefile: Include ../Makeconfig.
> 	(tests): Change dependencies to ....
> 	(tests-special): ... additions to this variable.
> 	* sysdeps/x86/Makefile (tests): Change dependencies to ....
> 	(tests-special): ... additions to this variable.
> 
> localedata:
> 2014-01-10  Joseph Myers  <joseph@codesourcery.com>
> 
> 	* Makefile (tests): Change dependencies to ....
> 	(tests-special): ... additions to this variable.
> 
> nptl:
> 2014-01-10  Joseph Myers  <joseph@codesourcery.com>
> 
> 	* Makefile (tests): Change dependencies to ....
> 	(tests-special): ... additions to this variable.
> 
> nptl_db:
> 2014-01-10  Joseph Myers  <joseph@codesourcery.com>
> 
> 	* Makefile: Include ../Makeconfig.
> 	(tests): Change dependencies to ....
> 	(tests-special): ... additions to this variable.
> 
> diff --git a/Makefile b/Makefile
> index fe10c82..719f5b7 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -250,7 +250,7 @@ mostlyclean: parent-mostlyclean
>  tests-clean:
>  	@$(MAKE) subdir_testclean no_deps=t
>  
> -tests: $(objpfx)c++-types-check.out $(objpfx)check-local-headers.out
> +tests-special += $(objpfx)c++-types-check.out $(objpfx)check-local-headers.out

OK.

>  ifneq ($(CXX),no)
>  
>  vpath c++-types.data $(+sysdep_dirs)
> @@ -312,12 +312,14 @@ installed-headers = argp/argp.h assert/assert.h catgets/nl_types.h \
>  		    time/sys/time.h time/sys/timeb.h wcsmbs/wchar.h \
>  		    wctype/wctype.h
>  
> -tests: $(objpfx)begin-end-check.out
> +tests-special += $(objpfx)begin-end-check.out

OK.

>  $(objpfx)begin-end-check.out: scripts/begin-end-check.pl
>  	$(PERL) scripts/begin-end-check.pl $(installed-headers) > $@; \
>  	$(evaluate-test)
>  endif
>  
> +tests: $(tests-special)

OK.

> +
>  # The realclean target is just like distclean for the parent, but we want
>  # the subdirs to know the difference in case they care.
>  realclean distclean: parent-clean
> diff --git a/Makerules b/Makerules
> index d818d7f..b1ff5e1 100644
> --- a/Makerules
> +++ b/Makerules
> @@ -1207,7 +1207,8 @@ endef
>  
>  .PHONY: update-abi check-abi
>  update-abi: $(patsubst %.so,update-abi-%,$(install-lib.so-versioned))
> -check-abi: $(patsubst %.so,check-abi-%,$(install-lib.so-versioned))
> +check-abi-list = $(patsubst %.so,check-abi-%,$(install-lib.so-versioned))
> +check-abi: $(check-abi-list)
>  ifdef subdir
>  subdir_check-abi: check-abi
>  subdir_update-abi: update-abi
> @@ -1218,13 +1219,14 @@ endif
>  
>  ifeq ($(subdir),elf)
>  check-abi: check-abi-libc
> +tests-special += check-abi-libc
>  update-abi: update-abi-libc
>  common-generated += libc.symlist
>  endif
>  
>  ifeq ($(build-shared),yes)
>  ifdef subdir
> -tests: check-abi
> +tests-special += $(check-abi-list)
>  endif
>  endif
>  

OK.

> diff --git a/Rules b/Rules
> index 00e4a36..a5015c3 100644
> --- a/Rules
> +++ b/Rules
> @@ -95,11 +95,11 @@ else
>  others: $(addprefix $(objpfx),$(extra-objs))
>  endif
>  ifeq ($(run-built-tests),no)
> -tests: $(addprefix $(objpfx),$(tests) $(test-srcs))
> -xtests: tests
> +tests: $(addprefix $(objpfx),$(tests) $(test-srcs)) $(tests-special)
> +xtests: tests $(xtests-special)

OK.

>  else
> -tests: $(tests:%=$(objpfx)%.out)
> -xtests: tests $(xtests:%=$(objpfx)%.out)
> +tests: $(tests:%=$(objpfx)%.out) $(tests-special)
> +xtests: tests $(xtests:%=$(objpfx)%.out) $(xtests-special)
>  endif
>  
>  ifeq ($(build-programs),yes)
> diff --git a/catgets/Makefile b/catgets/Makefile
> index 17ef907..7c11e36 100644
> --- a/catgets/Makefile
> +++ b/catgets/Makefile
> @@ -29,6 +29,13 @@ extra-objs	= $(gencat-modules:=.o)
>  tests = tst-catgets
>  test-srcs = test-gencat
>  
> +include ../Makeconfig
> +
> +ifeq ($(run-built-tests),yes)
> +tests-special += $(objpfx)de/libc.cat $(objpfx)test1.cat $(objpfx)test2.cat \
> +		 $(objpfx)test-gencat.out
> +endif

OK.

> +
>  gencat-modules	= xmalloc
>  
>  # To find xmalloc.c
> @@ -51,8 +58,6 @@ generated-dirs = de
>  tst-catgets-ENV = NLSPATH="$(objpfx)%l/%N.cat" LANG=de
>  
>  ifeq ($(run-built-tests),yes)
> -tests: $(objpfx)de/libc.cat $(objpfx)test1.cat $(objpfx)test2.cat \
> -       $(objpfx)test-gencat.out

OK.

>  # This test just checks whether the program produces any error or not.
>  # The result is not tested.
>  $(objpfx)test1.cat: test1.msg $(objpfx)gencat
> diff --git a/conform/Makefile b/conform/Makefile
> index 8e2b9d9..3f76773 100644
> --- a/conform/Makefile
> +++ b/conform/Makefile
> @@ -20,14 +20,16 @@
>  #
>  subdir	:= conform
>  
> -include ../Rules
> +include ../Makeconfig
>  
>  ifneq (yes,$(fast-check))
>  ifeq (no,$(cross-compiling))
> -tests: $(objpfx)run-conformtest.out
> +tests-special += $(objpfx)run-conformtest.out
>  endif
>  endif
>  
> +include ../Rules
> +
>  test-xfail-run-conformtest = yes
>  $(objpfx)run-conformtest.out: run-conformtest.sh conformtest.pl \
>  			      $(wildcard data/*.h-data) \
> diff --git a/elf/Makefile b/elf/Makefile
> index be5919b..dfa10ab 100644
> --- a/elf/Makefile
> +++ b/elf/Makefile
> @@ -157,8 +157,8 @@ tests-execstack-yes = tst-execstack tst-execstack-needed tst-execstack-prog
>  endif
>  endif
>  ifeq ($(run-built-tests),yes)
> -tests: $(objpfx)tst-leaks1-mem $(objpfx)tst-leaks1-static-mem \
> -       $(objpfx)noload-mem
> +tests-special += $(objpfx)tst-leaks1-mem $(objpfx)tst-leaks1-static-mem \
> +		 $(objpfx)noload-mem
>  endif
>  tlsmod17a-suffixes = 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19
>  tlsmod18a-suffixes = 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19
> @@ -252,12 +252,24 @@ modules-names += ifuncmod1 ifuncmod3 ifuncmod5 ifuncmod6
>  endif
>  endif
>  
> +ifeq (yes,$(build-shared))
> +ifeq ($(run-built-tests),yes)
> +tests-special += $(objpfx)tst-pathopt.out $(objpfx)tst-rtld-load-self.out
> +endif
> +tests-special += $(objpfx)check-textrel.out $(objpfx)check-execstack.out \
> +		 $(objpfx)check-localplt.out
> +endif
>  
> -include ../Rules
> +ifeq ($(run-built-tests),yes)
> +tests-special += $(objpfx)tst-unused-dep.out
> +endif
>  
>  check-abi: check-abi-ld
> +tests-special += check-abi-ld
>  update-abi: update-abi-ld
>  
> +include ../Rules
> +
>  ifeq (yes,$(build-shared))
>  # Make sure these things are built in the `make lib' pass so they can be used
>  # to run programs during the `make others' pass.
> @@ -429,12 +441,6 @@ CPPFLAGS-.os += $(if $(filter $(@F),$(patsubst %,%.os,$(all-rtld-routines))),\
>  test-modules = $(addprefix $(objpfx),$(addsuffix .so,$(strip $(modules-names))))
>  generated += $(addsuffix .so,$(strip $(modules-names)))
>  
> -ifeq (yes,$(build-shared))
> -ifeq ($(run-built-tests),yes)
> -tests: $(objpfx)tst-pathopt.out $(objpfx)tst-rtld-load-self.out
> -endif
> -endif
> -
>  $(objpfx)testobj1.so: $(libdl)
>  $(objpfx)testobj1_1.so: $(objpfx)testobj1.so $(libdl)
>  $(objpfx)testobj2.so: $(objpfx)testobj1.so $(libdl)
> @@ -889,8 +895,6 @@ CFLAGS-tst-pie1.c += $(pie-ccflag)
>  $(objpfx)tst-pie1: $(objpfx)tst-piemod1.so
>  
>  ifeq (yes,$(build-shared))
> -tests: $(objpfx)check-textrel.out $(objpfx)check-execstack.out
> -
>  all-built-dso := $(common-objpfx)libc.so \
>  		 $(filter-out $(common-objpfx)linkobj/libc.so, \
>  			      $(sort $(wildcard $(addprefix $(common-objpfx), \
> @@ -946,8 +950,6 @@ $(all-built-dso:=.jmprel): %.jmprel: % Makefile
>  	mv -f $@T $@
>  common-generated += $(all-built-dso:$(common-objpfx)%=%.jmprel)
>  
> -tests: $(objpfx)check-localplt.out
> -
>  localplt-built-dso := $(addprefix $(common-objpfx),\
>  				  libc.so \
>  				  math/libm.so \
> @@ -1140,10 +1142,6 @@ $(objpfx)tst-relsort1mod2.so: $(libm)
>  $(objpfx)tst-relsort1.out: $(objpfx)tst-relsort1mod1.so \
>  			   $(objpfx)tst-relsort1mod2.so
>  
> -ifeq ($(run-built-tests),yes)
> -tests: $(objpfx)tst-unused-dep.out
> -endif
> -
>  $(objpfx)tst-unused-dep.out: $(objpfx)testobj1.so
>  	$(test-wrapper-env) \
>  	LD_TRACE_LOADED_OBJECTS=1 \
> diff --git a/grp/Makefile b/grp/Makefile
> index 880cff9..2252ced 100644
> --- a/grp/Makefile
> +++ b/grp/Makefile
> @@ -31,6 +31,9 @@ tests := testgrp
>  
>  ifeq (yes,$(build-shared))
>  test-srcs :=  tst_fgetgrent
> +ifeq ($(run-built-tests),yes)
> +tests-special += $(objpfx)tst_fgetgrent.out
> +endif

OK.

>  endif
>  
>  
> @@ -53,7 +56,6 @@ endif
>  ifeq ($(run-built-tests),yes)
>  # tst_fgetgrent currently only works with shared libraries
>  ifeq (yes,$(build-shared))
> -tests: $(objpfx)tst_fgetgrent.out

OK.

>  $(objpfx)tst_fgetgrent.out: tst_fgetgrent.sh $(objpfx)tst_fgetgrent
>  	$(SHELL) $< $(common-objpfx) '$(test-program-prefix)'; $(evaluate-test)
>  
> diff --git a/iconv/Makefile b/iconv/Makefile
> index 03e2b41..d5d9d25 100644
> --- a/iconv/Makefile
> +++ b/iconv/Makefile
> @@ -60,6 +60,10 @@ CPPFLAGS-strtab = -DNOT_IN_libc
>  CPPFLAGS-charmap = -DNOT_IN_libc
>  CPPFLAGS-charmap-dir = -DNOT_IN_libc
>  
> +ifeq ($(run-built-tests),yes)
> +xtests-special += test-iconvconfig

OK.

> +endif
> +
>  include ../Rules
>  
>  $(inst_bindir)/iconv: $(objpfx)iconv_prog $(+force)
> @@ -68,10 +72,6 @@ $(inst_bindir)/iconv: $(objpfx)iconv_prog $(+force)
>  $(objpfx)iconv_prog: $(iconv_prog-modules:%=$(objpfx)%.o)
>  $(objpfx)iconvconfig: $(iconvconfig-modules:%=$(objpfx)%.o)
>  
> -ifeq ($(run-built-tests),yes)
> -xtests: test-iconvconfig
> -endif
> -
>  .PHONY: test-iconvconfig
>  test-iconvconfig: /dev/null $(objpfx)iconvconfig
>  	tmp=$(objpfx)gconv-modules.cache.$$$$; \
> diff --git a/iconvdata/Makefile b/iconvdata/Makefile
> index 34110da..ce2b30d 100644
> --- a/iconvdata/Makefile
> +++ b/iconvdata/Makefile
> @@ -265,17 +265,17 @@ endif
>  
>  endif # build-shared = yes
>  
> -include ../Rules
> -
>  ifeq ($(run-built-tests),yes)
>  ifeq (yes,$(build-shared))
> -tests: $(objpfx)iconv-test.out $(objpfx)tst-tables.out
> +tests-special += $(objpfx)iconv-test.out $(objpfx)tst-tables.out
>  ifneq (no,$(PERL))
> -tests: $(objpfx)mtrace-tst-loading
> +tests-special += $(objpfx)mtrace-tst-loading
>  endif
>  endif
>  endif
>  
> +include ../Rules
> +
>  tst-loading-ENV = MALLOC_TRACE=$(objpfx)tst-loading.mtrace
>  $(objpfx)mtrace-tst-loading: $(objpfx)tst-loading.out
>  	$(common-objpfx)malloc/mtrace $(objpfx)tst-loading.mtrace > $@; \
> diff --git a/intl/Makefile b/intl/Makefile
> index fbe4ff4..aaa3dce 100644
> --- a/intl/Makefile
> +++ b/intl/Makefile
> @@ -46,22 +46,23 @@ plural.c: plural.y
>  endif
>  $(objpfx)plural.o: plural.c
>  
> -include ../Rules
> -
>  ifeq ($(run-built-tests),yes)
>  ifeq (yes,$(build-shared))
>  ifneq ($(strip $(MSGFMT)),:)
> -tests: $(objpfx)tst-translit.out $(objpfx)tst-gettext2.out \
> -       $(objpfx)tst-codeset.out $(objpfx)tst-gettext3.out
> +tests-special += $(objpfx)tst-translit.out $(objpfx)tst-gettext.out \
> +		 $(objpfx)tst-gettext2.out $(objpfx)tst-codeset.out \
> +		 $(objpfx)tst-gettext3.out
>  ifeq ($(have-thread-library),yes)
> -tests: $(objpfx)tst-gettext4.out $(objpfx)tst-gettext5.out \
> -       $(objpfx)tst-gettext6.out
> +tests-special += $(objpfx)tst-gettext4.out $(objpfx)tst-gettext5.out \
> +		 $(objpfx)tst-gettext6.out
>  endif
>  ifneq (no,$(PERL))
> -tests: $(objpfx)mtrace-tst-gettext
> +tests-special += $(objpfx)mtrace-tst-gettext
>  endif
>  endif
>  
> +include ../Rules
> +
>  # Multiple tests use this data.  Create it once to avoid racing and
>  # spurious test failures.
>  codeset_mo = $(objpfx)domaindir/de_DE/LC_MESSAGES/codeset.mo
> diff --git a/io/Makefile b/io/Makefile
> index bb32b86..8a6562e 100644
> --- a/io/Makefile
> +++ b/io/Makefile
> @@ -20,6 +20,8 @@
>  #
>  subdir	:= io
>  
> +include ../Makeconfig
> +
>  headers := sys/stat.h bits/stat.h sys/statfs.h bits/statfs.h sys/vfs.h \
>  	   sys/statvfs.h bits/statvfs.h fcntl.h sys/fcntl.h bits/fcntl.h \
>  	   poll.h sys/poll.h bits/poll.h bits/fcntl2.h bits/poll2.h \
> @@ -71,6 +73,10 @@ tests		:= test-utime test-stat test-stat2 test-lfs tst-getcwd \
>  		   tst-mknodat tst-mkfifoat tst-ttyname_r bug-ftw5 \
>  		   tst-posix_fallocate
>  
> +ifeq ($(run-built-tests),yes)
> +tests-special += $(objpfx)ftwtest.out
> +endif
> +
>  include ../Rules
>  
>  CFLAGS-open.c = -fexceptions -fasynchronous-unwind-tables
> @@ -101,8 +107,6 @@ test-stat2-ARGS = Makefile . $(objpfx)test-stat2
>  tst-statvfs-ARGS = $(objpfx)tst-statvfs tst-statvfs.c /tmp
>  
>  ifeq ($(run-built-tests),yes)
> -tests: $(objpfx)ftwtest.out
> -
>  $(objpfx)ftwtest.out: ftwtest-sh $(objpfx)ftwtest
>  	$(SHELL) $< $(common-objpfx) '$(test-program-cmd)' > $@; \
>  	$(evaluate-test)
> diff --git a/libio/Makefile b/libio/Makefile
> index d892eb8..b59dd33 100644
> --- a/libio/Makefile
> +++ b/libio/Makefile
> @@ -174,17 +174,17 @@ shared-only-routines = oldiofopen oldiofdopen oldiofclose oldfileops	\
>  		       oldiofgetpos oldiofgetpos64 oldiofsetpos		\
>  		       oldiofsetpos64
>  
> -include ../Rules
> -
>  ifeq ($(run-built-tests),yes)
> -tests: $(objpfx)test-freopen.out
> +tests-special += $(objpfx)test-freopen.out
>  ifeq (yes,$(build-shared))
>  # Run tst-fopenloc.check only if shared library is enabled since it
>  # depends on tst-fopenloc.out.
> -tests: $(objpfx)tst-fopenloc.check
> +tests-special += $(objpfx)tst-fopenloc.check
>  endif
>  endif
>  
> +include ../Rules
> +
>  $(objpfx)test-freopen.out: test-freopen.sh $(objpfx)test-freopen
>  	$(SHELL) $< $(common-objpfx) '$(test-program-prefix)'	\
>  	$(common-objpfx)libio/; $(evaluate-test)
> diff --git a/localedata/Makefile b/localedata/Makefile
> index de1d5c7..cfd6539 100644
> --- a/localedata/Makefile
> +++ b/localedata/Makefile
> @@ -82,7 +82,7 @@ tests-static = bug-setlocale1-static
>  tests += $(tests-static)
>  ifeq (yes,$(build-shared))
>  ifneq (no,$(PERL))
> -tests: $(objpfx)mtrace-tst-leaks
> +tests-special += $(objpfx)mtrace-tst-leaks
>  endif
>  endif
>  endif
> @@ -94,6 +94,12 @@ install-others := $(addprefix $(inst_i18ndir)/, \
>  
>  tests: $(objdir)/iconvdata/gconv-modules
>  
> +tests-special += $(objpfx)sort-test.out $(objpfx)tst-fmon.out \
> +		 $(objpfx)tst-locale.out $(objpfx)tst-rpmatch.out \
> +		 $(objpfx)tst-trans.out $(objpfx)tst-mbswcs.out \
> +		 $(objpfx)tst-ctype.out $(objpfx)tst-wctype.out \
> +		 $(objpfx)tst-langinfo.out $(objpfx)tst-numeric.out
> +
>  include ../Rules
>  
>  # Install the charmap files in gzipped format.
> @@ -140,11 +146,6 @@ $(addprefix $(objpfx),$(CTYPE_FILES)): %: \
>  $(addsuffix .out,$(addprefix $(objpfx),$(tests))): %: \
>    $(addprefix $(objpfx),$(CTYPE_FILES))
>  
> -tests: $(objpfx)sort-test.out $(objpfx)tst-fmon.out $(objpfx)tst-locale.out \
> -       $(objpfx)tst-rpmatch.out $(objpfx)tst-trans.out \
> -       $(objpfx)tst-mbswcs.out $(objpfx)tst-ctype.out $(objpfx)tst-wctype.out \
> -       $(objpfx)tst-langinfo.out $(objpfx)tst-numeric.out
> -
>  $(objpfx)sort-test.out: sort-test.sh $(objpfx)collate-test $(objpfx)xfrm-test \
>  		       $(test-input-data) $(addprefix $(objpfx),$(CTYPE_FILES))
>  	$(SHELL) $< $(common-objpfx) '$(test-program-prefix)' $(test-input) \
> diff --git a/malloc/Makefile b/malloc/Makefile
> index c1dbbde..65ee274 100644
> --- a/malloc/Makefile
> +++ b/malloc/Makefile
> @@ -94,6 +94,14 @@ memusagestat-modules = memusagestat
>  $(objpfx)memusagestat: $(memusagestat-modules:%=$(objpfx)%.o)
>  	$(LINK.o) -o $@ $^ $(libgd-LDFLAGS) -lgd -lpng -lz -lm
>  
> +ifeq ($(run-built-tests),yes)
> +ifeq (yes,$(build-shared))
> +ifneq ($(PERL),no)
> +tests-special += $(objpfx)tst-mtrace.out
> +endif
> +endif
> +endif
> +
>  include ../Rules
>  
>  CFLAGS-mcheck-init.c = $(PIC-ccflag)
> @@ -109,7 +117,6 @@ lib: $(objpfx)libmcheck.a
>  ifeq ($(run-built-tests),yes)
>  ifeq (yes,$(build-shared))
>  ifneq ($(PERL),no)
> -tests: $(objpfx)tst-mtrace.out
>  $(objpfx)tst-mtrace.out: tst-mtrace.sh $(objpfx)tst-mtrace
>  	$(SHELL) $< $(common-objpfx) '$(test-program-prefix)'; $(evaluate-test)
>  endif
> diff --git a/misc/Makefile b/misc/Makefile
> index 66d0ab7..604e743 100644
> --- a/misc/Makefile
> +++ b/misc/Makefile
> @@ -78,7 +78,7 @@ gpl2lgpl := error.c error.h
>  tests := tst-dirname tst-tsearch tst-fdset tst-efgcvt tst-mntent tst-hsearch \
>  	 tst-error1 tst-pselect tst-insremque tst-mntent2 bug-hsearch1
>  ifeq ($(run-built-tests),yes)
> -tests: $(objpfx)tst-error1-mem
> +tests-special += $(objpfx)tst-error1-mem
>  endif
>  
>  CFLAGS-select.c = -fexceptions -fasynchronous-unwind-tables
> diff --git a/nptl/Makefile b/nptl/Makefile
> index be4e180..9a3e4a6 100644
> --- a/nptl/Makefile
> +++ b/nptl/Makefile
> @@ -369,6 +369,13 @@ xtests-static += tst-setuid1-static
>  # These tests are linked with libc before libpthread
>  tests-reverse += tst-cancel5 tst-cancel23 tst-vfork1x tst-vfork2x
>  
> +ifeq ($(run-built-tests),yes)
> +tests-special += $(objpfx)tst-stack3-mem $(objpfx)tst-oddstacklimit.out
> +ifeq ($(build-shared),yes)
> +tests-special += $(objpfx)tst-tls6.out $(objpfx)tst-cancel-wrappers.out
> +endif
> +endif
> +
>  include ../Rules
>  
>  ifeq (yes,$(build-shared))
> @@ -459,9 +466,6 @@ LDFLAGS-tst-atfork2 = -rdynamic
>  tst-atfork2-ENV = MALLOC_TRACE=$(objpfx)tst-atfork2.mtrace
>  $(objpfx)tst-atfork2mod.so: $(shared-thread-library)
>  
> -ifeq ($(run-built-tests),yes)
> -tests: $(objpfx)tst-stack3-mem
> -endif
>  tst-stack3-ENV = MALLOC_TRACE=$(objpfx)tst-stack3.mtrace
>  $(objpfx)tst-stack3-mem: $(objpfx)tst-stack3.out
>  	$(common-objpfx)malloc/mtrace $(objpfx)tst-stack3.mtrace > $@; \
> @@ -490,9 +494,6 @@ $(objpfx)tst-tls7.out: $(objpfx)tst-tls7mod.so
>  $(objpfx)tst-tls7mod.so: $(shared-thread-library)
>  
>  ifeq ($(build-shared),yes)
> -ifeq ($(run-built-tests),yes)
> -tests: $(objpfx)tst-tls6.out
> -endif
>  $(objpfx)tst-tls6.out: tst-tls6.sh $(objpfx)tst-tls5 \
>  		       $(objpfx)tst-tls5moda.so $(objpfx)tst-tls5modb.so \
>  		       $(objpfx)tst-tls5modc.so $(objpfx)tst-tls5modd.so \
> @@ -583,7 +584,6 @@ endif
>  
>  ifeq ($(run-built-tests),yes)
>  ifeq (yes,$(build-shared))
> -tests: $(objpfx)tst-cancel-wrappers.out
>  $(objpfx)tst-cancel-wrappers.out: tst-cancel-wrappers.sh
>  	$(SHELL) $< '$(NM)' \
>  		    $(common-objpfx)libc_pic.a \
> @@ -605,8 +605,6 @@ tst-stackguard1-ARGS = --command "$(host-test-program-cmd) --child"
>  tst-stackguard1-static-ARGS = --command "$(objpfx)tst-stackguard1-static --child"
>  
>  ifeq ($(run-built-tests),yes)
> -tests: $(objpfx)tst-oddstacklimit.out
> -
>  $(objpfx)tst-oddstacklimit.out: $(objpfx)tst-oddstacklimit $(objpfx)tst-basic1
>  	$(test-program-prefix) $< --command '$(host-test-program-cmd)' > $@; \
>  	$(evaluate-test)
> diff --git a/nptl_db/Makefile b/nptl_db/Makefile
> index 5d20dd5..ff873ca 100644
> --- a/nptl_db/Makefile
> +++ b/nptl_db/Makefile
> @@ -19,6 +19,8 @@
>  
>  subdir          := nptl_db
>  
> +include ../Makeconfig
> +
>  nptl_db-version = 1.0
>  
>  extra-libs = libthread_db
> @@ -49,9 +51,10 @@ libthread_db-inhibit-o = $(filter-out .os,$(object-suffixes))
>  # The ps_* callback functions are not defined.
>  libthread_db.so-no-z-defs = yes
>  
> +tests-special += $(objpfx)db-symbols.out
> +
>  include ../Rules
>  
> -tests: $(objpfx)db-symbols.out
>  $(objpfx)db-symbols.out: $(objpfx)db-symbols.v.i \
>  			 $(common-objpfx)nptl/libpthread.so
>  	LC_ALL=C $(READELF) -W -s $(filter %.so,$^) | $(AWK) -f $< > $@; \
> diff --git a/posix/Makefile b/posix/Makefile
> index 1eda4fb..7eb9642 100644
> --- a/posix/Makefile
> +++ b/posix/Makefile
> @@ -112,12 +112,30 @@ generated := $(addprefix wordexp-test-result, 1 2 3 4 5 6 7 8 9 10) \
>  	     tst-vfork3-mem tst-vfork3.mtrace getconf.speclist \
>  	     tst-fnmatch-mem tst-fnmatch.mtrace
>  
> +ifeq ($(run-built-tests),yes)
> +ifeq (yes,$(build-shared))
> +tests-special += $(objpfx)globtest.out $(objpfx)wordexp-tst.out
> +endif
> +endif
> +
> +# Run a test on the header files we use.
> +# XXX Please note that for now we ignore the result of this test.

Nit. Please make this a TODO: marker and describe briefly that we need
to check the result of this test, which we presently ignore.

> +tests-special += $(objpfx)annexc.out
> +ifeq ($(run-built-tests),yes)
> +tests-special += $(objpfx)bug-regex2-mem $(objpfx)bug-regex14-mem \
> +		 $(objpfx)bug-regex21-mem $(objpfx)bug-regex31-mem \
> +		 $(objpfx)tst-rxspencer-mem $(objpfx)tst-pcre-mem \
> +		 $(objpfx)tst-boost-mem $(objpfx)tst-getconf.out \
> +		 $(objpfx)bug-glob2-mem $(objpfx)tst-vfork3-mem \
> +		 $(objpfx)tst-fnmatch-mem
> +xtests-special += $(objpfx)bug-ga2-mem

OK.

> +endif
> +
>  include ../Rules
>  
>  ifeq ($(run-built-tests),yes)
>  # globtest and wordexp-test currently only works with shared libraries
>  ifeq (yes,$(build-shared))
> -tests: $(objpfx)globtest.out $(objpfx)wordexp-tst.out
>  $(objpfx)globtest.out: globtest.sh $(objpfx)globtest
>  	$(SHELL) $< $(common-objpfx) '$(test-via-rtld-prefix)' \
>  		'$(test-program-prefix)' '$(test-wrapper-env)'; \
> @@ -217,17 +235,6 @@ ptestcases.h: PTESTS PTESTS2C.sed
>  	LC_ALL=C sed -f PTESTS2C.sed < $< > $@T
>  	mv -f $@T $@
>  
> -# Run a test on the header files we use.
> -# XXX Please note that for now we ignore the result of this test.
> -tests: $(objpfx)annexc.out
> -ifeq ($(run-built-tests),yes)
> -tests: $(objpfx)bug-regex2-mem $(objpfx)bug-regex14-mem \
> -  $(objpfx)bug-regex21-mem $(objpfx)bug-regex31-mem $(objpfx)tst-rxspencer-mem\
> -  $(objpfx)tst-pcre-mem $(objpfx)tst-boost-mem $(objpfx)tst-getconf.out \
> -  $(objpfx)bug-glob2-mem $(objpfx)tst-vfork3-mem $(objpfx)tst-fnmatch-mem
> -xtests: $(objpfx)bug-ga2-mem
> -endif
> -
>  test-xfail-annexc = yes
>  $(objpfx)annexc.out: $(objpfx)annexc
>  	$(dir $<)$(notdir $<) '$(CC)' \
> diff --git a/resolv/Makefile b/resolv/Makefile
> index 7a102cd..273b7c2 100644
> --- a/resolv/Makefile
> +++ b/resolv/Makefile
> @@ -64,6 +64,13 @@ ifeq (yesyes,$(build-shared)$(have-thread-library))
>  tests: $(objpfx)ga_test
>  endif
>  
> +ifeq ($(run-built-tests),yes)
> +ifneq (no,$(PERL))
> +tests-special += $(objpfx)mtrace-tst-leaks
> +xtests-special += $(objpfx)mtrace-tst-leaks2
> +endif
> +endif
> +
>  generated := mtrace-tst-leaks tst-leaks.mtrace \
>  	     mtrace-tst-leaks2 tst-leaks2.mtrace
>  
> @@ -96,18 +103,8 @@ tst-leaks-ENV = MALLOC_TRACE=$(objpfx)tst-leaks.mtrace
>  $(objpfx)mtrace-tst-leaks: $(objpfx)tst-leaks.out
>  	$(common-objpfx)malloc/mtrace $(objpfx)tst-leaks.mtrace > $@; \
>  	$(evaluate-test)
> -ifeq ($(run-built-tests),yes)
> -ifneq (no,$(PERL))
> -tests: $(objpfx)mtrace-tst-leaks
> -endif
> -endif
>  
>  tst-leaks2-ENV = MALLOC_TRACE=$(objpfx)tst-leaks2.mtrace
>  $(objpfx)mtrace-tst-leaks2: $(objpfx)tst-leaks2.out
>  	$(common-objpfx)malloc/mtrace $(objpfx)tst-leaks2.mtrace > $@; \
>  	$(evaluate-test)
> -ifeq ($(run-built-tests),yes)
> -ifneq (no,$(PERL))
> -xtests: $(objpfx)mtrace-tst-leaks2
> -endif
> -endif
> diff --git a/stdio-common/Makefile b/stdio-common/Makefile
> index c990ffe..fa14d1d 100644
> --- a/stdio-common/Makefile
> +++ b/stdio-common/Makefile
> @@ -61,17 +61,16 @@ tests := tstscanf test_rdwr test-popen tstgetln test-fseek \
>  
>  test-srcs = tst-unbputc tst-printf
>  
> +ifeq ($(run-built-tests),yes)
> +tests-special += $(objpfx)tst-unbputc.out $(objpfx)tst-printf.out
> +endif
> +
>  include ../Rules
>  
>  ifeq ($(run-built-tests),yes)
> -.PHONY: do-tst-unbputc do-tst-printf
> -tests: do-tst-unbputc do-tst-printf
> -
> -do-tst-unbputc: $(objpfx)tst-unbputc.out
>  $(objpfx)tst-unbputc.out: tst-unbputc.sh $(objpfx)tst-unbputc
>  	$(SHELL) $< $(common-objpfx) '$(test-program-prefix)'; $(evaluate-test)
>  
> -do-tst-printf: $(objpfx)tst-printf.out
>  $(objpfx)tst-printf.out: tst-printf.sh $(objpfx)tst-printf
>  	$(SHELL) $< $(common-objpfx) '$(test-program-prefix)'; $(evaluate-test)
>  endif
> diff --git a/stdlib/Makefile b/stdlib/Makefile
> index c23df96..be6706d 100644
> --- a/stdlib/Makefile
> +++ b/stdlib/Makefile
> @@ -111,6 +111,13 @@ CFLAGS-tst-bsearch.c = $(stack-align-test-flags)
>  CFLAGS-tst-qsort.c = $(stack-align-test-flags)
>  CFLAGS-tst-makecontext2.c = $(stack-align-test-flags)
>  
> +# Run a test on the header files we use.
> +tests-special += $(objpfx)isomac.out
> +
> +ifeq ($(run-built-tests),yes)
> +tests-special += $(objpfx)tst-fmtmsg.out
> +endif
> +
>  include ../Rules
>  
>  # Testdir has to be named stdlib and needs to be writable
> @@ -123,13 +130,6 @@ tst-strtod5-ENV = LOCPATH=$(common-objpfx)localedata
>  testmb2-ENV = LOCPATH=$(common-objpfx)localedata
>  bug-fmtmsg1-ENV = SEV_LEVEL=foo,11,newsev
>  
> -# Run a test on the header files we use.
> -tests: $(objpfx)isomac.out
> -
> -ifeq ($(run-built-tests),yes)
> -tests: $(objpfx)tst-fmtmsg.out
> -endif
> -
>  $(objpfx)isomac.out: $(objpfx)isomac
>  	$(dir $<)$(notdir $<) '$(CC)'  \
>  	'-I../include $(+sysdep-includes) $(sysincludes) -I..' > $<.out; \
> diff --git a/string/Makefile b/string/Makefile
> index ec71eaa..77b8e03 100644
> --- a/string/Makefile
> +++ b/string/Makefile
> @@ -20,6 +20,8 @@
>  #
>  subdir	:= string
>  
> +include ../Makeconfig
> +
>  headers	:= string.h strings.h memory.h endian.h bits/endian.h \
>  	   argz.h envz.h byteswap.h bits/byteswap.h bits/byteswap-16.h \
>  	   bits/string.h bits/string2.h bits/string3.h
> @@ -56,6 +58,10 @@ tests		:= tester inl-tester noinl-tester testcopy test-ffs	\
>  
>  xtests = tst-strcoll-overflow
>  
> +ifeq ($(run-built-tests),yes)
> +tests-special += $(objpfx)tst-svc.out
> +endif
> +
>  include ../Rules
>  
>  tester-ENV = LANGUAGE=C
> @@ -74,7 +80,6 @@ CFLAGS-test-ffs.c = -fno-builtin
>  CFLAGS-tst-inlcall.c = -fno-builtin
>  
>  ifeq ($(run-built-tests),yes)
> -tests: $(objpfx)tst-svc.out
>  $(objpfx)tst-svc.out: tst-svc.input $(objpfx)tst-svc
>  	GCONV_PATH=$(common-objpfx)iconvdata LC_ALL=C \
>  	  $(test-program-cmd) < $(word 1,$^) > $@ \
> diff --git a/sysdeps/x86/Makefile b/sysdeps/x86/Makefile
> index 80d03d5..a96c8bd 100644
> --- a/sysdeps/x86/Makefile
> +++ b/sysdeps/x86/Makefile
> @@ -2,7 +2,7 @@ ifeq ($(subdir),elf)
>  CFLAGS-.os += $(if $(filter $(@F),$(patsubst %,%.os,$(all-rtld-routines))),\
>  		   -mno-sse -mno-mmx)
>  
> -tests: $(objpfx)tst-xmmymm.out
> +tests-special += $(objpfx)tst-xmmymm.out
>  $(objpfx)tst-xmmymm.out: ../sysdeps/x86/tst-xmmymm.sh $(objpfx)ld.so
>  	@echo "Checking ld.so for SSE register use.  This will take a few seconds..."
>  	$(SHELL) $< $(objpfx) '$(NM)' '$(OBJDUMP)' '$(READELF)' > $@; \
> 

Cheers,
Carlos.