[PATCH 1 of 1] test-suite: Added new test suite feature (experimental)

Martin Lund mgl@doredevelopment.dk
Mon May 17 11:28:00 GMT 2010


Hello Yann, all,

Please, feel free to munge the patch and change anything you want to get 
it in :)

Actually, the patch feature in its current form has been stalled in my 
drawer for quite a while because the last time I was ready to integrate 
it the state of the HG repo was so that I could not build and test any 
toolchains. So I decided to wait until it was in a better testable 
state. Only recently I opened that drawer again, a bit later than 
expected - sorry :)

I basically agree on all your proposed changes - please see comments below.

On 2010-05-16 17:08, Yann E. MORIN wrote:
> Martin, All,
>
> I don't know where the problem lies, but I was not able to easily apply
> your base64-encoded mail. I really hope it lies on _my_ side (or it
> would rendre Hg useless to exchange non-ASCII patches...)
>
>    
Let me know if it happens again - maybe it is something that needs to be 
fixed on my side.
> On Saturday 15 May 2010 17:19:05 Martin Lund wrote:
>    
>> # HG changeset patch
>> # User Martin Lund<mgl@doredevelopment.dk>
>> # Date 1273934735 -7200
>> # Node ID 7e1196581995247e24c585653be9e65e8555d0ea
>> # Parent  48e107b35ba9c2b3f31667dd5042cc69bc472592
>> test-suite: Added new test suite feature (experimental)
>>
>> This patch adds support for installing the gcc test suite. A helper
>> Makefile is provided for building and running the gcc tests.
>>
>> The default configuration runs all gcc tests and requires automatic
>> ssh/scp login access to a networked target board. See README for
>> more details.
>>
>> Note: Current feature is tested with the powerpc-unknown-linux-gnu
>> sample but it should work with others as well.
>>      
> Missing Signed-off-by.
>    
Hg tricked me :)
> [--SNIP--]
>    
>> diff -r 48e107b35ba9 -r 7e1196581995 contrib/gcc-test-suite/Makefile
>>      
> [--SNIP--]
>    
>> +# Targets
>> +all: test
>> +
>> +gcc-testsuite-${DG_GCC_VERSION}.tar.gz:
>> +#	wget -nc ${DG_GCC_URL}
>> +
>> +gcc-${DG_GCC_VERSION}: gcc-testsuite-${DG_GCC_VERSION}.tar.gz
>> +#	tar xzf gcc-testsuite-${DG_GCC_VERSION}.tar.gz
>>      
> Those two targets are useless, we already have the test-suite isntalled.
>    

The only reason these targets are left hanging there is that the 
makefile started out as a self contained thing downloading and 
extracting the gcc test suite by itself.

Actually, if we keep the Makefile able to download and extract any 
specificed gcc test suite version (by redefining DG_GCC_VERSION etc.) 
that would provide users/developers the flexibility to test a gcc 
toolchain with a different version gcc test suite. This is useful if you 
want to test a new testcase from a newer version of the gcc test suite 
with an older version toolchain. I recognize that in most situations 
this is not needed - so I have no problems if we let go of everything 
related to this feature. It can be re-introduced it if the need presents 
itself.

>    
>> +config:
>> +	@mkdir -p ${TMPDIR}
>> +	@{ echo 'lappend boards_dir "."'; \
>> +	  echo "set target_alias ${DG_TARGET}"; }>  ${TMPDIR}/site.exp
>> +	@{ echo -e "load_generic_config \"unix\""; \
>> +	echo -e "process_multilib_options \"\"" ; \
>> +	echo "set_board_info bmk,use_alarm 1" ; \
>> +	echo "set_board_info rsh_prog ssh" ; \
>> +	echo "set_board_info rcp_prog scp" ; \
>> +	echo "set_board_info hostname ${DG_TARGET_HOSTNAME}"; \
>> +	echo "set_board_info username ${DG_TARGET_USERNAME}"; }>  ${TMPDIR}/board.exp
>>      
> Split the two rules:
>
> ${TMPDIR}/site.exp: $(TOP_DIR)/default.cfg
>      @{ echo 'lappend boards_dir "."'; \
>         echo "set target_alias ${DG_TARGET}"; }>  $@
>
> ${TMPDIR}/board.exp: $(TOP_DIR)/default.cfg
>      @{ echo -e "load_generic_config \"unix\""; \
>         echo -e "process_multilib_options \"\"" ; \
>         echo "set_board_info bmk,use_alarm 1" ; \
>         echo "set_board_info rsh_prog ssh" ; \
>         echo "set_board_info rcp_prog scp" ; \
>         echo "set_board_info hostname ${DG_TARGET_HOSTNAME}"; \
>         echo "set_board_info username ${DG_TARGET_USERNAME}"; }>  $@
>
> Also, they depend on default.cfg because they use values from there.
>
>    
>> +test: gcc-${DG_GCC_VERSION} config
>>      
> test: ${TMPDIR}/site.exp ${TMPDIR}/board.exp
>
>    
>> +	cd ${TMPDIR}&&  \
>> +	runtest	--tool ${DG_TOOLNAME} \
>>      
> Don't 'cd $(TMP_DIR)', but use runtest's --outdir and --objdir
>    
I did actually use objdir at first but I had some issues with some 
temporary objects/binaries being placed wrongly so I did it the old 
fashioned way which seemed to work better. Requires more testing.

>    
>> +		--srcdir ${DG_SRC_DIR} \
>> +		--all \
>> +		--target ${DG_TARGET} \
>> +		--target_board board \
>> +		${DG_TESTS} \
>> +		GXX_UNDER_TEST=${DG_TARGET}-g++ ; \
>> +	mv ${TMPDIR}/*.log ${TOPDIR} ; \
>> +	mv ${TMPDIR}/*.sum ${TOPDIR}
>>      
> Do not use continuation lines between runtest and the 'mv', as make would
> not detect a failure in runtest, and would still try to move the files.
> Better let make handle that.
>
>    
The thing is that when you run 50k+ test cases the odds are that at 
least one will fail and thus runtest basically always return an error 
despite the fact that the test session has executed successfully. So I 
simply overrided this with the mv operation so that the subsequent 
commands can continue. Yes - its a hack :)


> And, why move the result files at all? Why not just do:
>      @echo "Result files available in" '$(TMP_DIR)'"
>
>    
I thought maybe for most it would be more user friendly to put the 
results at top level haha. Regardless of placement your friendly echo 
string is definitely required.

>> +clean:
>> +	rm -rf gcc-testsuite-${DG_GCC_VERSION}.tar.gz gcc-${DG_GCC_VERSION} ${TMPDIR} *.log *.sum
>>      
> Tarball and directory now not handled.
>    
Related to download/extract feature - can be deleted.

>    
>> diff -r 48e107b35ba9 -r 7e1196581995 contrib/gcc-test-suite/README
>> --- /dev/null	Thu Jan 01 00:00:00 1970 +0000
>> +++ b/contrib/gcc-test-suite/README	Sat May 15 16:45:35 2010 +0200
>> @@ -0,0 +1,71 @@
>> +
>> +Helper Makefile for testing gcc toolchains using the gcc-testsuite
>> +==================================================================
>> +
>> +Requirements
>> +------------
>> +
>> +* DejaGnu 'runtest' v1.4.4+
>> +* Make v3.81+
>> +* wget
>>      
> wget not needed, we do no download while running the test-suite.
>
>    
Related to download/extract feature - can be deleted.
>> +Configuration
>> +-------------
>> +
>> +Edit default.cfg to reflect your toolchain and target configuration.
>> +
>> +Alternatively, override configuration variables on the command line.
>> +
>> +Available config variables:
>>      
> I know most of those sound fairly obvious, but some explanations would
> be most welcome. For example, it's not obvious where DG_SRC_DIR should
> point to ( yes, reading the code makes it quite obvious, but as this
> file is documenting the test-suite, let it be self-sufficient! ;-) )
>
>    

I know - sorry about that. Somehow doing docs are always last on the 
todo list ;)

>> +DG_GCC_VERSION
>> +DG_GCC_URL
>>      
> Those two useless, we do no download while running the test-suite.
>    
Related to download/extract feature - can be deleted.
>    
>> +DG_TOOLNAME
>> +DG_TARGET
>>      
> As I see it, DG_TARGET is the tuple, right? In that case, it should
> be a constant, and the user should not have to define it a run-time,
> As that would be error-prone (note that we currently can not isntall
> two or more toolchains in the same prefix).
>
> Plus, you are already setting it in the build script.
>    
Yes. Its a flexibility not required for ct-ng users and it would be 
constant. Can be kept internal to the makefile.
>    
>> +DG_TARGET_HOSTNAME
>> +DG_TARGET_USERNAME
>> +DG_C_TESTS
>> +DG_CPP_TESTS
>> +DG_TOOLCHAIN_DIR
>> +DG_SRC_DIR
>>      
> DG_SRC_DIR could/should be automagically detected, no?
>    
Its pretty much constant in context of how gcc test suite is currently 
copied around in the build script and the constant gcc dir scheme. This 
could also be kept internal to the Makefile.
> [--SNIP--]
>    
>> +SSH automatic login configuration example
>> +-----------------------------------------
>> +
>> +On host do:
>> +ssh-keygen -t rsa (then simply press enter thru all steps)
>> +scp ~/.ssh/id_rsa.pub<username>@<target IP>:~/
>> +
>> +On target do:
>> +cd ~
>> +mkdir .ssh
>> +cat id_rsa.pub>>  .ssh/authorized_keys
>> +rm id_rsa.pub
>>      
> As already mentionned, use ssh-copy-id.
>    
Absolutely - its just a old habit from my side that I still do it this 
way. Old habits die hard they say  :)
>    
>> +Author
>> +------
>> +Martin Lund<mgl@doredevelopment.dk>
>>      
> Maybe also tell the user he/she can get rid of the test-suite afterwards,
> by deleting the test-suite/ sub-directory.
>    
Yes, we could also add this comment to the Kconfig description to 
emphasize it.
>    
>> diff -r 48e107b35ba9 -r 7e1196581995 scripts/build/test_suite.sh
>> --- /dev/null	Thu Jan 01 00:00:00 1970 +0000
>> +++ b/scripts/build/test_suite.sh	Sat May 15 16:45:35 2010 +0200
>> @@ -0,0 +1,41 @@
>> +# Wrapper to build the test suite facilities
>> +#
>> +# Current assumption: test suites are independent of each other
>> +#                     - no order handling required.
>>      
> That's perfectly fine with me. I don't expect dependencies between the
> test-suites at build-time, nor at run-time.
>
>    
>> +# List all test suite facilities, and parse their scripts
>> +CT_TEST_SUITE_FACILITY_LIST=
>> +for f in "${CT_LIB_DIR}/scripts/build/test_suite/"*.sh; do
>> +    _f="$(basename "${f}" .sh)"
>> +    __f="CT_TEST_SUITE_${_f}"
>> +    __f=`echo ${__f} | tr "[:lower:]" "[:upper:]"`
>> +    if [ "${!__f}" = "y" ]; then
>> +        CT_DoLog DEBUG "Enabling test suite '${_f}'"
>> +        . "${f}"
>> +        CT_TEST_SUITE_FACILITY_LIST="${CT_TEST_SUITE_FACILITY_LIST} ${_f}"
>> +    else
>> +        CT_DoLog DEBUG "Disabling test suite '${_f}'"
>> +    fi
>> +done
>> +
>> +# Download the test suite facilities
>> +do_test_suite_get() {
>> +    for f in ${CT_TEST_SUITE_FACILITY_LIST}; do
>> +        do_test_suite_${f}_get
>> +    done
>> +}
>> +
>> +# Extract and patch the test suite facilities
>> +do_test_suite_extract() {
>> +    for f in ${CT_TEST_SUITE_FACILITY_LIST}; do
>> +        do_test_suite_${f}_extract
>> +    done
>> +}
>> +
>> +# Build the test suite facilities
>> +do_test_suite() {
>> +    for f in ${CT_TEST_SUITE_FACILITY_LIST}; do
>> +        do_test_suite_${f}_build
>> +    done
>> +}
>> +
>> diff -r 48e107b35ba9 -r 7e1196581995 scripts/build/test_suite/gcc.sh
>> --- /dev/null	Thu Jan 01 00:00:00 1970 +0000
>> +++ b/scripts/build/test_suite/gcc.sh	Sat May 15 16:45:35 2010 +0200
>> @@ -0,0 +1,36 @@
>> +# This file adds the functions to build the GCC test suite
>> +# Copyright 2010 DoréDevelopment
>> +# Created by Martin Lund<mgl@doredevelopment.dk>
>> +# Licensed under the GPL v2. See COPYING in the root of this package
>> +
>> +do_test_suite_gcc_get() { :; }
>> +do_test_suite_gcc_extract() { :; }
>> +do_test_suite_gcc_build() { :; }
>> +
>> +# Overide functions depending on configuration
>> +if [ "${CT_TEST_SUITE_GCC}" = "y" ]; then
>> +
>> +do_test_suite_gcc_build() {
>> +
>> +    CT_DoStep INFO "Installing GCC test suite"
>> +
>> +    CT_DoExecLog ALL mkdir -p "${CT_TEST_SUITE_DIR}/gcc-test-suite/gcc-${CT_CC_VERSION}/gcc"
>> +    CT_DoExecLog ALL cp "${CT_TOP_DIR}/contrib/gcc-test-suite/Makefile" \
>> +                        "${CT_TEST_SUITE_DIR}/gcc-test-suite"
>>      
> s/CT_TOP_DIR/CT_LIB_DIR/
>
>    
>> +    CT_DoExecLog ALL cp "${CT_TOP_DIR}/contrib/gcc-test-suite/default.cfg" \
>> +                        "${CT_TEST_SUITE_DIR}/gcc-test-suite"
>>      
> s/CT_TOP_DIR/CT_LIB_DIR/
>
>    
>> +    CT_DoExecLog ALL cp "${CT_TOP_DIR}/contrib/gcc-test-suite/README" \
>>      
> s/CT_TOP_DIR/CT_LIB_DIR/
>
>    
>> +                        "${CT_TEST_SUITE_DIR}/gcc-test-suite"
>> +    CT_DoExecLog ALL cp -r "${CT_SRC_DIR}/gcc-${CT_CC_VERSION}/gcc/testsuite" \
>> +                           "${CT_TEST_SUITE_DIR}/gcc-test-suite/gcc-${CT_CC_VERSION}/gcc"
>> +    sed "s/DG_GCC_VERSION .*/DG_GCC_VERSION = ${CT_CC_VERSION}/g" \
>> +        ${CT_TEST_SUITE_DIR}/gcc-test-suite/default.cfg>  \
>> +        ${CT_TEST_SUITE_DIR}/gcc-test-suite/default.cfg.tmp
>>      
> DG_GCC_VERSION useless.
>    
Related to download/extract feature - can be deleted.
>    
>> +    sed "s/DG_TARGET .*/DG_TARGET = ${CT_TARGET}/g" \
>> +        ${CT_TEST_SUITE_DIR}/gcc-test-suite/default.cfg.tmp>  \
>> +        ${CT_TEST_SUITE_DIR}/gcc-test-suite/default.cfg
>> +    CT_DoExecLog ALL rm -f "${CT_TEST_SUITE_DIR}/gcc-test-suite/default.cfg.tmp"
>> +    CT_EndStep
>> +}
>> +
>> +fi # CT_TEST_SUITE_GCC
>> diff -r 48e107b35ba9 -r 7e1196581995 scripts/crosstool-NG.sh.in
>> --- a/scripts/crosstool-NG.sh.in	Fri Apr 30 22:25:45 2010 +0200
>> +++ b/scripts/crosstool-NG.sh.in	Sat May 15 16:45:35 2010 +0200
>> @@ -125,6 +125,7 @@
>>   . "${CT_LIB_DIR}/scripts/build/libc/${CT_LIBC}.sh"
>>   . "${CT_LIB_DIR}/scripts/build/cc/${CT_CC}.sh"
>>   . "${CT_LIB_DIR}/scripts/build/debug.sh"
>> +. "${CT_LIB_DIR}/scripts/build/test_suite.sh"
>>
>>   # Target tuple: CT_TARGET needs a little love:
>>   CT_DoBuildTargetTuple
>> @@ -159,6 +160,9 @@
>>       CT_COMPLIBS_DIR="${CT_BUILD_DIR}/static"
>>   fi
>>
>> +# Compute test suite install directory
>> +CT_TEST_SUITE_DIR=${CT_INSTALL_DIR}/test-suite
>> +
>>   # Note: we'll always install the core compiler in its own directory, so as to
>>   # not mix the two builds: core and final.
>>   CT_CC_CORE_STATIC_PREFIX_DIR="${CT_BUILD_DIR}/${CT_CC}-core-static"
>> @@ -519,6 +523,7 @@
>>           do_cc_get
>>           do_libc_get
>>           do_debug_get
>> +        do_test_suite_get
>>           CT_EndStep
>>       fi
>>
>> @@ -549,6 +554,7 @@
>>           do_cc_extract
>>           do_libc_extract
>>           do_debug_extract
>> +        do_test_suite_extract
>>           CT_EndStep
>>       fi
>>   fi
>> @@ -597,5 +603,6 @@
>>
>>   [ "${CT_LOG_FILE_COMPRESS}" = y ]&&  bzip2 -9 "${CT_LOG_FILE}"
>>   [ "${CT_INSTALL_DIR_RO}" = "y"  ]&&  chmod -R a-w "${CT_INSTALL_DIR}"
>> +[ "${CT_TEST_SUITE}" = "y" ]&&  chmod -R a+w "${CT_TEST_SUITE_DIR}"
>>
>>   trap - EXIT
>> diff -r 48e107b35ba9 -r 7e1196581995 steps.mk
>> --- a/steps.mk	Fri Apr 30 22:25:45 2010 +0200
>> +++ b/steps.mk	Sat May 15 16:45:35 2010 +0200
>> @@ -39,6 +39,7 @@
>>               libelf_target       \
>>               binutils_target     \
>>               debug               \
>> +            test_suite          \
>>               finish              \
>>
>>   # Make the list available to sub-processes (scripts/crosstool-NG.sh needs it)
>>      
> Do you want to handle these issues, or do you want me to munge your patch?
>
> Regards,
> Yann E. MORIN.
>
>    
It would be nice to get the patch in now so please munge it as you see 
fit. I will let you decide whether to keep the stuff related to the 
download/extract feature or not. Whatever you leave behind of the above 
listed changes I will follow up on and fix when I create the README updates.

Now, having this test suite integrated and up and running the next step 
is to figure out how to utilize it :)

In the future Im thinking that we should aim high and try and run this 
test suite at each release but only for eg. 4 samples representing the 
major architectures such as ARM, PowerPC, x86, MIPS. I have HW targets 
for all of these architectures expect the MIPS one so I can help here. I 
think that the results could be fitted quite nicely in a seperate column 
in the front page toolchain matrix.

What do you think?

Best regards, Martin


--
For unsubscribe information see http://sourceware.org/lists.html#faq



More information about the crossgcc mailing list