This is the mail archive of the crossgcc@sourceware.org mailing list for the crossgcc project.

See the CrossGCC FAQ for lots more information.


Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]
Other format: [Raw text]

Re: [PATCH 1 of 1] test-suite: Added new test suite feature (experimental)


Hello Yann, all,

Please, feel free to munge the patch and change anything you want to get it in :)

Actually, the patch feature in its current form has been stalled in my drawer for quite a while because the last time I was ready to integrate it the state of the HG repo was so that I could not build and test any toolchains. So I decided to wait until it was in a better testable state. Only recently I opened that drawer again, a bit later than expected - sorry :)

I basically agree on all your proposed changes - please see comments below.

On 2010-05-16 17:08, Yann E. MORIN wrote:
Martin, All,

I don't know where the problem lies, but I was not able to easily apply
your base64-encoded mail. I really hope it lies on _my_ side (or it
would rendre Hg useless to exchange non-ASCII patches...)

Let me know if it happens again - maybe it is something that needs to be fixed on my side.
On Saturday 15 May 2010 17:19:05 Martin Lund wrote:
# HG changeset patch
# User Martin Lund<mgl@doredevelopment.dk>
# Date 1273934735 -7200
# Node ID 7e1196581995247e24c585653be9e65e8555d0ea
# Parent  48e107b35ba9c2b3f31667dd5042cc69bc472592
test-suite: Added new test suite feature (experimental)

This patch adds support for installing the gcc test suite. A helper
Makefile is provided for building and running the gcc tests.

The default configuration runs all gcc tests and requires automatic
ssh/scp login access to a networked target board. See README for
more details.

Note: Current feature is tested with the powerpc-unknown-linux-gnu
sample but it should work with others as well.
Missing Signed-off-by.
Hg tricked me :)
[--SNIP--]
diff -r 48e107b35ba9 -r 7e1196581995 contrib/gcc-test-suite/Makefile
[--SNIP--]
+# Targets
+all: test
+
+gcc-testsuite-${DG_GCC_VERSION}.tar.gz:
+# wget -nc ${DG_GCC_URL}
+
+gcc-${DG_GCC_VERSION}: gcc-testsuite-${DG_GCC_VERSION}.tar.gz
+# tar xzf gcc-testsuite-${DG_GCC_VERSION}.tar.gz
Those two targets are useless, we already have the test-suite isntalled.

The only reason these targets are left hanging there is that the makefile started out as a self contained thing downloading and extracting the gcc test suite by itself.


Actually, if we keep the Makefile able to download and extract any specificed gcc test suite version (by redefining DG_GCC_VERSION etc.) that would provide users/developers the flexibility to test a gcc toolchain with a different version gcc test suite. This is useful if you want to test a new testcase from a newer version of the gcc test suite with an older version toolchain. I recognize that in most situations this is not needed - so I have no problems if we let go of everything related to this feature. It can be re-introduced it if the need presents itself.

+config:
+ @mkdir -p ${TMPDIR}
+ @{ echo 'lappend boards_dir "."'; \
+ echo "set target_alias ${DG_TARGET}"; }> ${TMPDIR}/site.exp
+ @{ echo -e "load_generic_config \"unix\""; \
+ echo -e "process_multilib_options \"\"" ; \
+ echo "set_board_info bmk,use_alarm 1" ; \
+ echo "set_board_info rsh_prog ssh" ; \
+ echo "set_board_info rcp_prog scp" ; \
+ echo "set_board_info hostname ${DG_TARGET_HOSTNAME}"; \
+ echo "set_board_info username ${DG_TARGET_USERNAME}"; }> ${TMPDIR}/board.exp
Split the two rules:

${TMPDIR}/site.exp: $(TOP_DIR)/default.cfg
     @{ echo 'lappend boards_dir "."'; \
        echo "set target_alias ${DG_TARGET}"; }>  $@

${TMPDIR}/board.exp: $(TOP_DIR)/default.cfg
     @{ echo -e "load_generic_config \"unix\""; \
        echo -e "process_multilib_options \"\"" ; \
        echo "set_board_info bmk,use_alarm 1" ; \
        echo "set_board_info rsh_prog ssh" ; \
        echo "set_board_info rcp_prog scp" ; \
        echo "set_board_info hostname ${DG_TARGET_HOSTNAME}"; \
        echo "set_board_info username ${DG_TARGET_USERNAME}"; }>  $@

Also, they depend on default.cfg because they use values from there.

+test: gcc-${DG_GCC_VERSION} config
test: ${TMPDIR}/site.exp ${TMPDIR}/board.exp

+ cd ${TMPDIR}&& \
+ runtest --tool ${DG_TOOLNAME} \
Don't 'cd $(TMP_DIR)', but use runtest's --outdir and --objdir
I did actually use objdir at first but I had some issues with some temporary objects/binaries being placed wrongly so I did it the old fashioned way which seemed to work better. Requires more testing.

+ --srcdir ${DG_SRC_DIR} \
+ --all \
+ --target ${DG_TARGET} \
+ --target_board board \
+ ${DG_TESTS} \
+ GXX_UNDER_TEST=${DG_TARGET}-g++ ; \
+ mv ${TMPDIR}/*.log ${TOPDIR} ; \
+ mv ${TMPDIR}/*.sum ${TOPDIR}
Do not use continuation lines between runtest and the 'mv', as make would
not detect a failure in runtest, and would still try to move the files.
Better let make handle that.

The thing is that when you run 50k+ test cases the odds are that at least one will fail and thus runtest basically always return an error despite the fact that the test session has executed successfully. So I simply overrided this with the mv operation so that the subsequent commands can continue. Yes - its a hack :)


And, why move the result files at all? Why not just do:
     @echo "Result files available in" '$(TMP_DIR)'"

I thought maybe for most it would be more user friendly to put the results at top level haha. Regardless of placement your friendly echo string is definitely required.

+clean:
+ rm -rf gcc-testsuite-${DG_GCC_VERSION}.tar.gz gcc-${DG_GCC_VERSION} ${TMPDIR} *.log *.sum
Tarball and directory now not handled.
Related to download/extract feature - can be deleted.

diff -r 48e107b35ba9 -r 7e1196581995 contrib/gcc-test-suite/README
--- /dev/null Thu Jan 01 00:00:00 1970 +0000
+++ b/contrib/gcc-test-suite/README Sat May 15 16:45:35 2010 +0200
@@ -0,0 +1,71 @@
+
+Helper Makefile for testing gcc toolchains using the gcc-testsuite
+==================================================================
+
+Requirements
+------------
+
+* DejaGnu 'runtest' v1.4.4+
+* Make v3.81+
+* wget
wget not needed, we do no download while running the test-suite.

Related to download/extract feature - can be deleted.
+Configuration
+-------------
+
+Edit default.cfg to reflect your toolchain and target configuration.
+
+Alternatively, override configuration variables on the command line.
+
+Available config variables:
I know most of those sound fairly obvious, but some explanations would
be most welcome. For example, it's not obvious where DG_SRC_DIR should
point to ( yes, reading the code makes it quite obvious, but as this
file is documenting the test-suite, let it be self-sufficient! ;-) )


I know - sorry about that. Somehow doing docs are always last on the todo list ;)


+DG_GCC_VERSION
+DG_GCC_URL
Those two useless, we do no download while running the test-suite.
Related to download/extract feature - can be deleted.
+DG_TOOLNAME
+DG_TARGET
As I see it, DG_TARGET is the tuple, right? In that case, it should
be a constant, and the user should not have to define it a run-time,
As that would be error-prone (note that we currently can not isntall
two or more toolchains in the same prefix).

Plus, you are already setting it in the build script.
Yes. Its a flexibility not required for ct-ng users and it would be constant. Can be kept internal to the makefile.
+DG_TARGET_HOSTNAME
+DG_TARGET_USERNAME
+DG_C_TESTS
+DG_CPP_TESTS
+DG_TOOLCHAIN_DIR
+DG_SRC_DIR
DG_SRC_DIR could/should be automagically detected, no?
Its pretty much constant in context of how gcc test suite is currently copied around in the build script and the constant gcc dir scheme. This could also be kept internal to the Makefile.
[--SNIP--]
+SSH automatic login configuration example
+-----------------------------------------
+
+On host do:
+ssh-keygen -t rsa (then simply press enter thru all steps)
+scp ~/.ssh/id_rsa.pub<username>@<target IP>:~/
+
+On target do:
+cd ~
+mkdir .ssh
+cat id_rsa.pub>> .ssh/authorized_keys
+rm id_rsa.pub
As already mentionned, use ssh-copy-id.
Absolutely - its just a old habit from my side that I still do it this way. Old habits die hard they say :)
+Author
+------
+Martin Lund<mgl@doredevelopment.dk>
Maybe also tell the user he/she can get rid of the test-suite afterwards,
by deleting the test-suite/ sub-directory.
Yes, we could also add this comment to the Kconfig description to emphasize it.
diff -r 48e107b35ba9 -r 7e1196581995 scripts/build/test_suite.sh
--- /dev/null Thu Jan 01 00:00:00 1970 +0000
+++ b/scripts/build/test_suite.sh Sat May 15 16:45:35 2010 +0200
@@ -0,0 +1,41 @@
+# Wrapper to build the test suite facilities
+#
+# Current assumption: test suites are independent of each other
+# - no order handling required.
That's perfectly fine with me. I don't expect dependencies between the
test-suites at build-time, nor at run-time.

+# List all test suite facilities, and parse their scripts
+CT_TEST_SUITE_FACILITY_LIST=
+for f in "${CT_LIB_DIR}/scripts/build/test_suite/"*.sh; do
+ _f="$(basename "${f}" .sh)"
+ __f="CT_TEST_SUITE_${_f}"
+ __f=`echo ${__f} | tr "[:lower:]" "[:upper:]"`
+ if [ "${!__f}" = "y" ]; then
+ CT_DoLog DEBUG "Enabling test suite '${_f}'"
+ . "${f}"
+ CT_TEST_SUITE_FACILITY_LIST="${CT_TEST_SUITE_FACILITY_LIST} ${_f}"
+ else
+ CT_DoLog DEBUG "Disabling test suite '${_f}'"
+ fi
+done
+
+# Download the test suite facilities
+do_test_suite_get() {
+ for f in ${CT_TEST_SUITE_FACILITY_LIST}; do
+ do_test_suite_${f}_get
+ done
+}
+
+# Extract and patch the test suite facilities
+do_test_suite_extract() {
+ for f in ${CT_TEST_SUITE_FACILITY_LIST}; do
+ do_test_suite_${f}_extract
+ done
+}
+
+# Build the test suite facilities
+do_test_suite() {
+ for f in ${CT_TEST_SUITE_FACILITY_LIST}; do
+ do_test_suite_${f}_build
+ done
+}
+
diff -r 48e107b35ba9 -r 7e1196581995 scripts/build/test_suite/gcc.sh
--- /dev/null Thu Jan 01 00:00:00 1970 +0000
+++ b/scripts/build/test_suite/gcc.sh Sat May 15 16:45:35 2010 +0200
@@ -0,0 +1,36 @@
+# This file adds the functions to build the GCC test suite
+# Copyright 2010 DorÃDevelopment
+# Created by Martin Lund<mgl@doredevelopment.dk>
+# Licensed under the GPL v2. See COPYING in the root of this package
+
+do_test_suite_gcc_get() { :; }
+do_test_suite_gcc_extract() { :; }
+do_test_suite_gcc_build() { :; }
+
+# Overide functions depending on configuration
+if [ "${CT_TEST_SUITE_GCC}" = "y" ]; then
+
+do_test_suite_gcc_build() {
+
+ CT_DoStep INFO "Installing GCC test suite"
+
+ CT_DoExecLog ALL mkdir -p "${CT_TEST_SUITE_DIR}/gcc-test-suite/gcc-${CT_CC_VERSION}/gcc"
+ CT_DoExecLog ALL cp "${CT_TOP_DIR}/contrib/gcc-test-suite/Makefile" \
+ "${CT_TEST_SUITE_DIR}/gcc-test-suite"
s/CT_TOP_DIR/CT_LIB_DIR/

+ CT_DoExecLog ALL cp "${CT_TOP_DIR}/contrib/gcc-test-suite/default.cfg" \
+ "${CT_TEST_SUITE_DIR}/gcc-test-suite"
s/CT_TOP_DIR/CT_LIB_DIR/

+ CT_DoExecLog ALL cp "${CT_TOP_DIR}/contrib/gcc-test-suite/README" \
s/CT_TOP_DIR/CT_LIB_DIR/

+ "${CT_TEST_SUITE_DIR}/gcc-test-suite"
+ CT_DoExecLog ALL cp -r "${CT_SRC_DIR}/gcc-${CT_CC_VERSION}/gcc/testsuite" \
+ "${CT_TEST_SUITE_DIR}/gcc-test-suite/gcc-${CT_CC_VERSION}/gcc"
+ sed "s/DG_GCC_VERSION .*/DG_GCC_VERSION = ${CT_CC_VERSION}/g" \
+ ${CT_TEST_SUITE_DIR}/gcc-test-suite/default.cfg> \
+ ${CT_TEST_SUITE_DIR}/gcc-test-suite/default.cfg.tmp
DG_GCC_VERSION useless.
Related to download/extract feature - can be deleted.
+    sed "s/DG_TARGET .*/DG_TARGET = ${CT_TARGET}/g" \
+        ${CT_TEST_SUITE_DIR}/gcc-test-suite/default.cfg.tmp>  \
+        ${CT_TEST_SUITE_DIR}/gcc-test-suite/default.cfg
+    CT_DoExecLog ALL rm -f "${CT_TEST_SUITE_DIR}/gcc-test-suite/default.cfg.tmp"
+    CT_EndStep
+}
+
+fi # CT_TEST_SUITE_GCC
diff -r 48e107b35ba9 -r 7e1196581995 scripts/crosstool-NG.sh.in
--- a/scripts/crosstool-NG.sh.in	Fri Apr 30 22:25:45 2010 +0200
+++ b/scripts/crosstool-NG.sh.in	Sat May 15 16:45:35 2010 +0200
@@ -125,6 +125,7 @@
  . "${CT_LIB_DIR}/scripts/build/libc/${CT_LIBC}.sh"
  . "${CT_LIB_DIR}/scripts/build/cc/${CT_CC}.sh"
  . "${CT_LIB_DIR}/scripts/build/debug.sh"
+. "${CT_LIB_DIR}/scripts/build/test_suite.sh"

  # Target tuple: CT_TARGET needs a little love:
  CT_DoBuildTargetTuple
@@ -159,6 +160,9 @@
      CT_COMPLIBS_DIR="${CT_BUILD_DIR}/static"
  fi

+# Compute test suite install directory
+CT_TEST_SUITE_DIR=${CT_INSTALL_DIR}/test-suite
+
  # Note: we'll always install the core compiler in its own directory, so as to
  # not mix the two builds: core and final.
  CT_CC_CORE_STATIC_PREFIX_DIR="${CT_BUILD_DIR}/${CT_CC}-core-static"
@@ -519,6 +523,7 @@
          do_cc_get
          do_libc_get
          do_debug_get
+        do_test_suite_get
          CT_EndStep
      fi

@@ -549,6 +554,7 @@
          do_cc_extract
          do_libc_extract
          do_debug_extract
+        do_test_suite_extract
          CT_EndStep
      fi
  fi
@@ -597,5 +603,6 @@

  [ "${CT_LOG_FILE_COMPRESS}" = y ]&&  bzip2 -9 "${CT_LOG_FILE}"
  [ "${CT_INSTALL_DIR_RO}" = "y"  ]&&  chmod -R a-w "${CT_INSTALL_DIR}"
+[ "${CT_TEST_SUITE}" = "y" ]&&  chmod -R a+w "${CT_TEST_SUITE_DIR}"

  trap - EXIT
diff -r 48e107b35ba9 -r 7e1196581995 steps.mk
--- a/steps.mk	Fri Apr 30 22:25:45 2010 +0200
+++ b/steps.mk	Sat May 15 16:45:35 2010 +0200
@@ -39,6 +39,7 @@
              libelf_target       \
              binutils_target     \
              debug               \
+            test_suite          \
              finish              \

# Make the list available to sub-processes (scripts/crosstool-NG.sh needs it)
Do you want to handle these issues, or do you want me to munge your patch?

Regards,
Yann E. MORIN.

It would be nice to get the patch in now so please munge it as you see fit. I will let you decide whether to keep the stuff related to the download/extract feature or not. Whatever you leave behind of the above listed changes I will follow up on and fix when I create the README updates.

Now, having this test suite integrated and up and running the next step is to figure out how to utilize it :)

In the future Im thinking that we should aim high and try and run this test suite at each release but only for eg. 4 samples representing the major architectures such as ARM, PowerPC, x86, MIPS. I have HW targets for all of these architectures expect the MIPS one so I can help here. I think that the results could be fitted quite nicely in a seperate column in the front page toolchain matrix.

What do you think?

Best regards, Martin


-- For unsubscribe information see http://sourceware.org/lists.html#faq


Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]