This is the mail archive of the
libc-alpha@sourceware.org
mailing list for the glibc project.
Re: [PATCH v4] Add helper script for glibc debugging
- From: Carlos O'Donell <carlos at redhat dot com>
- To: "Gabriel F. T. Gomes" <gabriel at inconstante dot net dot br>, libc-alpha at sourceware dot org
- Cc: arjun dot is at lostca dot se, ashankar at redhat dot com
- Date: Fri, 27 Sep 2019 21:53:02 -0400
- Subject: Re: [PATCH v4] Add helper script for glibc debugging
- References: <20190927221737.547-1-gabriel@inconstante.net.br>
On 9/27/19 6:17 PM, Gabriel F. T. Gomes wrote:
> From: "Gabriel F. T. Gomes" <gabrielftg@linux.ibm.com>
>
v3 is OK for master.
Thank you very much for this work!
Would you be willing to also update:
https://sourceware.org/glibc/wiki/Testing/Builds
We should suggest the new option.
Reviewed-by: Carlos O'Donell <carlos@redhat.com>
> Changes since v3:
>
> - Produce a more verbose summary;
> - Updated obsolete comment;
> - Added an extra reference to the commit message.
>
> Changes since v2:
>
> - Call gdb directly, instead of loading it with ld.so;
> - Use 'set auto-load safe-path' and 'set libthread-db-search-path',
> to let gdb know of libthread_db.so location;
> - Let environment variables be set from the command line (-e option).
>
> Changes since v1:
>
> - Added nptl_db to the argument to --library-path, which allows
> debugging with multiple threads (see additional comment below), as
> observed by Carlos;
> - Removed the hardcoded path to GDB, as suggested by Dmitry;
> - Removed the default test case and source directory paths, as
> suggested by Joseph, which removed some of the clutter in the
> script;
>
> Note about multi-threaded test case debugging:
>
> As Carlos correctly noticed, the first version of the script lacked the
> required GDB setup for multi-threaded debugging. This could be noticed
> by running a multi-threaded test case, such as nptl/tst-exec1, and
> observing the warning message in the output:
>
> warning: Unable to find libthread_db matching inferior's thread
> library, thread debugging will not be available.
>
> In this second version, which explicitly adds nptl_db to the argument of
> --library-path, the warning message is gone, and converted into the
> following info message:
>
> [Thread debugging using libthread_db enabled]
> Using host libthread_db library
> "/home/gabriel/build/x86_64/glibc//nptl_db/libthread_db.so.1".
>
> Not-for-commit comment from v1:
>
> At Cauldron, while Florian showed his development workflow to us, we had
> a brief discussion about debugging glibc with GDB. After that, I spent
> some time with Arjun and we shared our debugging scripts and tricks,
> then we came to the conclusion that this could be helpful to other
> people, maybe even more so to newcomers. So, I got my off-tree script
> and converted it into this auto-generated version.
>
> Still at Cauldron, Carlos mentioned that it wasn't trivial to let GDB
> know about the symbols within glibc. This script doesn't solve this for
> every glibc debugging needs (I suppose it doesn't help at all when
> debugging the initial steps of program loading, for example), but it
> makes it a little easier when debugging test cases, as it automatically
> loads the symbols from them.
>
> Maybe there are more trivial ways to do it, but that's what I have been
> using for some time.
>
> -- 8< --
> This patch adds a new make rule that generates a helper script for
> debugging glibc test cases. The new script, debugglibc.sh, is similar
> to testrun.sh, in the sense that it allows the execution of the
> specified test case, however, it opens the test case in GDB, setting the
> library path the same way that testrun.sh does. The commands are based
> on the instructions on the wiki for glibc debugging [1,2].
>
> By default, the script tells GDB to load the test case for symbol
> information, so that, when a breakpoint is hit, the call stack is
> displayed correctly (instead of printing lots of '??'s). For instance,
> after running 'make' and 'make check', one could do the following:
>
> $ ./debugglibc.sh nptl/tst-exec1 -b pthread_join
>
> Reading symbols from /home/gabriel/build/powerpc64le/glibc//elf/ld.so...done.
> Breakpoint 1 at 0x1444
> add symbol table from file "nptl/tst-exec1"
> [Thread debugging using libthread_db enabled]
> Using host libthread_db library "/home/gabriel/build/powerpc64le/glibc//nptl_db/libthread_db.so.1".
>
> Breakpoint 1, 0x00007ffff7fb1444 in _dl_start_user () from /home/gabriel/build/powerpc64le/glibc/elf/ld.so
> Breakpoint 2 at 0x7ffff7f49d48: file pthread_join.c, line 23.
>
> Notice that the script will always start GDB with the program running
> and halted at _dl_start_user. So, in order to reach the actual
> breakpoint of interest, one should hit 'c', not 'r':
>
> >>> c
> Continuing.
> [New Thread 0x7ffff7d1f180 (LWP 76443)]
> [Switching to Thread 0x7ffff7d1f180 (LWP 76443)]
>
> Thread 2 "ld.so" hit Breakpoint 2, __pthread_join (threadid=140737354087616, thread_return=0x0) at pthread_join.c:24
> 24 return __pthread_timedjoin_ex (threadid, thread_return, NULL, true);
>
> Then inspect the call stack with 'bt', as usual, and see symbols from
> both the test case and from the libraries themselves:
>
> >>> bt
> #0 __pthread_join (threadid=140737354087616, thread_return=0x0) at pthread_join.c:24
> #1 0x0000000010001f4c in tf (arg=<optimized out>) at tst-exec1.c:37
> #2 0x00007ffff7f487e8 in start_thread (arg=0x7ffff7510000) at pthread_create.c:479
> #3 0x00007ffff7e523a8 in clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:82
>
> Tested for powerpc64le and x86_64.
>
> [1] https://sourceware.org/glibc/wiki/Debugging/Loader_Debugging
> [2] https://sourceware.org/glibc/wiki/Testing/Builds#Required_gdb_setup
> ---
> Makefile | 158 ++++++++++++++++++++++++++++++++++++++++++++++++++++++-
> 1 file changed, 157 insertions(+), 1 deletion(-)
>
> diff --git a/Makefile b/Makefile
> index 67ddd01bfe..a50fb64dc2 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -187,7 +187,163 @@ $(common-objpfx)testrun.sh: $(common-objpfx)config.make \
> mv -f $@T $@
> postclean-generated += testrun.sh
>
> -others: $(common-objpfx)testrun.sh
> +define debugglibc
OK.
> +#!/bin/bash
> +
> +SOURCE_DIR="$(CURDIR)"
> +BUILD_DIR="$(common-objpfx)"
> +CMD_FILE="$(common-objpfx)debugglibc.gdb"
> +DIRECT=true
> +SYMBOLSFILE=true
> +unset TESTCASE
> +unset BREAKPOINTS
> +unset ENVVARS
OK.
> +
> +usage()
> +{
> +cat << EOF
> +Usage: $$0 [OPTIONS] <testcase>
> +
> + where <testcase> is the path to the program being tested.
> +
> +Options:
> +
> + -h, --help
> + Prints this message and leaves.
> +
> + The following options require one argument:
> +
> + -b, --breakpoint
> + Breakpoints to set at the beginning of the execution
> + (each breakpoint demands its own -b option, e.g. -b foo -b bar)
> + -e, --environment-variable
> + Environment variables to be set with 'exec-wrapper env' in GDB
> + (each environment variable demands its own -e option, e.g.
> + -e FOO=foo -e BAR=bar)
> +
> + The following options do not take arguments:
> +
> + -i, --no-direct
> + Selects whether to pass the flag --direct to gdb.
> + Required for glibc test cases and not allowed for non-glibc tests.
> + Default behaviour is to pass the flag --direct to gdb.
> + -s, --no-symbols-file
> + Do not tell GDB to load debug symbols from the testcase.
> +EOF
> +}
OK.
> +
> +# Parse input options
> +while [[ $$# > 0 ]]
> +do
> + key="$$1"
> + case $$key in
> + -h|--help)
> + usage
> + exit 0
> + ;;
> + -b|--breakpoint)
> + BREAKPOINTS="break $$2\n$$BREAKPOINTS"
> + shift
> + ;;
> + -e|--environment-variable)
> + ENVVARS="$$2 $$ENVVARS"
> + shift
> + ;;
> + -i|--no-direct)
> + DIRECT=false
> + ;;
> + -s|--no-symbols-file)
> + SYMBOLSFILE=false
> + ;;
> + *)
> + TESTCASE=$$1
> + ;;
> + esac
> + shift
> +done
OK.
> +
> +# Check for required argument
> +if [ ! -v TESTCASE ]
> +then
> + usage
> + exit 1
> +fi
> +
> +# Expand environment setup command
> +if [ -v ENVVARS ]
> +then
> + ENVVARSCMD="set exec-wrapper env $$ENVVARS"
> +fi
> +
> +# Expand direct argument
> +if [ "$$DIRECT" == true ]
> +then
> + DIRECT="--direct"
> +else
> + DIRECT=""
> +fi
> +
> +# Expand symbols loading command
> +if [ "$$SYMBOLSFILE" == true ]
> +then
> + SYMBOLSFILE="add-symbol-file $${TESTCASE}"
> +else
> + SYMBOLSFILE=""
> +fi
> +
> +# GDB commands template
> +template ()
> +{
> +cat <<EOF
> +set environment C -E -x c-header
> +set auto-load safe-path $${BUILD_DIR}/nptl_db:\$$debugdir:\$$datadir/auto-load
> +set libthread-db-search-path $${BUILD_DIR}/nptl_db
OK.
> +__ENVVARS__
> +__SYMBOLSFILE__
> +break _dl_start_user
> +run --library-path $(rpath-link):$${BUILD_DIR}/nptl_db \
> +__TESTCASE__ __DIRECT__
> +__BREAKPOINTS__
> +EOF
OK.
> +}
> +
> +# Generate the commands file for gdb initialization
> +template | sed \
> + -e "s|__ENVVARS__|$$ENVVARSCMD|" \
> + -e "s|__SYMBOLSFILE__|$$SYMBOLSFILE|" \
> + -e "s|__TESTCASE__|$$TESTCASE|" \
> + -e "s|__DIRECT__|$$DIRECT|" \
> + -e "s|__BREAKPOINTS__|$$BREAKPOINTS|" \
> + > $$CMD_FILE
> +
> +echo
> +echo "Debugging glibc..."
> +echo "Build directory : $$BUILD_DIR"
> +echo "Source directory : $$SOURCE_DIR"
> +echo "GLIBC Testcase : $$TESTCASE"
> +echo "GDB Commands : $$CMD_FILE"
> +echo "Env vars : $$ENVVARS"
OK.
> +echo
> +
> +# Start the test case debugging in two steps:
> +# 1. the following command invokes gdb to run the loader;
> +# 2. the commands file tells the loader to run the test case.
OK.
> +gdb -q \
> + -x $${CMD_FILE} \
> + -d $${SOURCE_DIR} \
> + $${BUILD_DIR}/elf/ld.so
> +endef
OK.
> +
> +# This is another handy script for debugging dynamically linked program
> +# against the current libc build for testing.
> +$(common-objpfx)debugglibc.sh: $(common-objpfx)config.make \
> + $(..)Makeconfig $(..)Makefile
> + $(file >$@T,$(debugglibc))
> + chmod a+x $@T
> + mv -f $@T $@
> +postclean-generated += debugglibc.sh debugglibc.gdb
> +
> +others: $(common-objpfx)testrun.sh $(common-objpfx)debugglibc.sh
>
> # Makerules creates a file `stubs' in each subdirectory, which
> # contains `#define __stub_FUNCTION' for each function defined in that
>
--
Cheers,
Carlos.