[PATCH v3/pushed] linux: Add maintenance commands to test libthread_db

[Gary Benson <gbenson at redhat dot com>]

Eli Zaretskii wrote:
> Gary Benson wrote:
> > +library.  This exercises all libthread_db functionality used by GDB on
> 
> "libthread_db" should be in @code or in @file.  And please use
> @value{GDBN} instead of a literal "GDB".

Done.  FWIW I used @code for "libthread_db", it's what's used in
@ref{set libthread-db-search-path}.

> > +GNU/Linux systems, and by extension also exercises the proc_service
> 
> proc_service should be in @code.

Done.

Pedro Alves wrote:
> This looks good to me too, except one thing.  I think you end up
> with duplicated test messages:
[snip]
> I suggest wrapping the several phases of the testcase in
> with_test_prefix or proc_with_prefix.  Something like:
[snip]

Done.  I'd changed the obvious messages for v2 of the patch, Simon
mentioned something about with_test_prefix but at the time it didn't
seem worth wrapping things just for one or two messages so I did it
manually.  I hadn't realized there were messages emitted by the setup
commands that were also coming out duplicated (because the setup
commands were the same).  So I've used with_test_prefix this time,
and used the recipe you linked to check the output.

> OK with me with that change.

Cool, I pushed the updated patch (included below).

Have a good weekend everyone!

Cheers,
Gary

---
This commit adds two new commands which may be used to test thread
debugging libraries used by GDB:

  * "maint check libthread-db" tests the thread debugging library GDB
     is using for the current inferior.

  * "maint set/show check-libthread-db" selects whether libthread_db
     tests should be run automatically as libthread_db is auto-loaded.
     The default is to not run tests automatically.

The test itself is a basic integrity check exercising all libthread_db
functions used by GDB on GNU/Linux systems.  By extension this also
exercises the proc_service functions provided by GDB that libthread_db
uses.

This functionality is useful for NPTL developers and libthread_db
developers.  It could also prove useful investigating bugs reported
against GDB where the thread debugging library or GDB's proc_service
layer is suspect.

gdb/ChangeLog:

	* linux-thread-db.c (valprint.h): New include.
	(struct check_thread_db_info): New structure.
	(check_thread_db_on_load, tdb_testinfo): New static globals.
	(check_thread_db, check_thread_db_callback): New functions.
	(try_thread_db_load_1): Run integrity checks if requested.
	(maintenance_check_libthread_db): New function.
	(_initialize_thread_db): Register "maint check libthread-db"
	and "maint set/show check-libthread-db".
	* NEWS: Mention the above new commands.

gdb/doc/ChangeLog:

	* gdb.texinfo (Maintenance Commands): Document "maint check
	libthread-db" and "maint set/show check-libthread-db".

gdb/testsuite/ChangeLog:

	* gdb.threads/check-libthread-db.exp: New file.
	* gdb.threads/check-libthread-db.c: Likewise.
---
 gdb/ChangeLog                                    |  12 +
 gdb/NEWS                                         |  10 +
 gdb/doc/ChangeLog                                |   5 +
 gdb/doc/gdb.texinfo                              |  20 ++
 gdb/linux-thread-db.c                            | 291 +++++++++++++++++++++++
 gdb/testsuite/ChangeLog                          |   5 +
 gdb/testsuite/gdb.threads/check-libthread-db.c   |  67 ++++++
 gdb/testsuite/gdb.threads/check-libthread-db.exp | 114 +++++++++
 8 files changed, 524 insertions(+)
 create mode 100644 gdb/testsuite/gdb.threads/check-libthread-db.c
 create mode 100644 gdb/testsuite/gdb.threads/check-libthread-db.exp

diff --git a/gdb/NEWS b/gdb/NEWS
index 8fb6a2a..13da2f1 100644
--- a/gdb/NEWS
+++ b/gdb/NEWS
@@ -36,6 +36,16 @@ set|show record btrace cpu
   Controls the processor to be used for enabling errata workarounds for
   branch trace decode.
 
+maint check libthread-db
+  Run integrity checks on the current inferior's thread debugging
+  library
+
+maint set check-libthread-db (on|off)
+maint show check-libthread-db
+  Control whether to run integrity checks on inferior specific thread
+  debugging libraries as they are loaded.  The default is not to
+  perform such checks.
+
 * Python API
 
   ** Type alignment is now exposed via the "align" attribute of a gdb.Type.
diff --git a/gdb/doc/gdb.texinfo b/gdb/doc/gdb.texinfo
index 4968b37..2c0ac33 100644
--- a/gdb/doc/gdb.texinfo
+++ b/gdb/doc/gdb.texinfo
@@ -35547,6 +35547,15 @@ modify XML target descriptions.
 Check that the target descriptions dynamically created by @value{GDBN}
 equal the descriptions created from XML files found in @var{dir}.
 
+@kindex maint check libthread-db
+@item maint check libthread-db
+Run integrity checks on the current inferior's thread debugging
+library.  This exercises all @code{libthread_db} functionality used by
+@value{GDBN} on GNU/Linux systems, and by extension also exercises the
+@code{proc_service} functions provided by @value{GDBN} that
+@code{libthread_db} uses.  Note that parts of the test may be skipped
+on some platforms when debugging core files.
+
 @kindex maint print dummy-frames
 @item maint print dummy-frames
 Prints the contents of @value{GDBN}'s internal dummy-frame stack.
@@ -35854,6 +35863,17 @@ number of blocks in the blockvector
 @end enumerate
 @end table
 
+@kindex maint set check-libthread-db
+@kindex maint show check-libthread-db
+@item maint set check-libthread-db [on|off]
+@itemx maint show check-libthread-db
+Control whether @value{GDBN} should run integrity checks on inferior
+specific thread debugging libraries as they are loaded.  The default
+is not to perform such checks.  If any check fails @value{GDBN} will
+unload the library and continue searching for a suitable candidate as
+described in @ref{set libthread-db-search-path}.  For more information
+about the tests, see @ref{maint check libthread-db}.
+
 @kindex maint space
 @cindex memory used by commands
 @item maint space @var{value}
diff --git a/gdb/linux-thread-db.c b/gdb/linux-thread-db.c
index ccfd9e4..192e087 100644
--- a/gdb/linux-thread-db.c
+++ b/gdb/linux-thread-db.c
@@ -47,6 +47,7 @@
 #include "nat/linux-namespaces.h"
 #include <algorithm>
 #include "common/pathstuff.h"
+#include "valprint.h"
 
 /* GNU/Linux libthread_db support.
 
@@ -117,6 +118,10 @@ static char *libthread_db_search_path;
    by the "set auto-load libthread-db" command.  */
 static int auto_load_thread_db = 1;
 
+/* Set to non-zero if load-time libthread_db tests have been enabled
+   by the "maintenence set check-libthread-db" command.  */
+static int check_thread_db_on_load = 0;
+
 /* "show" command for the auto_load_thread_db configuration variable.  */
 
 static void
@@ -534,6 +539,250 @@ dladdr_to_soname (const void *addr)
   return NULL;
 }
 
+/* State for check_thread_db_callback.  */
+
+struct check_thread_db_info
+{
+  /* The libthread_db under test.  */
+  struct thread_db_info *info;
+
+  /* True if progress should be logged.  */
+  bool log_progress;
+
+  /* True if the callback was called.  */
+  bool threads_seen;
+
+  /* Name of last libthread_db function called.  */
+  const char *last_call;
+
+  /* Value returned by last libthread_db call.  */
+  td_err_e last_result;
+};
+
+static struct check_thread_db_info *tdb_testinfo;
+
+/* Callback for check_thread_db.  */
+
+static int
+check_thread_db_callback (const td_thrhandle_t *th, void *arg)
+{
+  gdb_assert (tdb_testinfo != NULL);
+  tdb_testinfo->threads_seen = true;
+
+#define LOG(fmt, args...)						\
+  do									\
+    {									\
+      if (tdb_testinfo->log_progress)					\
+	{								\
+	  debug_printf (fmt, ## args);					\
+	  gdb_flush (gdb_stdlog);					\
+	}								\
+    }									\
+  while (0)
+
+#define CHECK_1(expr, args...)						\
+  do									\
+    {									\
+      if (!(expr))							\
+	{								\
+	  LOG (" ... FAIL!\n");						\
+	  error (args);							\
+	}								\
+    }									\
+  while (0)
+
+#define CHECK(expr)							\
+  CHECK_1 (expr, "(%s) == false", #expr)
+
+#define CALL_UNCHECKED(func, args...)					\
+  do									\
+    {									\
+      tdb_testinfo->last_call = #func;					\
+      tdb_testinfo->last_result						\
+	= tdb_testinfo->info->func ## _p (args);			\
+    }									\
+  while (0)
+
+#define CHECK_CALL()							\
+  CHECK_1 (tdb_testinfo->last_result == TD_OK,				\
+	   _("%s failed: %s"),						\
+	   tdb_testinfo->last_call,					\
+	   thread_db_err_str (tdb_testinfo->last_result))		\
+
+#define CALL(func, args...)						\
+  do									\
+    {									\
+      CALL_UNCHECKED (func, args);					\
+      CHECK_CALL ();							\
+    }									\
+  while (0)
+
+  LOG ("  Got thread");
+
+  /* Check td_ta_thr_iter passed consistent arguments.  */
+  CHECK (th != NULL);
+  CHECK (arg == (void *) tdb_testinfo);
+  CHECK (th->th_ta_p == tdb_testinfo->info->thread_agent);
+
+  LOG (" %s", core_addr_to_string_nz ((CORE_ADDR) th->th_unique));
+
+  /* Check td_thr_get_info.  */
+  td_thrinfo_t ti;
+  CALL (td_thr_get_info, th, &ti);
+
+  LOG (" => %d", ti.ti_lid);
+
+  CHECK (ti.ti_ta_p == th->th_ta_p);
+  CHECK (ti.ti_tid == (thread_t) th->th_unique);
+
+  /* Check td_ta_map_lwp2thr.  */
+  td_thrhandle_t th2;
+  memset (&th2, 23, sizeof (td_thrhandle_t));
+  CALL_UNCHECKED (td_ta_map_lwp2thr, th->th_ta_p, ti.ti_lid, &th2);
+
+  if (tdb_testinfo->last_result == TD_ERR && !target_has_execution)
+    {
+      /* Some platforms require execution for td_ta_map_lwp2thr.  */
+      LOG (_("; can't map_lwp2thr"));
+    }
+  else
+    {
+      CHECK_CALL ();
+
+      LOG (" => %s", core_addr_to_string_nz ((CORE_ADDR) th2.th_unique));
+
+      CHECK (memcmp (th, &th2, sizeof (td_thrhandle_t)) == 0);
+    }
+
+  /* Attempt TLS access.  Assuming errno is TLS, this calls
+     thread_db_get_thread_local_address, which in turn calls
+     td_thr_tls_get_addr for live inferiors or td_thr_tlsbase
+     for core files.  This test is skipped if the thread has
+     not been recorded; proceeding in that case would result
+     in the test having the side-effect of noticing threads
+     which seems wrong.
+
+     Note that in glibc's libthread_db td_thr_tls_get_addr is
+     a thin wrapper around td_thr_tlsbase; this check always
+     hits the bulk of the code.
+
+     Note also that we don't actually check any libthread_db
+     calls are made, we just assume they were; future changes
+     to how GDB accesses TLS could result in this passing
+     without exercising the calls it's supposed to.  */
+  ptid_t ptid = ptid_build (tdb_testinfo->info->pid, ti.ti_lid, 0);
+  struct thread_info *thread_info = find_thread_ptid (ptid);
+  if (thread_info != NULL && thread_info->priv != NULL)
+    {
+      LOG ("; errno");
+
+      scoped_restore_current_thread restore_current_thread;
+      switch_to_thread (ptid);
+
+      expression_up expr = parse_expression ("(int) errno");
+      struct value *val = evaluate_expression (expr.get ());
+
+      if (tdb_testinfo->log_progress)
+	{
+	  struct value_print_options opts;
+
+	  get_user_print_options (&opts);
+	  LOG (" = ");
+	  value_print (val, gdb_stdlog, &opts);
+	}
+    }
+
+  LOG (" ... OK\n");
+
+#undef LOG
+#undef CHECK_1
+#undef CHECK
+#undef CALL_UNCHECKED
+#undef CHECK_CALL
+#undef CALL
+
+  return 0;
+}
+
+/* Run integrity checks on the dlopen()ed libthread_db described by
+   INFO.  Returns true on success, displays a warning and returns
+   false on failure.  Logs progress messages to gdb_stdlog during
+   the test if LOG_PROGRESS is true.  */
+
+static bool
+check_thread_db (struct thread_db_info *info, bool log_progress)
+{
+  bool test_passed = true;
+
+  if (log_progress)
+    debug_printf (_("Running libthread_db integrity checks:\n"));
+
+  /* GDB avoids using td_ta_thr_iter wherever possible (see comment
+     in try_thread_db_load_1 below) so in order to test it we may
+     have to locate it ourselves.  */
+  td_ta_thr_iter_ftype *td_ta_thr_iter_p = info->td_ta_thr_iter_p;
+  if (td_ta_thr_iter_p == NULL)
+    {
+      void *thr_iter = verbose_dlsym (info->handle, "td_ta_thr_iter");
+      if (thr_iter == NULL)
+	return 0;
+
+      td_ta_thr_iter_p = (td_ta_thr_iter_ftype *) thr_iter;
+    }
+
+  /* Set up the test state we share with the callback.  */
+  gdb_assert (tdb_testinfo == NULL);
+  struct check_thread_db_info tdb_testinfo_buf;
+  tdb_testinfo = &tdb_testinfo_buf;
+
+  memset (tdb_testinfo, 0, sizeof (struct check_thread_db_info));
+  tdb_testinfo->info = info;
+  tdb_testinfo->log_progress = log_progress;
+
+  /* td_ta_thr_iter shouldn't be used on running processes.  Note that
+     it's possible the inferior will stop midway through modifying one
+     of its thread lists, in which case the check will spuriously
+     fail.  */
+  linux_stop_and_wait_all_lwps ();
+
+  TRY
+    {
+      td_err_e err = td_ta_thr_iter_p (info->thread_agent,
+				       check_thread_db_callback,
+				       tdb_testinfo,
+				       TD_THR_ANY_STATE,
+				       TD_THR_LOWEST_PRIORITY,
+				       TD_SIGNO_MASK,
+				       TD_THR_ANY_USER_FLAGS);
+
+      if (err != TD_OK)
+	error (_("td_ta_thr_iter failed: %s"), thread_db_err_str (err));
+
+      if (!tdb_testinfo->threads_seen)
+	error (_("no threads seen"));
+    }
+  CATCH (except, RETURN_MASK_ERROR)
+    {
+      if (warning_pre_print)
+	fputs_unfiltered (warning_pre_print, gdb_stderr);
+
+      exception_fprintf (gdb_stderr, except,
+			 _("libthread_db integrity checks failed: "));
+
+      test_passed = false;
+    }
+  END_CATCH
+
+  if (test_passed && log_progress)
+    debug_printf (_("libthread_db integrity checks passed.\n"));
+
+  tdb_testinfo = NULL;
+
+  linux_unstop_all_lwps ();
+
+  return test_passed;
+}
+
 /* Attempt to initialize dlopen()ed libthread_db, described by INFO.
    Return 1 on success.
    Failure could happen if libthread_db does not have symbols we expect,
@@ -627,6 +876,13 @@ try_thread_db_load_1 (struct thread_db_info *info)
 #undef TDB_DLSYM
 #undef CHK
 
+  /* Run integrity checks if requested.  */
+  if (check_thread_db_on_load)
+    {
+      if (!check_thread_db (info, libthread_db_debug))
+	return 0;
+    }
+
   if (info->td_ta_thr_iter_p == NULL)
     {
       struct lwp_info *lp;
@@ -1658,6 +1914,24 @@ info_auto_load_libthread_db (const char *args, int from_tty)
     uiout->message (_("No auto-loaded libthread-db.\n"));
 }
 
+/* Implement 'maintenance check libthread-db'.  */
+
+static void
+maintenance_check_libthread_db (const char *args, int from_tty)
+{
+  int inferior_pid = ptid_get_pid (inferior_ptid);
+  struct thread_db_info *info;
+
+  if (inferior_pid == 0)
+    error (_("No inferior running"));
+
+  info = get_thread_db_info (inferior_pid);
+  if (info == NULL)
+    error (_("No libthread_db loaded"));
+
+  check_thread_db (info, true);
+}
+
 void
 _initialize_thread_db (void)
 {
@@ -1708,6 +1982,23 @@ This options has security implications for untrusted inferiors."),
 Usage: info auto-load libthread-db"),
 	   auto_load_info_cmdlist_get ());
 
+  add_cmd ("libthread-db", class_maintenance,
+	   maintenance_check_libthread_db, _("\
+Run integrity checks on the current inferior's libthread_db."),
+	   &maintenancechecklist);
+
+  add_setshow_boolean_cmd ("check-libthread-db",
+			   class_maintenance,
+			   &check_thread_db_on_load, _("\
+Set whether to check libthread_db at load time."), _("\
+Show whether to check libthread_db at load time."), _("\
+If enabled GDB will run integrity checks on inferior specific libthread_db\n\
+as they are loaded."),
+			   NULL,
+			   NULL,
+			   &maintenance_set_cmdlist,
+			   &maintenance_show_cmdlist);
+
   /* Add ourselves to objfile event chain.  */
   gdb::observers::new_objfile.attach (thread_db_new_objfile);
 
diff --git a/gdb/testsuite/gdb.threads/check-libthread-db.c b/gdb/testsuite/gdb.threads/check-libthread-db.c
new file mode 100644
index 0000000..85a97a9
--- /dev/null
+++ b/gdb/testsuite/gdb.threads/check-libthread-db.c
@@ -0,0 +1,67 @@
+/* This testcase is part of GDB, the GNU debugger.
+
+   Copyright 2017-2018 Free Software Foundation, Inc.
+
+   This program is free software; you can redistribute it and/or modify
+   it under the terms of the GNU General Public License as published by
+   the Free Software Foundation; either version 3 of the License, or
+   (at your option) any later version.
+
+   This program is distributed in the hope that it will be useful,
+   but WITHOUT ANY WARRANTY; without even the implied warranty of
+   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+   GNU General Public License for more details.
+
+   You should have received a copy of the GNU General Public License
+   along with this program.  If not, see <http://www.gnu.org/licenses/>.  */
+
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <sys/types.h>
+#include <unistd.h>
+#include <pthread.h>
+#include <errno.h>
+
+static void
+break_here (void)
+{
+}
+
+static void *
+thread_routine (void *arg)
+{
+  errno = 42;
+
+  break_here ();
+
+  while (1)
+    sleep (1);
+
+  return NULL;
+}
+
+int
+main (int argc, char *argv)
+{
+  pthread_t the_thread;
+  int err;
+
+  err = pthread_create (&the_thread, NULL, thread_routine, NULL);
+  if (err != 0)
+    {
+      fprintf (stderr, "pthread_create: %s (%d)\n", strerror (err), err);
+      exit (EXIT_FAILURE);
+    }
+
+  errno = 23;
+
+  err = pthread_join (the_thread, NULL);
+  if (err != 0)
+    {
+      fprintf (stderr, "pthread_join: %s (%d)\n", strerror (err), err);
+      exit (EXIT_FAILURE);
+    }
+
+  exit (EXIT_SUCCESS);
+}
diff --git a/gdb/testsuite/gdb.threads/check-libthread-db.exp b/gdb/testsuite/gdb.threads/check-libthread-db.exp
new file mode 100644
index 0000000..ddd9c27
--- /dev/null
+++ b/gdb/testsuite/gdb.threads/check-libthread-db.exp
@@ -0,0 +1,114 @@
+# Copyright 2017-2018 Free Software Foundation, Inc.
+
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+# This test only works for native processes on GNU/Linux.
+if {[target_info gdb_protocol] != "" || ![istarget *-linux*]} {
+    continue
+}
+
+standard_testfile
+
+if {[gdb_compile_pthreads "${srcdir}/${subdir}/${srcfile}" "${binfile}" \
+	 executable debug] != "" } {
+    return -1
+}
+
+with_test_prefix "user-initiated check" {
+
+    # User-initiated check with libthread_db not loaded.
+    clean_restart ${binfile}
+
+    gdb_test "maint show check-libthread-db" \
+	"Whether to check libthread_db at load time is off."
+
+    gdb_test_no_output "set stop-on-solib-events 1"
+    gdb_run_cmd
+    gdb_test "" \
+	".*Stopped due to shared library event.*no libraries added or removed.*"
+
+    gdb_test "maint check libthread-db" \
+	"No libthread_db loaded" \
+	"no libpthread.so loaded"
+
+
+    # User-initiated check with NPTL uninitialized.
+    # libthread_db should fake a single thread with th_unique == NULL.
+    gdb_test "continue" \
+	".*Stopped due to shared library event.*Inferior loaded .*libpthread.*"
+
+    gdb_test_sequence "maint check libthread-db" \
+	"libpthread.so not initialized" {
+	    "\[\r\n\]+Running libthread_db integrity checks:"
+	    "\[\r\n\]+\[ \]+Got thread 0x0 => \[0-9\]+ => 0x0 ... OK"
+	    "\[\r\n\]+libthread_db integrity checks passed."
+	}
+
+    # User-initiated check with NPTL fully operational.
+    gdb_test_no_output "set stop-on-solib-events 0"
+    gdb_breakpoint break_here
+    gdb_continue_to_breakpoint break_here
+
+    gdb_test_sequence "maint check libthread-db" \
+	"libpthread.so fully initialized" {
+	    "\[\r\n\]+Running libthread_db integrity checks:"
+	    "\[\r\n\]+\[ \]+Got thread 0x\[1-9a-f\]\[0-9a-f\]+ => \[0-9\]+ => 0x\[1-9a-f\]\[0-9a-f\]+; errno = 23 ... OK"
+	    "\[\r\n\]+\[ \]+Got thread 0x\[1-9a-f\]\[0-9a-f\]+ => \[0-9\]+ => 0x\[1-9a-f\]\[0-9a-f\]+; errno = 42 ... OK"
+	    "\[\r\n\]+libthread_db integrity checks passed."
+	}
+}
+
+with_test_prefix "automated load-time check" {
+
+    # Automated load-time check with NPTL uninitialized.
+    with_test_prefix "libpthread.so not initialized" {
+	clean_restart ${binfile}
+
+	gdb_test_no_output "maint set check-libthread-db 1"
+	gdb_test_no_output "set debug libthread-db 1"
+	gdb_breakpoint break_here
+	gdb_run_cmd
+
+	gdb_test_sequence "" \
+	    "check debug libthread-db output" {
+		"\[\r\n\]+Running libthread_db integrity checks:"
+		"\[\r\n\]+\[ \]+Got thread 0x0 => \[0-9\]+ => 0x0 ... OK"
+		"\[\r\n\]+libthread_db integrity checks passed."
+		"\[\r\n\]+[Thread debugging using libthread_db enabled]"
+	    }
+    }
+
+    # Automated load-time check with NPTL fully operational.
+    with_test_prefix "libpthread.so fully initialized" {
+	clean_restart ${binfile}
+
+	gdb_test_no_output "maint set check-libthread-db 1"
+	gdb_test_no_output "set debug libthread-db 1"
+
+	set test_spawn_id [spawn_wait_for_attach $binfile]
+	set testpid [spawn_id_get_pid $test_spawn_id]
+
+	gdb_test_sequence "attach $testpid" \
+	    "check debug libthread-db output" {
+		"\[\r\n\]+Running libthread_db integrity checks:"
+		"\[\r\n\]+\[ \]+Got thread 0x\[1-9a-f\]\[0-9a-f\]+ => \[0-9\]+ => 0x\[1-9a-f\]\[0-9a-f\]+ ... OK"
+		"\[\r\n\]+\[ \]+Got thread 0x\[1-9a-f\]\[0-9a-f\]+ => \[0-9\]+ => 0x\[1-9a-f\]\[0-9a-f\]+ ... OK"
+		"\[\r\n\]+libthread_db integrity checks passed."
+		"\[\r\n\]+[Thread debugging using libthread_db enabled]"
+	    }
+
+	gdb_exit
+	kill_wait_spawned_process $test_spawn_id
+    }
+}
-- 
1.8.3.1