Re: [PATCH 5/5] Test attaching to a program that constantly spawns short-lived threads

[Pedro Alves <palves at redhat dot com>]

On 12/17/2014 11:10 AM, Yao Qi wrote:
> Pedro Alves <palves@redhat.com> writes:
> 
>> +if [is_remote target] then {
>> +    return 0
>> +}
> 
> We should check
> "![isnative] || [is_remote host] || [target_info exists use_gdb_stub]" instead?

Hmm, I don't think the isnative check would be right.
That would check whether the build and target triplets are the
same, but triplets aren't what matters here.  The issue is that
spawn_wait_for_attach assumes build and target _machines_
(boards) are the same.  And that's what "[is_remote target]"
encodes.

In principle, the test should work with remote host testing,
_if_ the build and target machines are the same, even if that's
not a usual scenario, and in that case, [is_remote target] is false.
If remote host testing, and build != target, then [is_remote target]
should be true.  So seems to me we shouldn't check "is_remote host"
either.

I agree with the use_gdb_stub check though.

So how about we add a helper procedure for this (and use it in
other similar attach tests from here on), so we don't have to
constantly think over the same things?  Added to the patch.

>> +		-re "$gdb_prompt $" {
>> +		    if {$eperm} {
>> +			kfail "gdb/NNNN" "$test (EPERM)"
> 
> Replace NNNN with a PR number?

Ah, completely forgot that, thanks.  I've already added a description
of the kernel issue bit above, so I'll just make this an xfail
instead.

>> +	    # Sleep a bit and try updating the thread list.  We should
>> +	    # know about all threads already at this point.  If we see
>> +	    # "New Thread" or similar being output, then "attach" is
>> +	    # failing to actually attach to all threads in the process,
>> +	    # which would be a bug.
>> +	    sleep 1
>> +	    set saw_new 0
>> +	    set test "info threads"
>> +	    gdb_test_multiple $test $test {
>> +		-re "New " {
>> +		    set saw_new 1
>> +		    exp_continue
>> +		}
>> +		-re "$gdb_prompt $" {
>> +		}
>> +	    }
>> +
>> +	    gdb_assert !$saw_new "no new threads"
> 
> Nit: I feel the test above can be simplified a little bit,
> 
> gdb_test_multiple $test $test {
>     -re "New .*$gdb_prompt $" {
>         fail "no new threads"
>     }
>     -re "$gdb_prompt $" {
>         pass "no new threads"
>     }
> }
> 

Indeed.  I did that change, thanks.

I also cleaned up the test further and fixed a few things.
E.g.: a stumbled on a silly bug here:

> pthread_attr_init (&detached_attr);
> pthread_attr_setdetachstate (&detached_attr, PTHREAD_CREATE_DETACHED);
> pthread_attr_init (&joinable_attr);
> pthread_attr_setdetachstate (&detached_attr, PTHREAD_CREATE_JOINABLE);
                                ^^^^^^^^^^^^^

That lead to resource exhaustion resulting in an occasional timeouts when
testing with the native-extended-gdbserver board.  And then we'd crash here:

+      fprintf (stderr, "unexpected error from pthread_create: %s (%d)\n",
+	       strerror (rc), rc);

because I missed including string.h, leaving strerror unprototyped.

Here's the new version.

>From b44fd42eebcb76a1c84cabae7b763e52c6b8239f Mon Sep 17 00:00:00 2001
From: Pedro Alves <palves@redhat.com>
Date: Wed, 17 Dec 2014 20:40:05 +0000
Subject: [PATCH] Test attaching to a program that constantly spawns
 short-lived threads

Before the previous fixes, on Linux, this would trigger several
different problems, like:

 [New LWP 27106]
 [New LWP 27047]
 warning: unable to open /proc file '/proc/-1/status'
 [New LWP 27813]
 [New LWP 27869]
 warning: Can't attach LWP 11962: No child processes
 Warning: couldn't activate thread debugging using libthread_db: Cannot find new threads: debugger service failed
 warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.

gdb/testsuite/
2014-12-17  Pedro Alves  <palves@redhat.com>

	* gdb.threads/attach-many-short-lived-threads.c: New file.
	* gdb.threads/attach-many-short-lived-threads.exp: New file.
	* lib/gdb.exp (can_spawn_for_attach): New procedure.
	(spawn_wait_for_attach): Error out if can_spawn_for_attach returns
	false.
---
 .../gdb.threads/attach-many-short-lived-threads.c  | 151 +++++++++++++++++++++
 .../attach-many-short-lived-threads.exp            | 132 ++++++++++++++++++
 gdb/testsuite/lib/gdb.exp                          |  24 ++++
 3 files changed, 307 insertions(+)
 create mode 100644 gdb/testsuite/gdb.threads/attach-many-short-lived-threads.c
 create mode 100644 gdb/testsuite/gdb.threads/attach-many-short-lived-threads.exp

diff --git a/gdb/testsuite/gdb.threads/attach-many-short-lived-threads.c b/gdb/testsuite/gdb.threads/attach-many-short-lived-threads.c
new file mode 100644
index 0000000..0528695
--- /dev/null
+++ b/gdb/testsuite/gdb.threads/attach-many-short-lived-threads.c
@@ -0,0 +1,151 @@
+/* This testcase is part of GDB, the GNU debugger.
+
+   Copyright 2014 Free Software Foundation, Inc.
+
+   This program is free software; you can redistribute it and/or modify
+   it under the terms of the GNU General Public License as published by
+   the Free Software Foundation; either version 3 of the License, or
+   (at your option) any later version.
+
+   This program is distributed in the hope that it will be useful,
+   but WITHOUT ANY WARRANTY; without even the implied warranty of
+   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+   GNU General Public License for more details.
+
+   You should have received a copy of the GNU General Public License
+   along with this program.  If not, see <http://www.gnu.org/licenses/>.  */
+
+#define _GNU_SOURCE
+#include <assert.h>
+#include <pthread.h>
+#include <stdlib.h>
+#include <stdio.h>
+#include <unistd.h>
+#include <string.h>
+
+pthread_t main_thread;
+pthread_attr_t detached_attr;
+pthread_attr_t joinable_attr;
+
+/* Number of threads we'll create of each variant
+   (joinable/detached).  */
+int n_threads = 50;
+
+/* Mutex used to hold creating detached threads.  */
+pthread_mutex_t dthrds_create_mutex;
+
+/* Wrapper for pthread_create.   */
+
+void
+create_thread (pthread_attr_t *attr,
+	       void *(*start_routine) (void *), void *arg)
+{
+  pthread_t child;
+  int rc;
+
+  while ((rc = pthread_create (&child, attr, start_routine, arg)) != 0)
+    {
+      fprintf (stderr, "unexpected error from pthread_create: %s (%d)\n",
+	       strerror (rc), rc);
+      sleep (1);
+    }
+}
+
+void
+break_fn (void)
+{
+}
+
+/* Data passed to joinable threads on creation.  This is allocated on
+   the heap and ownership transferred from parent to child.  (We do
+   this because it's not portable to cast pthread_t to pointer.)  */
+
+struct thread_arg
+{
+  pthread_t parent;
+};
+
+/* Entry point for joinable threads.  These threads first join their
+   parent before spawning a new child (and exiting).  The parent's tid
+   is passed as pthread_create argument, encapsulated in a struct
+   thread_arg object.  */
+
+void *
+joinable_fn (void *arg)
+{
+  struct thread_arg *p = arg;
+
+  pthread_setname_np (pthread_self (), "joinable");
+
+  if (p->parent != main_thread)
+    assert (pthread_join (p->parent, NULL) == 0);
+
+  p->parent = pthread_self ();
+
+  create_thread (&joinable_attr, joinable_fn, p);
+
+  break_fn ();
+
+  return NULL;
+}
+
+/* Entry point for detached threads.  */
+
+void *
+detached_fn (void *arg)
+{
+  pthread_setname_np (pthread_self (), "detached");
+
+  /* This should throttle threads a bit in case we manage to spawn
+     threads faster than they exit.  */
+  pthread_mutex_lock (&dthrds_create_mutex);
+
+  create_thread (&detached_attr, detached_fn, NULL);
+
+  /* Note this is called before the mutex is unlocked otherwise in
+     non-stop mode, when the breakpoint is hit we'd keep spawning more
+     threads forever while the old threads stay alive (stopped in the
+     breakpoint).  */
+  break_fn ();
+
+  pthread_mutex_unlock (&dthrds_create_mutex);
+
+  return NULL;
+}
+
+int
+main (int argc, char *argv[])
+{
+  int i;
+
+  if (argc > 1)
+    n_threads = atoi (argv[1]);
+
+  pthread_mutex_init (&dthrds_create_mutex, NULL);
+
+  pthread_attr_init (&detached_attr);
+  pthread_attr_setdetachstate (&detached_attr, PTHREAD_CREATE_DETACHED);
+  pthread_attr_init (&joinable_attr);
+  pthread_attr_setdetachstate (&joinable_attr, PTHREAD_CREATE_JOINABLE);
+
+  main_thread = pthread_self ();
+
+  /* Spawn the initial set of test threads.  Some threads are
+     joinable, others are detached.  This exercises different code
+     paths in the runtime.  */
+  for (i = 0; i < n_threads; ++i)
+    {
+      struct thread_arg *p;
+
+      p = malloc (sizeof *p);
+      p->parent = main_thread;
+      create_thread (&joinable_attr, joinable_fn, p);
+
+      create_thread (&detached_attr, detached_fn, NULL);
+    }
+
+  /* Long enough for all the attach/detach sequences done by the .exp
+     file.  */
+  sleep (180);
+  return 0;
+}
diff --git a/gdb/testsuite/gdb.threads/attach-many-short-lived-threads.exp b/gdb/testsuite/gdb.threads/attach-many-short-lived-threads.exp
new file mode 100644
index 0000000..ff39956
--- /dev/null
+++ b/gdb/testsuite/gdb.threads/attach-many-short-lived-threads.exp
@@ -0,0 +1,132 @@
+# Copyright 2008-2014 Free Software Foundation, Inc.
+
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+# Test attaching to a program that is constantly spawning short-lived
+# threads.  The stresses the edge cases of attaching to threads that
+# have just been created or are in process of dying.  In addition, the
+# test attaches, debugs, detaches, reattaches in a loop a few times,
+# to stress the behavior of the debug API around detach (some systems
+# end up leaving stale state behind that confuse the following
+# attach).
+
+if {![can_spawn_for_attach]} {
+    return 0
+}
+
+standard_testfile
+
+# The test proper.  See description above.
+
+proc test {} {
+    global binfile
+    global gdb_prompt
+    global decimal
+
+    clean_restart ${binfile}
+
+    set testpid [spawn_wait_for_attach $binfile]
+
+    set attempts 10
+    for {set attempt 1} { $attempt <= $attempts } { incr attempt } {
+	with_test_prefix "iter $attempt" {
+	    set attached 0
+	    set eperm 0
+	    set test "attach"
+	    gdb_test_multiple "attach $testpid" $test {
+		-re "new threads in iteration" {
+		    # Seen when "set debug libthread_db" is on.
+		    exp_continue
+		}
+		-re "warning: Cannot attach to lwp $decimal: Operation not permitted" {
+		    # On Linux, PTRACE_ATTACH sometimes fails with
+		    # EPERM, even though /proc/PID/status indicates
+		    # the thread is running.
+		    set eperm 1
+		    exp_continue
+		}
+		-re "debugger service failed.*$gdb_prompt $" {
+		    fail $test
+		}
+		-re "$gdb_prompt $" {
+		    if {$eperm} {
+			xfail "$test (EPERM)"
+		    } else {
+			pass $test
+		    }
+		}
+		-re "Attaching to program.*process $testpid.*$gdb_prompt $" {
+		    pass $test
+		}
+	    }
+
+	    # Sleep a bit and try updating the thread list.  We should
+	    # know about all threads already at this point.  If we see
+	    # "New Thread" or similar being output, then "attach" is
+	    # failing to actually attach to all threads in the process,
+	    # which would be a bug.
+	    sleep 1
+
+	    set test "no new threads"
+	    gdb_test_multiple "info threads" $test {
+		-re "New .*$gdb_prompt $" {
+		    fail $test
+		}
+		-re "$gdb_prompt $" {
+		    pass $test
+		}
+	    }
+
+	    # Force breakpoints always inserted, so that threads we might
+	    # have failed to attach to hit them even when threads we do
+	    # know about are stopped.
+	    gdb_test_no_output "set breakpoint always-inserted on"
+
+	    # Run to a breakpoint a few times.  A few threads should spawn
+	    # and die meanwhile.  This checks that thread creation/death
+	    # events carry on correctly after attaching.  Also, be
+	    # detaching from the program and reattaching, we check that
+	    # the program doesn't die due to gdb leaving a pending
+	    # breakpoint hit on a new thread unprocessed.
+	    gdb_test "break break_fn" "Breakpoint.*" "break break_fn"
+
+	    # Wait a bit, to give time for most threads to hit the
+	    # breakpoint, including threads we might have failed to
+	    # attach.
+	    sleep 2
+
+	    set bps 3
+	    for {set bp 1} { $bp <= $bps } { incr bp } {
+		gdb_test "continue" "Breakpoint.*" "break at break_fn: $bp"
+	    }
+
+	    if {$attempt < $attempts} {
+		gdb_test "detach" "Detaching from.*"
+	    } else {
+		gdb_test "kill" "" "kill process" "Kill the program being debugged.*y or n. $" "y"
+	    }
+
+	    gdb_test_no_output "set breakpoint always-inserted off"
+	    delete_breakpoints
+	}
+    }
+
+    remote_exec target "kill -9 ${testpid}"
+}
+
+if {[prepare_for_testing "failed to prepare" $testfile $srcfile {debug pthreads}] == -1} {
+    return -1
+}
+
+test
diff --git a/gdb/testsuite/lib/gdb.exp b/gdb/testsuite/lib/gdb.exp
index 08087f2..83fa1d0 100644
--- a/gdb/testsuite/lib/gdb.exp
+++ b/gdb/testsuite/lib/gdb.exp
@@ -3413,12 +3413,36 @@ proc gdb_exit { } {
     catch default_gdb_exit
 }

+# Return true if we can spawn a program on the target and attach to
+# it.
+
+proc can_spawn_for_attach { } {
+    # We use TCL's exec to get the inferior's pid.
+    if [is_remote target] then {
+	return 0
+    }
+
+    # The "attach" command doesn't make sense when the target is
+    # stub-like, where GDB finds the program already started on
+    # initial connection.
+    if {[target_info exists use_gdb_stub]} {
+	return 0
+    }
+
+    # Assume yes.
+    return 1
+}
+
 # Start a set of programs running and then wait for a bit, to be sure
 # that they can be attached to.  Return a list of the processes' PIDs.

 proc spawn_wait_for_attach { executable_list } {
     set pid_list {}

+    if ![can_spawn_for_attach] {
+	error "can't spawn for attach with this target/board"
+    }
+
     foreach {executable} $executable_list {
 	lappend pid_list [eval exec $executable &]
     }
-- 
1.9.3