[commit] Improve ptrace return type / argument checks for GNU/Linux

[Daniel Jacobowitz]

This patch is descended from a conversation I had with Mark Kettenis
in September 2005.  At the time I was trying to add support for a
ptrace variant which took and returned long long instead of long.
Mark talked me out of it, and while getting the ABI changed was a bit
of a hassle, I'm convinced it was a good move (thanks!).

But that still left the earlier problem.  We compile seventeen tests
trying to check the argument types for ptrace but none of them can
possibly work - the approach we're using is not compatible with a
varargs prototype in the system headers.  So I've added an explicit
check for that, and forced us to assume the vararg arguments are
"int,int,long,long".  That's what we've been assuming all along, and
it works perfectly; the only change is we now find it deliberately,
rather than by accident.

Tested on x86_64-linux and mips-linux, and committed.
 
-- 
Daniel Jacobowitz
CodeSourcery

2008-07-27  Daniel Jacobowitz  <dan@codesourcery.com>

	* configure.ac: Check for the GNU/Linux ptrace signature.
	* configure: Regenerated.

---
 gdb/configure    |   92 ++++++++++++++++++++++++++++++++++++++++++++++++++++++-
 gdb/configure.ac |   17 +++++++---
 2 files changed, 103 insertions(+), 6 deletions(-)

Index: gdb-only/gdb/configure.ac
===================================================================
--- gdb-only.orig/gdb/configure.ac	2008-07-26 23:21:30.000000000 -0400
+++ gdb-only/gdb/configure.ac	2008-07-26 23:32:59.000000000 -0400
@@ -643,16 +643,23 @@ AC_CHECK_DECLS(ptrace, [], [
   : ${gdb_cv_func_ptrace_ret='int'}
   : ${gdb_cv_func_ptrace_args='int,int,long,long'}
 ], $gdb_ptrace_headers)
-# Check return type.
+# Check return type.  Varargs (used on GNU/Linux) conflict with the
+# empty argument list, so check for that explicitly.
 AC_CACHE_CHECK([return type of ptrace], gdb_cv_func_ptrace_ret,
   AC_TRY_COMPILE($gdb_ptrace_headers,
-    [extern int ptrace ();],
-    gdb_cv_func_ptrace_ret='int',
-    gdb_cv_func_ptrace_ret='long'))
+    [extern long ptrace (enum __ptrace_request, ...);],
+    gdb_cv_func_ptrace_ret='long',
+    AC_TRY_COMPILE($gdb_ptrace_headers,
+      [extern int ptrace ();],
+      gdb_cv_func_ptrace_ret='int',
+      gdb_cv_func_ptrace_ret='long')))
 AC_DEFINE_UNQUOTED(PTRACE_TYPE_RET, $gdb_cv_func_ptrace_ret,
   [Define as the return type of ptrace.])
 # Check argument types.
 AC_CACHE_CHECK([types of arguments for ptrace], gdb_cv_func_ptrace_args, [
+  AC_TRY_COMPILE($gdb_ptrace_headers,
+    [extern long ptrace (enum __ptrace_request, ...);],
+    [gdb_cv_func_ptrace_args='int,int,long,long'],[
 for gdb_arg1 in 'int' 'long'; do
  for gdb_arg2 in 'pid_t' 'int' 'long'; do
   for gdb_arg3 in 'int *' 'caddr_t' 'int' 'long' 'void *'; do
@@ -676,7 +683,7 @@ gdb_cv_func_ptrace_args="$gdb_arg1,$gdb_
 done
 # Provide a safe default value.
 : ${gdb_cv_func_ptrace_args='int,int,long,long'}
-])
+])])
 ac_save_IFS=$IFS; IFS=','
 set dummy `echo "$gdb_cv_func_ptrace_args" | sed 's/\*/\*/g'`
 IFS=$ac_save_IFS