The strategy used right now to detect ptrace argument types is to try to re-declare the ptrace function with various argument combinations until it works. This relies on the fact that a function can't be re-declared with different arguments in C. For example, ptrace is declared like this in OpenBSD: int ptrace(int _request, pid_t _pid, caddr_t _addr, int _data); When the configure test tries to re-declare it as, let's say, int ptrace(long, int, int, int); the compilation fails and it knows that this isn't the right combination of arguments. When building with --enable-build-with-cxx, that test is done with the c++ compiler (see 54019719152ab269fb4cec2c6a8a245ba6af6e49). This strategy simply doesn't work in C++, since it's valid to re-declare a function with different parameters (because overloading is allowed). So the first arguments combination is always considered the right one and is used. The build fails later when ptrace is actually used. Here are some possible ideas for a solution. - We only need to use C++ for the first part of the test, to properly detect if the first argument is an enum or not. We could revert to C for the second part of the test. - Instead of just re-declaring the function, try to compile a call to it with casts to the types to try: ptrace((long) 0, (int) 0, (int) 0, (int) 0) However, it might not work because of implicit silent conversion between int types. - Use extern "C" to re-declare ptrace. It wasn't obvious to try, because the AC_TRY_COMPILE autoconf macro inserts the code in the body of main, and it doesn't seem valid to use extern "C" in that scope. The declaration should be moved outside, to the global scope. That would require using another (probably lower level) autoconf macro.
The first argument of AC_TRY_COMPILE provides access to the global scope.
Ah, right. It's called "includes", which is why I dismissed it, but we can still use it for other purposes.
TBH, I prefer the extern "C" approach.
The master branch has been updated by Pedro Alves <palves@sourceware.org>: https://sourceware.org/git/gitweb.cgi?p=binutils-gdb.git;h=9b30624b65697a5f51bb7120c48686ab5506067f commit 9b30624b65697a5f51bb7120c48686ab5506067f Author: Pedro Alves <palves@redhat.com> Date: Mon Apr 18 17:42:50 2016 +0100 Fix PR gdb/19250: ptrace prototype is not detected properly in C++ mode The ptrace args/return types detection doesn't work properly in C++ mode, on non-GNU/Linux hosts. For example, on gcc70 (NetBSD 5.1), where the prototype is: int ptrace(int, __pid_t, void*, int); configure misdetects it as: $ grep PTRACE_TYPE config.h #define PTRACE_TYPE_ARG1 int #define PTRACE_TYPE_ARG3 int * #define PTRACE_TYPE_ARG4 int /* #undef PTRACE_TYPE_ARG5 */ #define PTRACE_TYPE_RET int resulting in: ../../src/gdb/amd64bsd-nat.c: In function 'void amd64bsd_fetch_inferior_registers(target_ops*, regcache*, int)': ../../src/gdb/amd64bsd-nat.c:56: warning: dereferencing type-punned pointer will break strict-aliasing rules ../../src/gdb/amd64bsd-nat.c: In function 'void amd64bsd_store_inferior_registers(target_ops*, regcache*, int)': ../../src/gdb/amd64bsd-nat.c:104: warning: dereferencing type-punned pointer will break strict-aliasing rules ../../src/gdb/amd64bsd-nat.c:110: warning: dereferencing type-punned pointer will break strict-aliasing rules We could address this [1], however despite ptrace.m4's claim: # Needs to be tested in C++ mode, to detect whether we need to cast # the first argument to enum __ptrace_request. it appears that there's actually no need to test in C++ mode. Always running the ptrace tests in C mode works just the same on GNU/Linux. I remember experimenting with several different ways to handle the original issue back then, and maybe that was needed in some other attempt and then I didn't realize it ended up not really necessary. Confirmed that this fixes the NetBSD 5.1 C++ build, and confirmed that C and C++ builds on Fedora 23 are unaffected. [1] - https://sourceware.org/ml/gdb-patches/2016-04/msg00374.html gdb/ChangeLog: 2016-04-18 Pedro Alves <palves@redhat.com> * ptrace.m4 (GDB_AC_PTRACE): Don't run tests in C++ mode. * configure: Regenerate. gdb/gdbserver/ChangeLog: 2016-04-18 Pedro Alves <palves@redhat.com> * configure: Regenerate.
Should be fixed now.