Unable to debug core dumps generated by shared libraries loaded in Python

["Josh Davidson" <joshuadavidson at gmail dot com>]

I'm having an issue debugging some segfaults that are being generated by an
application that runs in a Python interpreter.  The machine in question is
running RHEL 6 x86_64 with GDB 7.2.  Whenever I attempt to debug core file
that was generated from Python, I receive a bunch of "Cannot access memory
address <address here>" messages and receive unhelpful stack traces.  The
weird thing is, is that I can view the core files on other RHEL 6 machines
with identical packages installed.

Here is an example:
> gdb python core.4586
GNU gdb (GDB) Red Hat Enterprise Linux (7.2-64.el6_5.2)
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
<http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /opt/goes/bin/python...(no debugging symbols
found)...done.

warning: core file may not match specified executable file.
[New Thread 4774]
[New Thread 4887]
[New Thread 4886]
[New Thread 4776]
[New Thread 4888]
[New Thread 4711]
[New Thread 4890]
[New Thread 4586]
[New Thread 4591]
[New Thread 4680]
[New Thread 4703]
[New Thread 4783]
[New Thread 4695]
[New Thread 4694]
[New Thread 4590]
[New Thread 4889]
[New Thread 4773]
[New Thread 4775]
Cannot access memory at address 0xf
Cannot access memory at address 0xf
Cannot access memory at address 0xf
Cannot access memory at address 0xf
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols
found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Cannot access memory at address 0xf
Core was generated by `/opt/goes/bin/python /opt/goes/bin/py.test
--mission=AST43 --skip_teardown_dump'.
Program terminated with signal 11, Segmentation fault.
#0  0x00007fab3f43d766 in ?? ()
Missing separate debuginfos, use: debuginfo-install
goes-python-2.7.5-2.el6.x86_64
(gdb) bt
#0  0x00007fab3f43d766 in ?? ()
#1  0x00007fab281cef30 in ?? ()
#2  0x0000000000000000 in ?? ()


If I hop over to a different machine, gdb works fine:
http://pastebin.com/raw.php?i=9ZZYNgHb

It doesn't appear to be an environment issue since I have a roaming home
directory, and have dumped my env and re-sourced.  My colleagues have the
same issue on this particular machine.

I attempted to use a newer version of gdb (7.5) via the developer toolset
collection and hit the same snag:  http://pastebin.com/raw.php?i=XaUxp3gM
The only thing that changes is the format is slightly different.  I also
uninstalled and re-installed the system packages for GDB .

I'm at a loss right now. Non-Python applications that I've debugged on this
machine seem to open fine.  We are using a non-system version of Python, but
that is consistent on all of our machines.  I fired up gdb against the core
file using strace, but nothing is jumping out at me (other than that the
good runs actually open shared libraries).

Here is the strace from the bad machine:
http://sebsauvage.net/paste/?70af687dc1ca7989#/BV9w+3k+RJlNwj7HjPuEgEcjWi6CT
Eg3AFTf0L7Dq8
Now one of the good machines:
http://sebsauvage.net/paste/?6c245b4182aaf267#1ItDb+EmTs28o7KKS7c6Kp8H8g4La9
YC+xgA5TZ1wKQ=
And if you're really curious, an strace from the bad machine using GDB 7.5:
http://sebsauvage.net/paste/?96b5609b2c55912c#M/WgGU3vlGRGgN2rYyURn1LrXXSa2I
zr6RX7++0+8Yo=

One thing I'll point out is that GDB does curiously open the system Python
installation in /usr, but it does that in all cases, so that in and of
itself, doesn't seem to be the issue.

Any ideas on what is going on here?

Thanks,
Josh