Finding GDB's Python Libraries

[Joel Brobecker]

Hi Andrew,

On Mon, Jul 06, 2020 at 06:33:46PM +0100, Andrew Burgess wrote:
> I have been thinking about how we tell GDB where to find the Python
> libraries, and I'm currently going around in circles trying to figure
> out what the right thing to do is.
> 
> Some background reading, some original work to tell GDB about where to
> find Python libraries:
> 
>   https://sourceware.org/legacy-ml/gdb-patches/2010-05/msg00434.html
> 
> This allowed GDB to figure out where the Python it was configured with
> was located, and from that Python can figure out where its libraries are.
> 
> Then there was this:
> 
>   https://sourceware.org/pipermail/gdb-patches/2020-February/165389.html
> 
> which built on the above, but was more flexible in some cross
> compiling situations, not requiring Python to actually be installed
> into the GDB install tree (just the libraries can be installed).
> 
> I recently ran into a situation where I needed to cross-build GDB for
> a MinGW target, including Python support.
> 
> By writing a support script as described here:
> 
>   https://sourceware.org/gdb/wiki/CrossCompilingWithPythonSupport
> 
> I was able to successfully build a MinGW GDB on Linux.  I configured
> this GDB something like this:
> 
>   /project/gdb/src/configure --target=.....  --host=i686-w64-mingw32 \
>                              --prefix=/project/gdb/install
>                              --with-python=/project/gdb/x-compile-script.sh
>                              --with-python-libdir=/project/gdb/install/lib/
> 
> After building and installing GDB I then copied the Python libraries
> and the Python DLL from an unpacked i686-w64-mingw32 Python package
> into the GDB install directory. With this done I was able to zip the
> GDB install directory and distribute it as needed, it all worked fine,
> except....
> 
> ... I ran into a case where a user had PYTHONHOME set in their
> environment, and the Python libraries that this path pointed to were
> not compatible with the MinGW Python version that GDB was linking
> against.
> 
> The motivation for originally setting PYTHONHOME in the environment
> have been lost to the mists of time, but this started me thinking.
> 
> If we have configured GDB to use a specific set of Python libraries,
> even going so far (in this case) to ship the libraries along with GDB,
> should we allow PYTHONHOME to override that library selection?
> 
> My initial thought was sure, that makes sense, that's what PYTHONHOME
> is for.  But, what if the user really needed PYTHONHOME to be set in
> order to correctly use their installed (nothing to do with GDB) Python
> interpreter?  In that case they now have a situation, through no fault
> of their own, where GDB doesn't "just work".
> 
> 
> A quick digression on --with-python and --with-python-libdir:
> Currently, even when the --with-python-libdir flag is not passed at
> configure time, if configure chooses to use Python, then a setting for
> this flag is created within the configure script.  As far as GDB is
> concerned it is impossible to tell if the user specified
> --with-python-libdir, or if configure filled this in for us.  I think
> this might be important shortly.
> 
> 
> Currently within GDB we call Py_SetProgramName.  This call tells the
> Python library where the Python executable is, and from this path the
> Python library figures out where the libraries are.  Funnily the value
> we pass to Py_SetProgramName is actually built within GDB based on the
> location where the libraries are.  So,
> 
>   --with-python-libdir=/usr/lib
> 
> Causes us to call:
> 
>   Py_SetProgramName ("/usr/bin/python")
> 
> >From which the Python library figures out that the libraries are
> located at:
> 
>   /usr/lib
> 
> It doesn't actually matter to the Python library if there isn't a
> python at /usr/bin/python, so long as the libraries are where it
> expects them all is good.  This is why we can do:
> 
>   --with-python=/usr/bin/python
>   --with-python-libdir=/project/gdb/install/lib
> 
> Even if there is never a /project/gdb/install/bin/python, our the
> Python library will still find the libraries correctly.
> 
> Setting the program name in this way still allows for PYTHONHOME to
> override the location of the libraries.  However, we could switch to
> using Py_SetPythonHome, which also tells the Python library where to
> finds its library files, but doesn't honour PYTHONHOME.
> 
> In some cases using Py_SetPythonHome seems like a better choice, but
> in other cases not listening to PYTHONHOME feels like a mistake, and
> Py_SetProgramName feels like a better choice.  The problem I'm
> struggling with is what would be a good set of rules for choosing
> between these two options?
> 
> I initially thought that maybe if the user has specified a
> --with-python-libdir that is under the install prefix of GDB then we
> should not honour PYTHONPATH (so use Py_SetPythonHome), however, that
> doesn't feel strong enough.
> 
> What if the user does:
> 
>   --with-python=/opt/my-special-python/bin/python
>   --with-python-libdir=/opt/my-special-python/lib
> 
> Should they not expect GDB to pick this up in preference to any other
> Python?
> 
> So then I wondered if we could detect the case where configure has
> found the "default" Python on the machine, so, if the user configures
> like:
> 
>   --with-python=yes
> 
> Then they will get whatever python configure can find, maybe in this
> case, and this case only we should listen to PYTHONHOME, but if the
> user has specifically specified a particular version of python in any
> location, then we should force GDB to use that python above all
> others?
> 
> If we did go down this route (or make any use of Py_SetPythonHome)
> then I would suggest we have GDB look for a new environment variable
> GDB_PYTHONHOME, which would be just like PYTHONHOME, but only for GDB,
> and would override the value from --with-python-libdir.
> 
> I'd be interested to hear if anyone has any thoughts on this issue.

At AdaCore, we ship a version that overrides any value that PYTHONHOME
might have, for exactly the reason that you explained. You hit that issue
on Windows, but from our perspective, this isn't just OS-specific either.

I found the following reference to a discussion where we proposed
the approach:
https://sourceware.org/pipermail/gdb-patches/2010-November/078220.html
https://sourceware.org/pipermail/gdb-patches/2010-December/078618.html
https://sourceware.org/pipermail/gdb-patches/2011-January/079287.html

It looks like the discussed died because I gave up due to lack of
support, that Doug clarified by saying we should use honor
GDB_PYTHONHOME and disregard PYTHONHOME instead. My answer was
not very clear, but it sounds like I was thinking "someone else"
can take care of it (I'm wondering how I could have thought that,
but anyways).

Attached is a local patch that we have internally at AdaCore.
I think you could build on it to add support for GDB_PYTHONHOME
and submit it, if others agree that this is going in the right
direction.

-- 
Joel
-------------- next part --------------
A non-text attachment was scrubbed...
Name: python-home-adacore.diff
Type: text/x-diff
Size: 2088 bytes
Desc: not available
URL: <https://sourceware.org/pipermail/gdb/attachments/20200706/983ec4fb/attachment.bin>