Re: Analyzing process and core dump files in changed root environments

[Mark Wielaard <mjw at redhat dot com>]

Hi Jakub,

On Tue, 2016-11-29 at 13:37 +0100, h wrote:
> I have a question regarding analyzing processes running in a change root 
> environment or in an own mount namespace with re-mounted /.
> I want to debug a process running in a non-system root using elfutils or gdb
> from system root, and vice versa.
> More precisely, I want to create a container shipping ABRT + GDB + elfutils 
> and the container should allow users to analyze core dump files on Fedora 
> Atomic. I need to run eu-stack on a core dump file generated by a program 
> from system root and I need to run it from a container (non-system root).
> 
> For example, if I run the following command on Fedora Atomic 25, it crashes 
> in the system namespace and ABRT saves its core file to /var/spool/abrt/
> ccpp* in ABRT's namespace. If I enter the ABRT container and run eu-stack on
> the core dump file, I get an error because the container misses some 
> libraries:
> [...]
> You may ask what is the binary argument. It's a copy of the crashed program.
> It can be replaced with the `/host/bin/ostree` path (I bind mount the hots' 
> / to the container's /host).
> 
> GDB allows me to configure 'sysroot', so it knows where to look for 
> libraries.  Thus if I run GDB on the same core dump file, I get the 
> following ouptut:
> $ gdb --batch --ex 'set sysroot /host' --ex 'file binary' --ex 'core 
> coredump' --ex 'bt'
> [...]
> Do elfutils support such an option too?

We already talked briefly on irc. And the short answer is:  No, at the
moment not. Thanks for filing
https://bugzilla.redhat.com/show_bug.cgi?id=1399637

The slightly longer versions is: No, not "globally" and only for some
files/types. You can set how to find debuginfo files by providing a path
or alternative callback when setting up a Dwfl. See Dwfl_Callbacks
debuginfo_path and find_debuginfo. Usually you would use
dwfl_standard_find_debuginfo for this. You can also provide a find_elf
callback that finds the Elf image for a module. Usually you would use
either dwfl_build_id_find_elf, dwfl_linux_proc_find_elf or
dwfl_linux_kernel_find_elf for that.

We probably should provide alternatives to those that take some sysroot.
But there are some parts inside the link_map scanning code that cannot
easily use those callbacks at the moment and use direct open calls that
also would need sysroot "hooks" (there are already some XXX hook for
sysroot comments).

But maybe the above is too flexible. Which (types of) files would you
expect a "sysroot" setting to affect? Just direct file paths, lookups
through build-ids, separate debuginfo files, kernel module lookups,
etc.?

Thanks,

Mark