Re: [RFC PATCH 0/6] Support for Linux kernel thread aware debug

[John Baldwin <jhb at FreeBSD dot org>]

On 2/4/19 1:12 AM, Omair Javaid wrote:
> On Tue, 29 Jan 2019 at 22:30, John Baldwin <jhb@freebsd.org> wrote:
>>
>> On 1/28/19 9:03 PM, Omair Javaid wrote:
>>> This patch series implements linux kernel target which allows linux kernel
>>> tasks to be viewed as GDB threads. We have had multiple set of patches
>>> submitted over last few years aiming to insert add-ons into GDB for debugging
>>> Linux kernel. This patch series builds on top of Linux kernel infrastructure
>>> submitted by Philipp Rudo in his various sets of patches aiming to debug
>>> Linux kernel dumps.
>>
>> I just have one minor suggestion / comment about file names.  I maintain
>> FreeBSD kernel patches for gdb out-of-tree (for various reasons), and those
>> patches use some similar things (e.g. a different OSABI).  My comment has
>> to do with the filenames.  Other osabi-specific files tend to use more
>> verbose names such as 'linux-arm-nat.c'.  I wonder if it makes sense to
>> spell out linux here as well.  I have been using 'arm-fbsd-kern.c' as a
>> complement to 'arm-fbsd-tdep.c' for the kernel gdbarch.  The architecture
>> independent files follow the patter 'fbsd-k*.c' (e.g. fbsd-kld.c for modules
>> and fbsd-kthr.c for thread enumeration), but I would be happy to move those
>> to something like 'fbsd-kern-ld.c' and 'fbsd-kern-thr.c'.  For your current
>> patchset that might mean something like 'linux-kern-tdep.c' instead of
>> 'lk-tdep.c'.  I would also be fine with 'arm-linux-kern-tdep.c' instead of
>> 'arm-linux-kern.c' perhaps if other folks feel like that is more consistent.
> 
> Hi John,
> 
> Andreas has aptly described the reason behind choosing the
> nomenclature of new linux kernel specific files as it is right now and
> i am open to any suggestion that my come up during reviews.

Ok.  I'm not firmly tied to a specific naming scheme, and I'll defer to
whatever the global maintainers prefer.

> Also I was wondering if you can share details of kernel debugging
> features which has implemented in your out of tree FreeBSD patches for
> GDB. Also share some insight on ways to test this patchset.

So I don't have any formalized testing. :-/  For testing cross-debug of
crash dumps I modified the libkvm library in FreeBSD's base system to
support reading the kernel crash dump format of other architectures and
can then generate a crash dump in a VM / qemu instance and copy it to a
64-bit x86 host to examine against a sysroot.  However, most of my testing
is just by using the kernel target as most of my work is FreeBSD kernel
and userland stuff, so I exercise it that way.  I do have various gdb scripts
(still written in the ancient gdb script language from gdb 6.x rather than
python) at github.com/bsdjhb/kdbg/tree/master/gdb  They aren't generally
useful on other kernels though and are specific to FreeBSD.

In terms of features, the current FreeBSD patches support most FreeBSD
architectures (sparc64 is untested and I haven't gotten stack traces on
ppc to work right, though I think I can reuse what I just did for risc-v
to fix ppc at some point).  A few years ago I reworked the libkvm library
in FreeBSD to support cross-debugging of crashdumps, so you can generally
cross-debug crashdumps on FreeBSD.  Currently this only works on a FreeBSD
host because I haven't pulled out a portable version of libkvm that could
be compiled on a non-FreeBSD host yet.  It does support kernel modules by
reusing the shared library interface.  This is implemented by having a
custom set of solib_ops that get attached to the kernel-specific gdbarch
during that OSABI's init handler.  It has some custom commands that are
wrappers around 'thread N' that let you switch to a thread by process ID
or kernel thread ID.  The main things it requires of each architecture are
a method for populating a regcache from a kernel thread register state
('struct pcb' on FreeBSD), and custom frame unwinders to walk across
exception frames on the stack.  Currently these are marked as signal frames
so that GDB will unwind across them ok when you have a page fault in the
kernel due to a NULL function pointer (other frame unwinders stop unwinding
when they see a NULL pc, but signal frames can keep unwinding past those).

I gave a talk a couple of years ago at a BSD conference about GDB that
includes some kgdb slides.  You can see the slides at
https://people.freebsd.org/~jhb/papers/bsdcan/2016/slides.pdf if that is
helpful.

-- 
John Baldwin