Re: Towards better x86 system debugging support

[Jan Kiszka <jan dot kiszka at web dot de>]

Mark Kettenis wrote:
>> Date: Sun, 04 Jan 2009 14:34:00 +0100
>> From: Jan Kiszka <jan.kiszka@web.de>
>>
>> Hi,
>>
>> Unfortunately, the x86 support is incomplete in so far that neither the
>> gdb remote protocol nor the gdb backend are aware of most special
>> registers x86 system-level software uses. This comes with several drawbacks:
> 
> Yes there are some limitations.  What's there is mostly adequeate for
> debugging traditional C kernel code.  And I for myself have always
> considered implementing the bits needed for more low-level code an
> effort too big to be worth it (how much 16-bit code is there in the
> Linux kernel?).  But if you think otherwise, feel free to contribute
> the appropriate code.

Virtualization changed this picture a bit, specifically as some VMs like
QEMU allow guest debugging. There are a lot of legacy OSes around, some
of them use non-flat memory models, and some of them (at least ours) now
run inside a VM. Then, as long as x86 CPUs start in real mode, you may
want to debug boot loaders. And finally you may get the idea to debug
the VM itself while it runs legacy code (=> nested VMs - the next
logical step might be to teach VT-x/AMD-V awareness to gdb...).

> 
>>  o Current code bit width (16, 32 or 64) is unknown to the debugger,
>>    so correct disassembling is not automatically possible
> 
> It's tricky to figure this out when running on real hardware, but I
> suppose it would be trivial for an emulator like QEMU to tell GDB what
> mode it's in.

On real hw you need support by the target OS. Think of kernel debuggers
(kgdb...) - they typically run in privileged context along their OS and
have access to most information (except for the shadow segment register
states).

>  The idea has always been that GDB would become
> multi-arch and that every frame would have an architecture associated
> with it.  Perhaps that's the way to go.

Yes, would be nice. But I'm primarily looking for an iterative path, and
this feature sounds a bit like it could come later on that way.

> 
>>  o Real mode cannot be detected, which would include setting 16 bit
>>    disassembly mode and calculating segment bases appropriately
> 
> Sorry, what are you trying to say here?

gdb assumes protected mode so far. There is some old patch, I'm using it
in some more lengthy real mode sessions, which adds "set i386 real-mode
on" to enable at least proper code address translation + you have to
"set arch i8086". If we could detect CR0.PE=0, we would be able to
switch such things automatically (given non-flat addressing support as
well, of course).

> 
>>  o Manually setting the architecture (set arch i8086/i386/i386:x86-64)
>>    influences the register set layout of the remote protocol, preventing
>>    straightforward switches from 16-bit bootloader/BIOS code to 64-bit
>>    kernel code (to give just one example)
> 
> Even worse, it changes the layout of GDB's internal register cache.
> And I'm not even sure it is feasable to unify the i386 and amd64
> layouts in a meaningful way.

/me wonders how other archs with "legacy" modes handle this. Is x86
special here? And if so, why? Historically?

> 
>>  o Only flat memory models are supported and debugging becomes very
>>    hairy when some segment uses a non-zero base address - note that this
>>    also prevents support for TLS variable lookup (which is GS or
>>    FS-based)
> 
> This problem is not unique to the i386/amd64 targets.  There are other
> architectures that have more than one address space.  Unfortunately, I
> believe nobody has really implemented a satisfactory solution to
> implement them properly in GDB.

For x86, a colleague implemented an extended address resolution via
<segment>:<offset>. Works in practice but we likely need to
rebase/rework this once the register set extension is done.

> 
> Not that the TLS problem has been solved already (in a satisfactory
> way I think).  I'm not sure making GDB aware of %gs and %fs would
> really help here.  Systems use those registers in different ways, and
> they don't really help when you want to look at variables in a
> different thread than the one you're running isn't it?

I was thinking of system level debugging here where you don't have OS
support. But you are right, teaching the TLS (or per-CPU) lookup
algorithms to gdb might not be a good idea. Maybe some helper gdb
scripts would be better (given they can do things like "x $fs:0").

> 
>> As a first step toward enhanced x86 support, I think there is a need for
>> an extended register set in the remote protocol. The following registers
>> should be added:
> 
> Sorry, but I think you're starting at the wrong end here.  You should
> probably first add support for additional registers to GDB's internal
> register cache.  It doen't make sense to worry about the remote
> protocol until that's done.

Yes and no. Due to the fact that several projects are affected (at least
gdb and qemu, maybe also kgdb) and that we already have temporary
patches for many use cases, I'm looking for a smooth, iterative
realization path. Just having extended registers won't help as much as
already have a source (e.g. QEMU's gdbstub) for retrieving them from
real targets.

On the other hand, I now see that an earlier step is teaching gdb to use
non-static target register descriptions that would come via xml and that
this way the remote protocol is already defined.

> 
>>  o GDTR, LDTR, IDTR, TR (visible part, ie. selector value)
>>  o CR0..4
>>  o DR0..7
>>  o selected MSRs, at least
>>     - IA32_EFER (64-bit mode detection)
>>     - IA32_FS_Base (TLS)
>>     - IA32_GS_Base (TLS)
>>     - IA32_KernelGSbase (TLS)
>>  o Shadow states of segment registers, GDTR, LDTR, IDTR and TR
>>    (relevant for virtual targets where the VM often has access to these
>>    hidden states, helpful when debugging targets that modify in-use
>>    descriptor table entries)
>>
>> If anyone thinks that there should be more registers or MSRs included,
>> please extend this list!
> 
> I don't think it is a good idea to start adding MSRs to GDB's internal
> register cache.  There are zillions of them.  It is probably a better
> idea to implement them as a seperate address space.

Sounds reasonable - you wouldn't have to decide which ones to include or
to add later on. But how would I access this address space, specifically
remotely?

BTW, this approach would also be interesting to add the PIO address
space to gdb (we are currently doing this via maintenance packets that
issue QEMU monitor commands to access I/O ports...).

> 
> One thing that's probably a good idea before starting hacking on all
> this is to fully decouple the layout of the internal register cache
> from the remote protocol such that when you change things there, you
> don't have to worry about breaking existing remote stubs.

Mmm, makes sense. Will dig into this.

Jan

Attachment: signature.asc
Description: OpenPGP digital signature