Multi-process GDB
This page describes the work to add multi-process support to GDB.
GDB HEAD already contains basic support for multi-process inferior control, and implements that in the remote target, but it can only be used to debug systems that share code across all inferiors. This project aims to extend that work by providing support for multi-process and multi-executable debugging for linux systems as well. This means being able to load several programs under a single GDB session; "run" or "attach" to several processes under a single GDB session, each possibly running a different program; or following all the forks and execs of an inferior and its children tree.
Getting the code / Helping
Discussions are held on the main GDB mailing list. Patches should be posted to the gdb-patches mailing list. Development is currently taking place on the mainline.
The original code for this effort was located in the multiprocess-20081120-branch branch in the CVS repository. It prototyped the specification posted here: specification. This is loselly based on Parallel Tools Consortium's High Performance Debugger Forum's Command Interface for Parallel Debugging (pdf file) (Old web site http://www.ptools.org/hpdf/).
During the 2009 the code in this branch become obsolete, the gdbserver work was merged into the trunk, just before the release only gdb multiexec work is still pending. The recent development is done in a set of patches, see http://sourceware.org/ml/gdb-patches/2009-07/msg00134.html.
Things found on the branch, not on mainline yet
- GDB can load multiple executables; symbol lookup and breakpoints take those into account. See the 'info inferiors' and 'focus' commands.
- experimental follow-fork support added to the remote target (set follow-fork-mode/set detach-on-fork).
- base support for itsets.
To try out the multi-exec support, start GDB with several executables listed on the command line. You need to connect to gdbserver with 'target extended-remote' to run several processes simultaneously.
Running multiple processes simultaneously with gdbserver
Here's a small example, spawning several instances of the same executable.
Launch gdbserver in a separate terminal:
./gdbserver --multi :9999
Startup GDB, and load an example executable:
./gdb (gdb) file /home/pedro/gdb/tests/threads Reading symbols from /home/pedro/gdb/tests/threads...done.
Notice that executable files are listed as inferiors:
(gdb) info inferiors 1 0 #threads# /home/pedro/gdb/tests/threads
Let's insert a breakpoint, and set it running. Notice that when debugging against a remote stub that supports "run", we have to separately tell it what to run.
(gdb) b main Breakpoint 1 at 0x400640: file threads.c, line 35. (gdb) set remote exec-file /home/pedro/gdb/tests/threads (gdb) tar extended-remote :9999 Remote debugging using :9999 (gdb) run Starting program: /home/pedro/gdb/tests/threads Breakpoint 1, main () at threads.c:35 35 long i = 0;
You can issue more runs:
(gdb) r Starting program: /home/pedro/gdb/tests/threads [Switching to Thread 18132.18132] Breakpoint 1, main () at threads.c:35 35 long i = 0;
The new inferior processes have been assigned internal inferior GDB ids, distinct from the executable inferior id:
(gdb) info inferiors 3 18135 threads * 2 18132 threads 1 0 #threads# /home/pedro/gdb/tests/threads
The thread list holds all threads of all inferiors:
(gdb) info threads 2 Thread 18135.18135 0x00007fd146ec2990 in _dl_debug_state () from /lib64/ld-linux-x86-64.so.2 * 1 Thread 18132.18132 main () at threads.c:35 (gdb)
Each inferior got a copy of the breakpoint location, so you can disable individual locations.
(gdb) info breakpoints
Num Type Disp Enb Address What
1 breakpoint keep y <MULTIPLE> inf 1
breakpoint already hit 2 times
1.1 y 0x0000000000400640 in main at threads.c:35 inf 1
1.2 y 0x0000000000400640 in main at threads.c:35 inf 3
1.3 y 0x0000000000400640 in main at threads.c:35 inf 2
branch TODO (some of this is out of date for mainline)
- target stack design decisions
struct target_ops is overloaded. Split further into target interface and target inferior. The former should stay in struct target_ops, and represents access to a system or debug interface, e.g., ptrace, executable, core, win32 debug api, threads library, etc. The latter represents an inferior under a target interface control, and is coded by the new `struct inferior' structure. For now we're sticking to a single target stack. target_has_execution, target_has_stack, target_has_FOO should perhaps be an inferior property. The `tmp_inf' hack should go away: Give each executable a target side id? E.g., add it a 'target' field, so an exec has a unique ptid as well. Or, when the current inferior_ptid points to null_ptid, make current_inferior return the current executable?
- qOffsets: Breakpoint addresses are cloned to all inferiors using the same exec. This doesn't take into account targets that relocate sections/segments (see 'qOffsets' in the remote serial protocol). E.g., on uClinux (to be confirmed). This also related to target_ops.to_sections (as in, they should go away: update, to_sections have been removed in mainline).
- Shared libraries per inferior need to be implemented. GDB's current shared library support is agnostic to multiple inferiors.
- The current linux multi-forks support should be generalized, and split from the checkpoints support. The checkpoints support should be generalized as well --- the fact that it's implemented on top of multi-forks on linux is an implementation detail.
- Here's a suggested generalization of the commands:
current command
new command
delete fork NUM
kill inferior NUM
detach fork NUM
detach inferior NUM
restart NUM
keep, but apply only to checkpoints
checkpoint
keep, adds a new checkpoint. make follow fork child/parent have no effect on this command (always follow the parent)
info forks
delete, replaced by the generic "info inferiors"
fork NUM
delete, replaced by the generic 'focus NUM', or by a new 'inferior NUM'
- Here's a suggested generalization of the commands:
- Breakpoints in the single-process case output differently, which may or not be confusing.
- Mourning/deleting shared library and/or thread event breakpoints needs work.
- Thread event breakpoints also need tweaking to work correctly.
- This is related to the blind cloning of breakpoint locations to all inferiors, and, to shared libraries per-inferior. Even if all inferiors loaded the thread event breakpoints at the same address, we can't insert a thread event breakpoint until the threads library is loaded in the new inferior.
Here's the symptom, when starting a second instance of the same program:
(gdb) start Temporary breakpoint 2 at 0x400640: file threads.c, line 35. Starting program: /home/pedro/gdb/tests/threads Warning: Cannot insert breakpoint -2. Error accessing memory address 0x7ffff7bc9790: Input/output error. Cannot insert breakpoint -3. Error accessing memory address 0x7ffff7bc97a0: Input/output error. (gdb) maint info breakpoints Num Type Disp Enb Address What -2 thread events keep y <MULTIPLE> inf 0 -2.1 y 0x00007ffff7bc9790 inf 0 -2.2 y 0x00007ffff7bc9790 inf 2 -2.3 y 0x00007ffff7bc9790 inf 3 -3 thread events keep y <MULTIPLE> inf 0 -3.1 y 0x00007ffff7bc97a0 inf 0 -3.2 y 0x00007ffff7bc97a0 inf 2 -3.3 y 0x00007ffff7bc97a0 inf 3 2 breakpoint del y <MULTIPLE> inf 1 2.1 y 0x0000000000400640 in main at threads.c:35 inf 1 2.2 y 0x0000000000400640 in main at threads.c:35 inf 3 2.3 y 0x0000000000400640 in main at threads.c:35 inf 2 -4 shlib events keep y <MULTIPLE> inf 0 -4.1 y 0x00007ffff7dee990 <_dl_debug_state> inf 0 -4.2 y 0x00007ffff7dee990 <_dl_debug_state> inf 3 -4.3 y 0x00007ffff7dee990 <_dl_debug_state> inf 2 (gdb)
The simplest solution would be to make thread event breakpoints inferior specific, so they aren't cloned across inferiors using the same exec.
- breakpoints module:
- always inserted-mode mode isn't aware or multiple inferiors yet.
- breakpoint_init_inferior (called from generic_mourn_inferior and from init_wait_for_inferior), delete breakpoints too blindly. It should care to delete locations or inferior specific breakpoints of the inferior that exited, but leave the others in.
- breakpoint locations are tied to an inferior, but, this is only correct when each inferior has its own address space. We should have an address space object/entity: breakpoint locations, shared libraries and exec files should keep a reference to an address space, instead of a pointer to an exec or inferior. On most targets, there'll be an address space object per-inferior, but e.g., in DICOS, a single breakpoint is visible to all processes, and shared libraries too. E.g., notice that check_duplicates_for accounts for duplicates only if the inferiors match --- this is wrong on DICOS. In the DICOS case, we'd have a single address-space object shared between inferiors.
- following an exec needs a bit more thought and implementation work to fit into the single-breakpoint multiple-locations model 100%. e.g.,
(gdb) b main; run; run;...
(gdb) info inferiors
* 3 2311 foll-exec
2 2308 foll-exec
1 0 #foll-exec# /home/pedro/gdb/fsf_multi2/build/gdb/testsuite/gdb.base/foll-exec
(gdb) info breakpoints
Num Type Disp Enb Address What
1 breakpoint keep y <MULTIPLE> inf 1
breakpoint already hit 2 times
1.1 y 0x00000000004005c0 in main
at ../../../src/gdb/testsuite/gdb.base/foll-exec.c:15 inf 1
1.2 y 0x00000000004005c0 in main
at ../../../src/gdb/testsuite/gdb.base/foll-exec.c:15 inf 3
1.3 y 0x00000000004005c0 in main
at ../../../src/gdb/testsuite/gdb.base/foll-exec.c:15 inf 2Now, if we let inferior 3 run, and it execl's, GDB should try to reset breakpoints in its the new executable image. Symbolic breakpoints like "main", should be re-evaluated in the context of the new exec. inferior 2 is also using breakpoint 1 (through location 1.3), which means that we can't reset the whole breakpoint. We would have to reset only breakpoint location 1.2. Fixing this probably also touches the mechanism where we blindly clone breakpoint locations to all inferiors sharing the same exec (clone_breakpoint_location). This brings us to another issue:
- breakpoints continued...
- It isn't currently possible to have "break main" (or rbreak) create a breakpoint that is hit in every instance of "main", in all executables loaded. Making this work also probably touches the mechanism where we blindly clone breakpoint locations to all inferiors sharing the same exec (clone_breakpoint_location). It should also be possible to have "break foo.c:123" insert in all possible locations -- consider that foo.c is part of a shared library that is loaded by all different programs you're trying to debug. It may be clearer to imagine that you're debugging in an IDE, and try to insert a breakpoint in the file you're editing, at current cursor position --- you don't really care which inferior will trigger the breakpoint.
- inferior control:
- most of what init_wait_for_inferior does is wrong in a multi-process environment, especially in a non-stop + multi-process environment.
- Linux native support.
- gdbserver/remote.
- - support for following forks
- Prototyped in the branch. Worked needed:
- - needs rethinking to see if we should make it work without the multi-process extensions. - modeling vfork-done in the protocol is missing. - try to see if it is possible to provide a higher level abstraction.
- prototyped in the branch.
- Prototyped in the branch. Worked needed:
- - support for following forks
- Current multi-process inferior control in GDB HEAD wants to resume all inferiors by default. The old multi-fork support only resumed a single inferior at a time by default. This is a conflict: We either have to make it resume a single inferior by default, with a switch similar to 'set scheduler-locking' (suggestion 'set schedule-multiple on/off'), or, really implement itset support in inferior control side, or, enrich the resumption commands to take further options (continue, step, etc.). MI also has to be considered.