Tools to debug multiple cores/processes at the same time?

[Andrew Burgess]

* Paul Smith via Gdb <gdb@sourceware.org> [2021-08-18 15:55:57 -0400]:

> In my environment I'm working on a distributed system where the same
> process is running on a number of different systems.  The way these
> processes work is that they should all have (approximately) the same
> set of in-memory content at the same time.
> 
> Our test environment is set up so that if something "bad" happens, we
> SIGABRT all the processes on all the systems, so I often end up with 4,
> 5, 6, or more core files, all from the same process, with approximately
> the same in-memory content (obviously not identical addresses but the
> same structure) taken at about the same time.
> 
> The host systems can be assumed, at least initially, to be identical so
> no issue with different system libraries etc.
> 
> What would be really useful is for me to be able to open all these
> cores at the same time, controlled with a single UI (CLI prompt is
> fine), and be able to both (a) run commands that manipulate a single
> core, and also (b) run commands which manipulate all core files at
> once.
> 
> That would include, for example, setting convenience variables locally
> for each core, but then running the same command that might use those
> variables on every core and getting back the results.
> 
> I'm not really sure how this might work best: maybe a separate terminal
> per core file and one "controlling" terminal that I would enter
> commands into, or even commands run one after the other in the same
> terminal with some kind of header.
> 
> Don't know.  I can imagine multiple ways it might work.
> 
> What I'm really wondering is, has anyone heard of something like this,
> or is there any support (even partial support) for something like this
> in GDB today?  I get that it seems like a somewhat specialized request.
> 
> Debugging multiple live processes at the same time would also be great,
> of course.

I'm going to suggest something a little different.

Sometimes I need to debug the same thing running in two "almost"
identical setups, where in one case something will go wrong, and in
another everything's fine.  I start up two GDB sessions, and want to
send the same commands to each GDB until I find where the two systems
start to diverge.

I make use of a terminal call Terminator[1], this terminal allows me
to place multiple terminals into a single "group" and broadcast
keystrokes to every terminal in that group.  I can always pull a
terminal out of the group and interact with one session alone, then
place it back into the group, so long as I leave it in a consistent
state with the other group members I can continue to debug them all in
sync.

I'm sure there are other terminals that must offer this functionality.

The suggestions from others for multi-target and/or custom python
commands are all good too, but that above approach has always been
good enough for my needs, and might be of use to you.

Thanks,
Andrew

[1] https://terminator-gtk3.readthedocs.io/en/latest/