Tools to debug multiple cores/processes at the same time?

[Philippe Waroquiers]

On Wed, 2021-08-18 at 17:09 -0400, Paul Smith via Gdb wrote:
> On Wed, 2021-08-18 at 16:53 -0400, Paul Koning wrote:
> > There is multiple target support, which I haven't had a chance to try
> > yet but from what I remember reading is a recently added GDB
> > feature.  It should handle your case plus even stranger ones, such as
> > multiple processes running different code, or even running on
> > different instruction sets.  It's just what you need for debugging
> > distributed applications.
> 
> Yeah, I saw that, and it might be the kernel of something that could
> support what I'm talking about, except I don't think it manages
> different "convenience variable namespaces" per inferior.
> 
> But the main thing is it doesn't seem like it supports running the same
> command in all inferiors, other than by manually setting each one and
> running the command.  Maybe some kind of embedded Python facility could
> be created to do this, such as John mentions... I'm not sure what the
> python support is currently for multiple inferiors.
> 
> Let me give an example of a debugging session:
> 
> * start a session with 3 core files
> 
> * run a command that shows info about each core (assume a python
> "showinfo" command that show all threads with some details about each
> one).  Maybe a command like:
>   (gdb) all: showinfo
>   i1: <info>
>   i2: <info>
>   i3: <info>
info threads should show the threads of all inferiors.

> 
> * For each core, go to a certain thread (of course the thread numbers
> will be different in each core):
>   (gdb) i1: thr 7
>   (gdb) i2: thr 9
>   (gdb) i3: thr 23
thread INFERIORNR.THREADNR
will switch to the listed thread.

> 
> * Run a command that returns some value and stores it in a convenience
> variable, on all cores:
>   (gdb) all: set $v = $findstuff()
> 
> * Run a command that shows some value, on all cores:
>   (gdb) all: p $v->someDetail
>   i1: $1 = 7
>   i2: $1 = 44
>   i3: $1 = 165
> 
> I dunno, something like that.  Alternatively maybe it would be simpler
> for output for each inferior to go to a separate terminal or whatever.
The command  'thread apply' can apply a command to all threads
or to a selected list of threads.
If you have multiple inferiors, the command can then be run for all threads
of all inferiors.
I do not see an (easy) way to run a command only once per inferior,
when an inferior has multiple threads.
For this, likely, the best would be to have a new gdb command such as
   inferior apply all | INFERIOR-LIST

Note that the command 'show convenience' lists a convenience variable $_inferior.

Philippe