Re: Is multiprocessor debugging multithreaded debugging?

[Ramana Radhakrishnan <ramana dot radhakrishnan at codito dot com>]

Hi Steven, 

> >
> I dont think "lock step" is achievable with Asynchronous 
> multiprocessing.  Each processor has a different clock domain, they also 
> do not have any "synchronizing" features in hardware.  The best you 
> could approximate is "command 2 targets to Single step at once" which 
> one actually single steps first would of course be unknown. And in fact 
> if one processor is faster than another the processor that actually 
> single stepped last, could be the first to complete its instruction.

Yes it is this approximation that I meant. If you wanted to define an
order of stepping that could be one extra feature. But you really cannot
have a granularity more than an instruction.


> 
> In my case, if i needed to do this, i worked out which instruction 
> should be executed first (for my particular test), went to that instance 
> of GDB, single stepped.  Went to the next instance of GDB and then 
> stepped it.  Given the asynchronous nature of what you describe i can 
> imagine all sorts of race conditions which would mean you would get 
> inconsistent debug results.  Things like, why did that happen, that bit 
> shouldnt have been set yet type stuff.
> 
> >Providing relational breakpoints between multiple processors might be
> >another nifty feature. 
> >
> >Lets say something like b <here> if val_in_pgm_in_proc2 is 0xcafebabe. 
> >  
> >
> I also think to do this GDB would need to "simulate" the behavior.  It 
> would have to put an unconditional break on <here>.  Every time the 
> target breaks on <here> break the processor with the val_in_pgm_in_proc2 
> variable, inspect it. restart the proc2 target (which might then 
> actually change the value from 0xcafebabe, so you are not in the state 
> you think you are) and then make a decision about whether to continue 
> from the break at <here> or report it to the user.  Ive yet to see a 
> JTAG of BDM type interface that allows "live" inspection of data, while 
> the CPU is executing code.

Agreed ! Neither have I . 

> 
> I think all of these things are advanced User Interface concerns, and 
> actually have little if anything to do with the fundamental GDB itself.  
> Maybe what is really required, if anything, is a GDB front end, that can 
> interface to multiple GDB back end instances (through MI), each one a 
> potentially different processor architecture, and do these types of 
> operations.  Tabbed pages for each CPU, simulated simultaneous step on n 
> of m processors, cross processor break trigger conditioning.  Sounds 
> like a bitching front end, but i dont think GDB itself should be made to 
> do these things natively, as they are not actually native debug issues, 
> but simulated "pseudo" operations.


User Interfaces are a concern and if one could not use these features
from the command line or emacs for that matter (which happens to be my
favourite GDB frontend.)  . 

> 
> Now if such a front end was GPL, could work on the majority of hosts 
> that support GDB, and was assigned to the FSF it might (if the 
> maintainers agree) even be able to live in the GDB CVS Tree (like the 
> TUI front end), and be a standard example of an MI/GUI front end. (But 
> now im dreaming).  Id shudder to think how unwieldy multi CPU debugging 
> would be without multiple "screens" of information, you can switch 
> between, so i really dont know how you would achieve anything really 
> useful without a GUI in any event.

I agree about the UI design part, a per CPU command or a per CPU view /
a process command and viewing registers seamlessly depending on the
context of the debuggee under consideration would be a nifty feature to
have. 

A set context to cpu#x would be another command. The problem with
frontends again is that users have different choices of frontends, so to
get people to migrate from one to another is another set of issues. And
again keeping it in gdb allows users to continue using their frontends
or customizing it for features in gdb  . Also it allows users to
seamlessly continue using their existing features. 

> 
> In fact what you describe is really no different to debugging two 
> applications on the one PC, which interact, at the same time.  Again, 
> something people do not infrequently, but they do it with multiple 
> instances of the debugger running.

Agreed again ! What I have been interested in at some point in the past,
was designing a unified frontend and possibly looking at an interface
within GDB to do such stuff. We did do some prototyping with exactly
this kind of a setup where we could look at multiple binaries within the
same session and switch between processes and so on and so forth.  


> 
> Maybe if Anupama can highlight what he thinks the deficiencies with the 
> current arrangement are, i could see the problems with my view.

Its a she actually ! 

> 
> Now if you were talking about getting GDB to be able to read/set JTAG 
> boundary scan information (on targets where that is applicable), and 
> process BSDL files to show the states of all pins in a meaningful way I 
> think that would be a useful project in this realm, but i think making 
> GDB natively handle asynchronous multiprocessor debugging is a waste of 
> time that could be better spent, as there is no problem to solve that i 
> can see.

If I understand this right you are talking about reading and setting
JTAG boundary scan info , processing BSDL regarding the design for the
particular board under consideration and then debug the model using that
for the program being debugged and relate the two up.Map between test
program source and the state of the pins . 

Care to elaborate this further ? If it is not really relevant to
discussion on gdb@ maybe we can carry this conversation offlist ? 

cheers
Ramana

> 
> Steven
> 
> >My 2 bits on the topic. 
> >
> >cheers
> >Ramana
> >
> >
> >
> >On Wed, 2005-09-28 at 21:16 +1100, Steven Johnson wrote:
> >  
> >
> >>Anupama Chandwani wrote:
> >>
> >>    
> >>
> >>>In continuation with my prev mail..
> >>>I want to extend gdb to debug homing ogenous multiprocessor system
> >>> 
> >>>
> >>>(say multiple ARM or x86 processors on single chip) by remote
> >>>debugging in a single session of gdb.
> >>>
> >>>What i want to know is are there enough applications being written on
> >>>such multi processors? Also are there different executables being
> >>>required to be debugged simultaneously? Coz this is what i want to
> >>>extend in further.. Each processor running a different executable so
> >>>the processors dont share memory & run with different images of code.
> >>> 
> >>>
> >>>      
> >>>
> >>This is commonly called "Asynchronous" Multi Processing.
> >>
> >>    
> >>
> >>>An application of such debugger could be while building an OS but that
> >>>wouldnt involve different executables.. So are there applications
> >>>requiring to run different executables on each processor? Say for
> >>>example a prog gives a certain bug on when there is certain other
> >>>program running on the other processor or something similar to
> >>>this....
> >>> 
> >>>
> >>>      
> >>>
> >>Yes in the embedded world, there are many examples of Asynchronous Multi 
> >>Processor designs.  They are by far the easiest multi processor design 
> >>to implement.  I for example have worked on a board that had 3 MSP430's, 
> >>each had a unique function, and they intercommunicated over a custom 
> >>parallel bus to coordinate their activities.  Worked sweet, had high 
> >>performance, and was really cheap.
> >>
> >>    
> >>
> >>>As far as i know this done by multiplexing the JTAG interface (for
> >>>x86) &different sessions of gdb right now. Any other? And any flaws or
> >>>inconvenience with present methods?
> >>> 
> >>>
> >>>      
> >>>
> >>This is exactly how it is done, multiple sessions of GDB.  This, in my 
> >>opinion is the right way to go.  Not all Asynchronous multi processor 
> >>designs have homogeneous pprocessors (ie, you may have an MPC860 
> >>handling comms, and a MIPS Chip doing some number crunching.  1 is a 
> >>power PC, the other is a MIPS.  Both have different debug interfaces. 
> >>
> >>Now if you had a system say, where you had 3 MIPS Chips, hooked up on 
> >>the same EJTAG interface, you would need to handle that with some nifty 
> >>EJTAG code in your (pseudo) stub to ensure each device was uniquely 
> >>addressed and they didnt interfere with one another, so that you could 
> >>start up 3 GDB sessions to debug your 3 processors, but then it becomes 
> >>a problem for the stub. 
> >>
> >>What im saying is I dont think a single instance of GDB needs to be 
> >>complicated to try and debug multiple "tasks" simultaneously.  I dont 
> >>have any problems with running GDB as many times as I want.  For example 
> >>with the MSP430 example, I had (at various times) GDB running 5 times on 
> >>the one PC.  One was debugging a local PC app that talked to my MSP430 
> >>board.  3 were talking to the MSP430 board, the last was talking to yet 
> >>another device (that had an MPC862 as its processor), I just ran each in 
> >>a separate "Desktop" under KDE and then switched to the one i had to 
> >>deal with at the time.  No problems, worked easily.
> >>
> >>Hope that gives you insight into one application of what you discussed.
> >>Steven
> >>    
> >>