Using systemtap on MPI applications

[Frank Ch. Eigler]

Hi -

"Olausson, Bjoern"  wrote:

> [...]
> I am curious if there is a smart way to trace (basically IO) for MPI
> applications running on multiple nodes?
>
> I guess it would be possible to either run stap globally or run
> "mpirun <options> stap script.stp -c mpi-application"
> [...]

That would be the brute-force method.  It would require installing the
compiler etc. on all the hosts, unless you run a central stap-server
instance to do the compilation part of the work (passes 1-4).

Another possibility now is to use "stap --remote HOST1 --remote HOST2
..."  from a central box, which internally does "ssh HOST1 stapsh" to
maintain a two-way link, and perform remote execution (pass 5).


It would be nice if stap --remote learned about mpi (openmpi?), so as
to use mpirun or similar to manage remote startup of stapsh and
multiplex stdin/stdout/stderr communications with all the hosts in your
hostfile:
% stap --remote mpi:/path/to/hostfile

Or even
% stap --remote mpirun:HOST1 --remote mpirun:HOST2
may be worth doing, using individual "mpirun -H" jobs per host.


- FChE