Re: runtime libraries application on schedule()

[William Cohen <wcohen at redhat dot com>]

Martin Hunt wrote:
> > Use a jprobe to instrument the entry to schedule().
> > Record the pid and the return address.
>
>
> pid is easy.  I don't know of a way to get the return address.
> I assume some inline assembly to get ebp or rbp would do it.  Something
> else to add to the runtime library.
>
>
> > The information could be analyzed in a sevral different ways:
> >
> > 1) sum based on return address to find out which schedule points are 
> > getting hit all the time
> > 2) sum the counts by pid to figure out which pids are getting 
> > rescheduled a lot
>
>
> That's easy enough.
>
> asmlinkage void __sched inst_schedule(void)
> {
>   _stp_map_key_long (schedpid, current->pid);
>   _stp_map_set_int64 (schedpid, _stp_map_get_int64(schedstr) + 1);
>
>   jprobe_return();
> }
>
> I'll attach full source to a working probe.
>
>
> > 3) find out pid hitting which schedule() points
> > 4) find out schedule() point being encountered by which process
>
>
> I don't understand what you mean by these.  What is a schedule point?
>
> Martin

I wanted to try out the runtime library and I wrote a subset of the 
scheduling instrumentation. Right now it just hashes on the return 
address, no pid. The attached tar.gz contains the code for the module.

I built it with:

make  -C /lib/modules/`uname -r`/build M=`pwd` modules \
STP_RUNTIME=/home/wcohen/dprobes/systemtap/src/runtime

I was able to install and remove the instrumentation. Below is the 
resulting output in /var/log/messages when the instrumentation was 
installed during a "du"

Mar 10 11:53:56 slingshot kernel: STP: kprobe for schedule registered
Mar 10 11:54:55 slingshot kernel: STP: kprobe unregistered
Mar 10 11:54:55 slingshot kernel: STP: schedule() called 39767 times.
Mar 10 11:54:55 slingshot kernel: STP: schedule point break down.
Mar 10 11:54:55 slingshot kernel: STP: 0xc0103f4a, 7931
Mar 10 11:54:55 slingshot kernel: STP: 0xc012180f, 1
Mar 10 11:54:55 slingshot kernel: STP: 0xc0301f95, 287
Mar 10 11:54:55 slingshot kernel: STP: 0xc0123fac, 4
Mar 10 11:54:55 slingshot kernel: STP: 0xc0130691, 850
Mar 10 11:54:55 slingshot kernel: STP: 0xc030200c, 7245
Mar 10 11:54:55 slingshot kernel: STP: 0xe0866ff5, 2
Mar 10 11:54:55 slingshot kernel: STP: 0xc0301ec0, 11023
Mar 10 11:54:55 slingshot kernel: STP: 0xc01020d0, 11219
Mar 10 11:54:55 slingshot kernel: STP: 0xe08668be, 13
Mar 10 11:54:55 slingshot kernel: STP: 0xc0145980, 17
Mar 10 11:54:55 slingshot kernel: STP: 0xc03017b4, 207
Mar 10 11:54:55 slingshot kernel: STP: 0xc0168eb5, 3
Mar 10 11:54:55 slingshot kernel: STP: 0xc012523a, 3
Mar 10 11:54:55 slingshot kernel: STP: 0xc0301dfe, 505
Mar 10 11:54:55 slingshot kernel: STP: 0xc011e049, 23
Mar 10 11:54:55 slingshot kernel: STP: 0xc014bde1, 127
Mar 10 11:54:55 slingshot kernel: STP: 0xc0126a7f, 280
Mar 10 11:54:55 slingshot kernel: STP: 0xc0301e52, 20
Mar 10 11:54:55 slingshot kernel: STP: 0xe0861d21, 5
Mar 10 11:54:55 slingshot kernel: STP: 0xc013409e, 1
Mar 10 11:54:55 slingshot kernel: STP: 0xc013bfe7, 1

Looking at where the addresses map to:

0xc0103f4a work_resched
0xc030200c schedule_timeout
0xc0301ec0 io_schedule
0xc01020d0 cpu_idle

The the 0xe0zzzzzz addresses map to modules, found by looking at 
/proc/modules. It looks like all of them map to jdb.

 $ grep 0xe086 /proc/modules
jbd 61592 1 ext3, Live 0xe0861000

-Will

Attachment: kprobe_schedule.tar.gz
Description: Unix tar archive