Example looking at the periodic timers

[William Cohen]

One case that people would like to use SystemTap for is finding out why some latency sensitive application is slowed down every x seconds.  They know that something keeps running on the machine and slowing things down, but they don't know what it is.

I have been playing around with the timer tracepoints and came up with the attached script, periodic.stp.  When the script exits it prints out a list of the periodic timers, sorted from most frequently to least frequently fired timer. It also include the average period for the timer.  The functions that have [dg] are things put into the delayed_work_queues.

Here is the example output:


$ ./periodic.stp -c "sleep 10"
WARNING: missing unwind/symbol data for module 'uprobes'
#timer            function                              period(us)      count
0xffffe8ffffc118a8 0xffffffffa0e38d30                          1000      10000
0xffffe8ffffc918a8 0xffffffffa0e38d30                          1000      10000
0xffffe8ffffd118a8 0xffffffffa0e38d30                           999      10000
0xffffe8ffffd918a8 0xffffffffa0e38d30                           999      10000
0xffff880028216fa8 do_dbs_timer[dq]                           10000        999
0xffffffffa115e960 0xffffffffa0e38c10                         10000        999
0xffff880028296fa8 do_dbs_timer[dq]                            9999        999
0xffff880028316fa8 do_dbs_timer[dq]                           10000        999
0xffff880028396fa8 do_dbs_timer[dq]                            9999        999
0xffffffffa115e9c0 0xffffffffa0e38b50                         19999        499
0xffff880130c9fe50 monitor_timeout                           200002         49
0xffffffff81ea3b20 clocksource_watchdog                      499993         19
0xffff880028212d40 vmstat_update[dq]                         999963          9
0xffff880132cf9318 br_fdb_cleanup                            999962          9
0xffff880028292d40 vmstat_update[dq]                         999999          9
0xffff880028312d40 vmstat_update[dq]                        1000000          9
0xffff880028392d40 vmstat_update[dq]                        1000002          9
0xffff8801303c96c0 i915_gem_retire_work_handler[dq]         1000001          9
0xffff88013156e128 flush_to_ldisc[dq]                        113108          8
0xffff880130315928 flush_to_ldisc[dq]                        180399          5
0xffff8801303c97e8 intel_gpu_idle_timer                     1607001          5
0xffff880028212de0 cache_reap[dq]                           1999911          4
0xffff880132cf9228 br_hello_timer_expired                   2000003          4
0xffff880028292de0 cache_reap[dq]                           2000002          4
0xffff8801336f06e0 e1000_watchdog                           2000000          4
0xffff880028312de0 cache_reap[dq]                           1999994          4
0xffff880028392de0 cache_reap[dq]                           2000000          4
0xffff880131767e18 cupsd(10171)                             1500570          4
0xffff880130303540 intel_crtc_idle_timer                    3658003          2
0xffff8801303ca540 intel_crtc_idle_timer                    3658001          2
0xffff880134d2fdf0 bdi-default(29)                          5000032          1
0xffff8801336f0358 dev_watchdog                             5000050          1
0xffffffff81fab180 sync_supers_timer_fn                     5999999          1
0xffff88012e1b0b58 cfq_idle_slice_timer                     4152994          1
0xffff880130161dc0 ips-adjust(874)                          4999998          1
0xffff880130c21dc0 flush-253:0(1347)                        5000016          1
0xffff880130c4fdc0 flush-253:4(1348)                        5000015          1

What do people think of the script?  Any suggestion on improvements before checking it into the examples?

-Will


-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: periodic.stp
URL: <http://sourceware.org/pipermail/systemtap/attachments/20111213/848a969b/attachment.ksh>