cycle_thief.stp - Track IRQ's and Other Processes Stealing Cycles from a Task

Description:
  The cycle_thief.stp script instruments the scheduler and IRQ handler
  to determine which processes and interrupts are competing with the
  specified task for the cpu cycles. This script uses the '-c' or '-x'
  options to focus on a specific task. The script output the number of
  times the task migrates between processors, histograms showing the
  length of time on and off processor, lists of processes running while
  the task is off the processor, and the interrupts that occurred while
  the task was running.


$ stap cycle_thief.stp

task 0 migrated: 1

task 0 on processor (us):
 value |-------------------------------------------------- count
     2 |                                                    0
     4 |                                                    0
     8 |@                                                   1
    16 |                                                    0
    32 |                                                    0
    64 |                                                    0
   128 |@@@@@@@@                                            8
   256 |@                                                   1
   512 |                                                    0
  1024 |@                                                   1
  2048 |@@@@@@@@                                            8
  4096 |@@@@@@@@@@@                                        11
  8192 |@@                                                  2
 16384 |@@                                                  2
 32768 |@@@@@@@@@@@@                                       12
 65536 |@@@@@                                               5
131072 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@                     30
262144 |                                                    0
524288 |                                                    0


task 0 off processor (us)
value |-------------------------------------------------- count
    1 |                                                    0
    2 |                                                    0
    4 |@@@@                                                4
    8 |@@@@@@@@@@@@@@@@@@@@@@@@@@@                        27
   16 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@                 34
   32 |@@@@                                                4
   64 |@@@@@                                               5
  128 |@@@@@                                               5
  256 |                                                    0
  512 |@                                                   1
 1024 |                                                    0
 2048 |                                                    0


other pids taking processor from task 0
   PID      count command
 25145         33 stapio
 19189         17 kworker/0:3
     7         13 rcu_sched
 22717          9 kworker/1:1
   380          8 jbd2/dm-0-8
     9          8 rcuos/1
   570          6 kworker/0:1H
     8          3 rcuos/0
   478          3 systemd-journal
 24747          3 kworker/0:1
   491          2 systemd-udevd
 19120          2 sshd
 25142          2 stapio
   615          1 alsactl
   625          1 irqbalance
 25144          1 stapio
 31816          1 kworker/u4:1
 25141          1 stap
   724          1 NetworkManager
 25143          1 systemd-udevd

irq taking processor from task 0
   irq      count    min(us)    avg(us)    max(us)
    44          9          3          8         19

The output from running the script for ~10 seconds.

The histograms provide a visual of the time a process spent on, or off. The
sampled processes are listed below the histograms to give a sense of what
was running at the time.

As mentioned in the description, the script can be focused on a certain process
with -c/-x.

$ stap cycle_thief.stp -T 10

task 29018 migrated: 2

task 29018 on processor (us):
value |-------------------------------------------------- count
    2 |                                                   0
    4 |                                                   0
    8 |@                                                  1
   16 |                                                   0
   32 |@                                                  1
   64 |@                                                  1
  128 |                                                   0
  256 |@                                                  1
  512 |                                                   0
 1024 |                                                   0


task 29018 off processor (us)
   value |-------------------------------------------------- count
       2 |                                                   0
       4 |                                                   0
       8 |@                                                  1
      16 |@                                                  1
      32 |                                                   0
      64 |                                                   0
         ~
 2097152 |                                                   0
 4194304 |                                                   0
 8388608 |@                                                  1
16777216 |                                                   0
33554432 |                                                   0


other pids taking processor from task 29018
   PID      count command
     0         41 swapper/0
 19189         24 kworker/0:3
     7         12 rcu_sched
     8          8 rcuos/0
 29002          5 kworker/0:0
   380          4 jbd2/dm-0-8
   570          3 kworker/0:1H
     6          3 kworker/u4:0
   491          2 systemd-udevd
 29016          2 stapio
 29017          1 systemd-udevd
   478          1 systemd-journal
   625          1 irqbalance
    14          1 migration/1

irq taking processor from task 29018
   irq      count    min(us)    avg(us)    max(us)