Script to look for system call errors

[Eugene Teo]

Hi all,

On Thu, Apr 2, 2009 at 5:50 PM, Mark Wielaard <mjw@redhat.com> wrote:
> On Wed, 2009-04-01 at 22:18 +0800, Eugene Teo wrote:
>> Last night I was working on a script to look for system call errors.
>> This is particularly useful for identifying misbehaving userspace
>> applications. The script prints out useful information about the top 20
>> errors every 5 secs.
>>
>> With the script, I found out that mixer_applet2 wakes up very
>> frequently[1], gnome-panel tries to add a collection of watches every
>> second for files that do not exist, hald tries to open some
>> battery-related sysfs files over and over again even though I am running
>> the script in a virtual machine. mjw has a couple of interesting
>> observations with the script as well.
>
> Yeah, good script! It would be nice if you could set the timeout and
> number of output lines. Some observations that immediately sprung up
> from using it:

Thanks for the feedback!

The return probes are shortened. You can now set the timeout and
number of output lines. By default, timeout is 5 secs, and the number
of output lines is 20.

The script depends on a couple of functions that were committed
recently - http://is.gd/qylL and http://is.gd/qylX

You can run the script in the following ways:
$ ./errsnoop.stp
or
$ ./errsnoop.stp 1 25
or
$ ./errsnoop.stp -DMAXMAPENTRIES=20480 1 25

Thanks, Eugene
-------------- next part --------------
          SYSCALL         PROCESS   PID HITS ERRSTR       ARGSTR
             read gnome-power-man  5134    4  11 (EAGAIN) 3, 0x08ba044c, 4096
            ioctl hald-addon-stor  1677    3   1 (EPERM)  5, 21286, 0x7fffffff
             open hald-addon-stor  1677    3   5 (EIO)    "/dev/sr0", O_RDONLY|O_EXCL|O_LARGEFILE|O_NONBLOCK
           select            sshd  2233    2   1 (EPERM)  11, 0x01e3c8e8, 0x01e3c8d8, 0x00000000, NULL
          recvmsg            ntpd  1824    1  11 (EAGAIN) 22, 0xbfd94b18, 
             read            Xorg  5081    1  11 (EAGAIN) 22, 0x0a165d98, 4096
             read gdm-simple-gree  5148    1  11 (EAGAIN) 14, 0x094cb80c, 4096
inotify_add_watch gdm-simple-gree  5148    1  13 (EACCES) 19, "/home/test", 16789454
           select gnome-power-man  5134    1   1 (EPERM)  4, 0xbf87ff9c, 0xbf87ff1c, 0x00000000, NULL
           select gnome-power-man  5134    1   1 (EPERM)  4, 0xbf87ff2c, 0xbf87feac, 0x00000000, NULL
           select            Xorg  5081    1   1 (EPERM)  256, 0x0821c420, 0x00000000, 0x00000000, [77.344000]
             read        sendmail  1872    1  27 (EFBIG)  6, 0x021d5550, 1024
          recvmsg            ntpd  1824    1  48 (ELNRNG) 22, 0xbfd94b18, 
             read            Xorg  5081    1   4 (EINTR)  22, 0x0a165d98, 4096
             open        sendmail  1872    1   6 (ENXIO)  "/proc/loadavg", O_RDONLY
           writev            Xorg  5081    1  32 (EPIPE)  22, 0xbfa614a4, 1
           select            ntpd  1824    1   1 (EPERM)  23, 0xbfd94a98, 0x00000000, 0x00000000, [0.000000]
           select            ntpd  1824    1   1 (EPERM)  23, 0xbfd9508c, 0x00000000, 0x00000000, [39.998485]
             read gnome-power-man  5134    1  32 (EPIPE)  3, 0x08ba044c, 4096
           sendto            ntpd  1824    1  48 (ELNRNG) 22, 0xbfd94a30, 48, , {AF_INET, 203.116.x.y, 123}, 16
-------------- next part --------------
A non-text attachment was scrubbed...
Name: errsnoop.stp
Type: application/octet-stream
Size: 1101 bytes
Desc: not available
URL: <http://sourceware.org/pipermail/systemtap/attachments/20090403/b0ed054b/attachment.obj>