Script to look for system call errors

[Mark Wielaard]

Hi Eugene,

On Wed, 2009-04-01 at 22:18 +0800, Eugene Teo wrote:
> Last night I was working on a script to look for system call errors.
> This is particularly useful for identifying misbehaving userspace
> applications. The script prints out useful information about the top 20
> errors every 5 secs.
> 
> With the script, I found out that mixer_applet2 wakes up very
> frequently[1], gnome-panel tries to add a collection of watches every
> second for files that do not exist, hald tries to open some
> battery-related sysfs files over and over again even though I am running
> the script in a virtual machine. mjw has a couple of interesting
> observations with the script as well.

Yeah, good script! It would be nice if you could set the timeout and
number of output lines. Some observations that immediately sprung up
from using it:

- I am using a XMCDP session and gnome-screensaver continuously seems to
do non-blocking reading to the server:
sys_read  gnome-screensav  2048   870  11 (EAGAIN) 3, 0x00000000020c5504, 4096

- The gnome panel issue you already mentioned, every second I see:
sys_inotify_add_watch      gnome-panel  2049     1   2 (ENOENT) 18, "/home/mark/.config/menus/settings-merged", 16789454
sys_inotify_add_watch      gnome-panel  2049     1   2 (ENOENT) 18, "/home/mark/.config/menus/preferences-merged", 16789454
sys_inotify_add_watch      gnome-panel  2049     1   2 (ENOENT) 18, "/etc/xdg/menus/system-settings-merged", 16789454
sys_inotify_add_watch      gnome-panel  2049     1   2 (ENOENT) 18, "/home/mark/.config/menus/system-settings-merged", 16789454
sys_inotify_add_watch      gnome-panel  2049     1   2 (ENOENT) 18, "/etc/xdg/menus/server-settings-merged", 16789454

- The multiload-applet seems to use some ioctl and stat on my xen block
device and lvm volume device those don't support/exist, but it keeps
trying anyway up to 30 times per second:
sys_ioctl  multiload-apple  2127    30  95 (EOPNOTSUPP) 19, 35585, 0x00007fff8d47f810
sys_ioctl  multiload-apple  2127     6  99 (EADDRNOTAVAIL) 19, 35093, 0x00007fff8d47f810
 sys_open  multiload-apple  2127     6   2 (ENOENT) "/sys/block/xvdb/xvdb1/stat", O_RDONLY
 sys_open  multiload-apple  2127     6   2 (ENOENT) "/sys/block/mapper/VolGroup/mapper/VolGroup00-LogVol00/stat", O_RDONLY

- During a window resize metacity continuously tries to load a
non-existing cursor icon files that don't exist during the drag:
sys_open         metacity  2047    17   2 (ENOENT) "/home/mark/.icons/Bluecurve/index.theme", O_RDONLY
sys_open         metacity  2047    17   2 (ENOENT) "/home/mark/.icons/Bluecurve/cursors/left_side", O_RDONLY
sys_open         metacity  2047    17   2 (ENOENT) "/home/mark/.icons/default/index.theme", O_RDONLY
sys_open         metacity  2047    17   2 (ENOENT) "/home/mark/.icons/default/cursors/left_side", O_RDONLY
sys_open         metacity  2047    17   2 (ENOENT) "/usr/share/pixmaps/default/cursors/left_side", O_RDONLY
sys_open         metacity  2047    17   2 (ENOENT) "/usr/share/icons/default/cursors/left_side", O_RDONLY

And that was just playing with the script for 30 seconds. Lots of
interesting stuff should come out of this. Thanks for writing it.

BTW. Frank upded the limit you were hitting in 76eb4a PR4105: support up
to 9 (up from 5) array index dimensions.

Cheers,

Mark