Script to measure resource usage based on process arguments

Mike Mason mmlnx@us.ibm.com
Tue Jan 9 01:27:00 GMT 2007 

Forgot to mention one important thing... I've seen this script crash the system a few times.  The sequence of events that cause the crash are a bit strange.  I start the script, then start a kernel build.  The script and build both run to completion.  Then I start the script again and it crashes. I've attached an example of the console messages.  Do these messages give anybody a clue as to what's happening? 

BTW, I see the crash when I run the script in an xterm in Xvnc.  I don't see the crash when I run the script in a telnet session.  Might be irrelevant, but I thought I'd mention it.

Mike

Console messages
================

Error creating systemtap /proc entries.
Unable to handle kernel paging request at ffffffff8832e1dd RIP: 
 [<ffffffff8832e1dd>]
PGD 203027 PUD 205027 PMD 396e6067 PTE 0
Oops: 0010 [1] SMP 
last sysfs file: /module/scsi_mod/sections/.text
CPU 0 
Modules linked in: ipv6 autofs4 hidp rfcomm l2cap bluetooth sunrpc dm_mirror dm_mod video sbs i2c_ec button battery asus_acpi ac lp parport_pc parport snd_hda_intel snd_hda_codec snd_seq_dummy snd_seq_oss snd_seq_midi_event snd_seq snd_seq_device snd_pcm_oss sg e1000 snd_mixer_oss snd_pcm serio_raw snd_timer ehci_hcd snd uhci_hcd ide_cd i2c_i801 shpchp soundcore snd_page_alloc cdrom i2c_core pcspkr ext3 jbd ahci libata sd_mod scsi_mod
Pid: 2467, comm: Xvnc Not tainted 2.6.18-1.2257.fc5 #1
RIP: 0010:[<ffffffff8832e1dd>]  [<ffffffff8832e1dd>]
RSP: 0000:ffffffff80632f18  EFLAGS: 00010246
RAX: ffff81002d773fd8 RBX: 0000000000000000 RCX: ffff81001aae2378
RDX: 0000000000000000 RSI: 10369bb9244ede90 RDI: 0000000000000000
RBP: ffffffff80671b00 R08: 0000000000b31860 R09: 000000000000000a
R10: 00002aaaaadafa18 R11: 0000000000000004 R12: 0000000000000100
R13: ffffffff8832e1dd R14: 0000000000000027 R15: 0000000000000105
FS:  00002aaaaaacd140(0000) GS:ffffffff805d8000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: ffffffff8832e1dd CR3: 000000002e048000 CR4: 00000000000006e0
Process Xvnc (pid: 2467, threadinfo ffff81002d772000, task ffff81003848b810)
Stack:  ffffffff80290ec5 ffffffff80632f20 ffffffff80632f20 ffffffff80287132
 0000000000000001 ffffffff805de110 000000000000000a 0000000000000000
 ffffffff80211cff ffff81002d773f58 0000000000000046 ffffffff80632f98
Call Trace:
Inexact backtrace:
 <IRQ>  [<ffffffff80290ec5>] run_timer_softirq+0x13b/0x1be
 [<ffffffff80287132>] scheduler_tick+0x31c/0x345
 [<ffffffff80211cff>] __do_softirq+0x5e/0xd5
 [<ffffffff8025d9f0>] call_softirq+0x1c/0x28
 [<ffffffff8026aab8>] do_softirq+0x2c/0x99
 [<ffffffff8025d382>] apic_timer_interrupt+0x66/0x6c
 <EOI> 

Code:  Bad RIP value.
RIP  [<ffffffff8832e1dd>]
 RSP <ffffffff80632f18>
CR2: ffffffff8832e1dd
 <1>Unable to handle kernel paging request<3>BUG: sleeping function called from invalid context at kernel/rwsem.c:20
in_atomic():1, irqs_disabled():1

Call Trace:
 [<ffffffff80269935>] show_trace+0x34/0x47
 [<ffffffff8026995a>] dump_stack+0x12/0x17
 [<ffffffff8029bb57>] down_read+0x15/0x23
 [<ffffffff8029414e>] blocking_notifier_call_chain+0x13/0x36
 [<ffffffff80214e81>] do_exit+0x20/0x928
 [<ffffffff80265023>] do_page_fault+0x79a/0x815
 [<ffffffff8025d4dd>] error_exit+0x0/0x84
DWARF2 unwinder stuck at error_exit+0x0/0x84
Leftover inexact backtrace:
 <IRQ>  [<ffffffff80290ec5>] run_timer_softirq+0x13b/0x1be
 [<ffffffff80287132>] scheduler_tick+0x31c/0x345
 [<ffffffff80211cff>] __do_softirq+0x5e/0xd5
 [<ffffffff8025d9f0>] call_softirq+0x1c/0x28
 [<ffffffff8026aab8>] do_softirq+0x2c/0x99
 [<ffffffff8025d382>] apic_timer_interrupt+0x66/0x6c
 <EOI> 
Kernel panic - not syncing: Aiee, killing interrupt handler!
  at ffffffff8832e1dd RIP: 
 [<ffffffff8832e1dd>]
PGD 203027 PUD 205027 PMD 396e6067 PTE 0
Oops: 0010 [2] SMP 
last sysfs file: /module/scsi_mod/sections/.text
CPU 1 
Modules linked in: ipv6 autofs4 hidp rfcomm l2cap bluetooth sunrpc dm_mirror dm_mod video sbs i2c_ec button battery asus_acpi ac lp parport_pc parport snd_hda_intel snd_hda_codec snd_seq_dummy snd_seq_oss snd_seq_midi_event snd_seq snd_seq_device snd_pcm_oss sg e1000 snd_mixer_oss snd_pcm serio_raw snd_timer ehci_hcd snd uhci_hcd ide_cd i2c_i801 shpchp soundcore snd_page_alloc cdrom i2c_core pcspkr ext3 jbd ahci libata sd_mod scsi_mod
Pid: 24434, comm: insmod Not tainted 2.6.18-1.2257.fc5 #1
RIP: 0010:[<ffffffff8832e1dd>]  [<ffffffff8832e1dd>]
RSP: 0018:ffff81003fe0bf18  EFLAGS: 00010246
RAX: ffff810004d7bfd8 RBX: 0000000000000000 RCX: ffff81001aae2678
RDX: 0000000000000001 RSI: 10369bb9244ede90 RDI: 0000000000000000
RBP: ffff810037ebc000 R08: 0000000000b31860 R09: 000000000000000a
R10: ffff810001182d90 R11: 0000000000000246 R12: 0000000000000100
R13: ffffffff8832e1dd R14: 0000000000000100 R15: 0000000000000003
FS:  0000000000000000(0000) GS:ffff810037e999c0(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: ffffffff8832e1dd CR3: 0000000006a2b000 CR4: 00000000000006e0
Process insmod (pid: 24434, threadinfo ffff810004d7a000, task ffff810039312080)
Stack:  ffffffff80290ec5 ffff81003fe0bf20 ffff81003fe0bf20 ffffffff80287132
 0000000000000011 ffffffff805de110 000000000000000a 0000000000000001
 ffffffff80211cff ffff810004d7bde8 0000000000000046 ffff81003fe0bf98
Call Trace:
Inexact backtrace:
 <IRQ>  [<ffffffff80290ec5>] run_timer_softirq+0x13b/0x1be
 [<ffffffff80287132>] scheduler_tick+0x31c/0x345
 [<ffffffff80211cff>] __do_softirq+0x5e/0xd5
 [<ffffffff8025d9f0>] call_softirq+0x1c/0x28
 [<ffffffff8026aab8>] do_softirq+0x2c/0x99
 [<ffffffff8025d382>] apic_timer_interrupt+0x66/0x6c
 <EOI>  [<ffffffff80262eac>] _spin_unlock_irq+0xb/0xc
 [<ffffffff8020cf32>] __pagevec_lru_add_active+0xb9/0xdc
 [<ffffffff80239e82>] exit_mmap+0x22/0xee
 [<ffffffff8023c058>] mmput+0x41/0x96
 [<ffffffff802150f4>] do_exit+0x293/0x928
 [<ffffffff8024816e>] cpuset_exit+0x0/0x6c
 [<ffffffff8025c80e>] system_call+0x7e/0x83


Code:  Bad RIP value.
RIP  [<ffffffff8832e1dd>]
 RSP <ffff81003fe0bf18>
CR2: ffffffff8832e1dd
BUG: warning at drivers/char/vt.c:3359/do_unblank_screen() (Not tainted)

Call Trace:
 [<ffffffff80269935>] show_trace+0x34/0x47
 [<ffffffff8026995a>] dump_stack+0x12/0x17
 [<ffffffff8038b5a6>] do_unblank_screen+0x67/0x142
 [<ffffffff8027c523>] bust_spinlocks+0x1c/0x46
 [<ffffffff802634ce>] oops_end+0x15/0x53
 [<ffffffff80264fd3>] do_page_fault+0x74a/0x815
 [<ffffffff8025d4dd>] error_exit+0x0/0x84
DWARF2 unwinder stuck at error_exit+0x0/0x84
Leftover inexact backtrace:
 <IRQ>  [<ffffffff80290ec5>] run_timer_softirq+0x13b/0x1be
 [<ffffffff80287132>] scheduler_tick+0x31c/0x345
 [<ffffffff80211cff>] __do_softirq+0x5e/0xd5
 [<ffffffff8025d9f0>] call_softirq+0x1c/0x28
 [<ffffffff8026aab8>] do_softirq+0x2c/0x99
 [<ffffffff8025d382>] apic_timer_interrupt+0x66/0x6c
 <EOI>  [<ffffffff80262eac>] _spin_unlock_irq+0xb/0xc
 [<ffffffff8020cf32>] __pagevec_lru_add_active+0xb9/0xdc
 [<ffffffff80239e82>] exit_mmap+0x22/0xee
 [<ffffffff8023c058>] mmput+0x41/0x96
 [<ffffffff802150f4>] do_exit+0x293/0x928
 [<ffffffff8024816e>] cpuset_exit+0x0/0x6c
 [<ffffffff8025c80e>] system_call+0x7e/0x83

More information about the Systemtap mailing list