Bug 13193 - Running bz6503.exp results in unresponsive/crashed machine
Summary: Running bz6503.exp results in unresponsive/crashed machine
Status: RESOLVED FIXED
Alias: None
Product: systemtap
Classification: Unclassified
Component: testsuite (show other bugs)
Version: unspecified
: P2 normal
Target Milestone: ---
Assignee: Unassigned
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2011-09-16 13:38 UTC by Mark Wielaard
Modified: 2012-02-21 15:11 UTC (History)
2 users (show)

See Also:
Host:
Target:
Build:
Last reconfirmed:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Mark Wielaard 2011-09-16 13:38:23 UTC
This happens on both "real hardware" with x86_64 f14 (2.6.35.14-96.fc14.x86_64) and under kvm with x86_64 rhel6 (2.6.32-131.12.1.el6.x86_64).

The following is the result of running make installcheck RUNTESTFLAGS=bz6503.exp on the rhel6 kvm instance, with echo 9 > /proc/sysrq-trigger and adding -DDEBUG_SYMBOLS to the testcase:

SysRq : Changing Loglevel
Loglevel set to 9
stap_42a0620f5f7fd9b83fd77b5a99e4b3d7__2646: systemtap: 1.7/0.152, base: ffffffffa0507000, memory: 221data/20text/2ctx/10net/106alloc kb, probes: 373
JFFS2 version 2.2. (NAND) (SUMMARY)  © 2001-2006 Red Hat, Inc.
BUG: soft lockup - CPU#0 stuck for 67s! [migration/0:5]
Modules linked in: jffs2(+) ext2(+) zlib_deflate stap_42a0620f5f7fd9b83fd77b5a99e4b3d7__2646(U) ebtable_nat ebtables ipt_MASQUERADE iptable_nat
BUG: soft lockup - CPU#1 stuck for 67s! [migration/1:8]
Modules linked in: jffs2(+) ext2(+) zlib_deflate stap_42a0620f5f7fd9b83fd77b5a99e4b3d7__2646(U) ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat xt_CHECKSUM iptable_mangle bridge stp llc autofs4 nfs lockd fscache(T) nfs_acl auth_rpcgss sunrpc xt_physdev ipt_REJECT nf_conntrack_ipv4 nf_defrag_ipv4 iptable_filter ip_tables ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables ipv6 dm_mirror dm_region_hash dm_log vhost_net macvtap macvlan tun uinput sg microcode virtio_balloon snd_intel8x0 snd_ac97_codec ac97_bus snd_seq snd_seq_device snd_pcm snd_timer snd soundcore snd_page_alloc virtio_net i2c_piix4 i2c_core ext4 mbcache jbd2 virtio_blk sr_mod cdrom virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mod [last unloaded: speedstep_lib]
CPU 1:
Modules linked in: jffs2(+) ext2(+) zlib_deflate stap_42a0620f5f7fd9b83fd77b5a99e4b3d7__2646(U) ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat xt_CHECKSUM iptable_mangle bridge stp llc autofs4 nfs lockd fscache(T) nfs_acl auth_rpcgss sunrpc xt_physdev ipt_REJECT nf_conntrack_ipv4 nf_defrag_ipv4 iptable_filter ip_tables ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables ipv6 dm_mirror dm_region_hash dm_log vhost_net macvtap macvlan tun uinput sg microcode virtio_balloon snd_intel8x0 snd_ac97_codec ac97_bus snd_seq snd_seq_device snd_pcm snd_timer snd soundcore snd_page_alloc virtio_net i2c_piix4 i2c_core ext4 mbcache jbd2 virtio_blk sr_mod cdrom virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mod [last unloaded: speedstep_lib]
Pid: 8, comm: migration/1 Tainted: G           ---------------- T 2.6.32-131.12.1.el6.x86_64 #1 Bochs
RIP: 0010:[<ffffffff810c36f4>]  [<ffffffff810c36f4>] stop_machine_cpu_stop+0x74/0xe0
RSP: 0000:ffff88009cbd5dd0  EFLAGS: 00000293
RAX: 0000000000000001 RBX: ffff88009cbd5df0 RCX: ffff8800022911e8
RDX: 0000000000000000 RSI: ffff880099994100 RDI: ffff880099079c18
RBP: ffffffff8100bc8e R08: ffff88009cbd4000 R09: 0000000000000001
R10: 00000000ffffffff R11: 0000000000000000 R12: ffff88009cbd5d50
R13: ffffffff814dad57 R14: ffff88009cbd5df0 R15: ffff88009ac86f00
FS:  0000000000000000(0000) GS:ffff880002280000(0000) knlGS:0000000000000000
CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
CR2: 00000031ae41b397 CR3: 0000000037335000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Call Trace:
 [<ffffffff810c3680>] ? stop_machine_cpu_stop+0x0/0xe0
 [<ffffffff810c35aa>] ? cpu_stopper_thread+0xda/0x1b0
 [<ffffffff814dad57>] ? thread_return+0x4e/0x777
 [<ffffffff8105dc32>] ? default_wake_function+0x12/0x20
 [<ffffffff810c34d0>] ? cpu_stopper_thread+0x0/0x1b0
 [<ffffffff8108de16>] ? kthread+0x96/0xa0
 [<ffffffff8100c1ca>] ? child_rip+0xa/0x20
 [<ffffffff8108dd80>] ? kthread+0x0/0xa0
 [<ffffffff8100c1c0>] ? child_rip+0x0/0x20
BUG: soft lockup - CPU#2 stuck for 67s! [modprobe:2652]
Modules linked in: jffs2(+) ext2(+) zlib_deflate stap_42a0620f5f7fd9b83fd77b5a99e4b3d7__2646(U) ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat xt_CHECKSUM iptable_mangle bridge stp llc autofs4 nfs lockd fscache(T) nfs_acl auth_rpcgss sunrpc xt_physdev ipt_REJECT nf_conntrack_ipv4 nf_defrag_ipv4 iptable_filter ip_tables ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables ipv6 dm_mirror dm_region_hash dm_log vhost_net macvtap macvlan tun uinput sg microcode virtio_balloon snd_intel8x0 snd_ac97_codec ac97_bus snd_seq snd_seq_device snd_pcm snd_timer snd soundcore snd_page_alloc virtio_net i2c_piix4 i2c_core ext4 mbcache jbd2 virtio_blk sr_mod cdrom virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mod [last unloaded: speedstep_lib]
CPU 2:
Modules linked in: jffs2(+) ext2(+) zlib_deflate stap_42a0620f5f7fd9b83fd77b5a99e4b3d7__2646(U) ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat xt_CHECKSUM iptable_mangle bridge stp llc autofs4 nfs lockd fscache(T) nfs_acl auth_rpcgss sunrpc xt_physdev ipt_REJECT nf_conntrack_ipv4 nf_defrag_ipv4 iptable_filter ip_tables ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables ipv6 dm_mirror dm_region_hash dm_log vhost_net macvtap macvlan tun uinput sg microcode virtio_balloon snd_intel8x0 snd_ac97_codec ac97_bus snd_seq snd_seq_device snd_pcm snd_timer snd soundcore snd_page_alloc virtio_net i2c_piix4 i2c_core ext4 mbcache jbd2 virtio_blk sr_mod cdrom virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mod [last unloaded: speedstep_lib]
Pid: 2652, comm: modprobe Tainted: G           ---------------- T 2.6.32-131.12.1.el6.x86_64 #1 Bochs
RIP: 0010:[<ffffffff810a4536>]  [<ffffffff810a4536>] smp_call_function_many+0x1b6/0x210
RSP: 0018:ffff880098ccbcc8  EFLAGS: 00000202
RAX: 0000000000000011 RBX: ffff880098ccbd08 RCX: 0000000000000004
RDX: 0000000000000010 RSI: 0000000000000000 RDI: 0000000000000286
RBP: ffffffff8100bc8e R08: 0000000000000000 R09: ffff880037bc0c00
R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000002
R13: ffff880098ccbc78 R14: 0000000000000000 R15: 0000000000000020
FS:  00007f5f3eebf700(0000) GS:ffff880002300000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00000000024ef0a8 CR3: 0000000099006000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Call Trace:
 [<ffffffff810a451a>] ? smp_call_function_many+0x19a/0x210
 [<ffffffff81159560>] ? do_ccupdate_local+0x0/0x40
 [<ffffffff810a45b2>] ? smp_call_function+0x22/0x30
 [<ffffffff8106f2f4>] ? on_each_cpu+0x24/0x50
 [<ffffffff8115c53f>] ? do_tune_cpucache+0x12f/0x630
 [<ffffffff8115cc1b>] ? enable_cpucache+0x3b/0xf0
 [<ffffffff814c575f>] ? setup_cpu_cache+0x22f/0x340
 [<ffffffff8115b1b2>] ? kmem_cache_alloc+0x182/0x190
 [<ffffffff8115d8aa>] ? kmem_cache_create+0x3fa/0x580
 [<ffffffffa008c184>] ? init_jffs2_fs+0x0/0xc7 [jffs2]
 [<ffffffffa0362510>] ? jffs2_i_init_once+0x0/0x40 [jffs2]
 [<ffffffffa008c184>] ? init_jffs2_fs+0x0/0xc7 [jffs2]
 [<ffffffffa008c1ba>] ? init_jffs2_fs+0x36/0xc7 [jffs2]
 [<ffffffff8100204c>] ? do_one_initcall+0x3c/0x1d0
 [<ffffffff810aca7f>] ? sys_init_module+0xdf/0x250
 [<ffffffff8100b172>] ? system_call_fastpath+0x16/0x1b
 nf_nat xt_CHECKSUM iptable_mangle bridge stp llc autofs4 nfs lockd fscache(T) nfs_acl auth_rpcgss sunrpc xt_physdev ipt_REJECT nf_conntrack_ipv4 nf_defrag_ipv4 iptable_filter ip_tables ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables ipv6 dm_mirror dm_region_hash dm_log vhost_net macvtap macvlan tun uinput sg microcode virtio_balloon snd_intel8x0 snd_ac97_codec ac97_bus snd_seq snd_seq_device snd_pcm snd_timer snd soundcore snd_page_alloc virtio_net i2c_piix4 i2c_core ext4 mbcache jbd2 virtio_blk sr_mod cdrom virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mod [last unloaded: speedstep_lib]
CPU 0:
Modules linked in: jffs2(+) ext2(+) zlib_deflate stap_42a0620f5f7fd9b83fd77b5a99e4b3d7__2646(U) ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat xt_CHECKSUM iptable_mangle bridge stp llc autofs4 nfs lockd fscache(T) nfs_acl auth_rpcgss sunrpc xt_physdev ipt_REJECT nf_conntrack_ipv4 nf_defrag_ipv4 iptable_filter ip_tables ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables ipv6 dm_mirror dm_region_hash dm_log vhost_net macvtap macvlan tun uinput sg microcode virtio_balloon snd_intel8x0 snd_ac97_codec ac97_bus snd_seq snd_seq_device snd_pcm snd_timer snd soundcore snd_page_alloc virtio_net i2c_piix4 i2c_core ext4 mbcache jbd2 virtio_blk sr_mod cdrom virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mod [last unloaded: speedstep_lib]
Pid: 5, comm: migration/0 Tainted: G           ---------------- T 2.6.32-131.12.1.el6.x86_64 #1 Bochs
RIP: 0010:[<ffffffff810c36f8>]  [<ffffffff810c36f8>] stop_machine_cpu_stop+0x78/0xe0
RSP: 0018:ffff88009cbcbdd0  EFLAGS: 00000293
RAX: 0000000000000001 RBX: ffff88009cbcbdf0 RCX: ffff8800022111e8
RDX: 0000000000000000 RSI: ffff8800953d4ac0 RDI: ffff880099079c18
RBP: ffffffff8100bc8e R08: ffff88009cbca000 R09: 0000000000000001
R10: 00000000ffffffff R11: 0000000000000000 R12: ffff88009cbcbd50
R13: ffffffff814dad57 R14: ffff88009cbcbdf0 R15: ffff880037763200
FS:  0000000000000000(0000) GS:ffff880002200000(0000) knlGS:0000000000000000
CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
CR2: 00000031af203088 CR3: 0000000098cc3000 CR4: 00000000000006f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Call Trace:
 [<ffffffff810c3680>] ? stop_machine_cpu_stop+0x0/0xe0
 [<ffffffff810c35aa>] ? cpu_stopper_thread+0xda/0x1b0
 [<ffffffff814dad57>] ? thread_return+0x4e/0x777
 [<ffffffff8105dc32>] ? default_wake_function+0x12/0x20
 [<ffffffff810c34d0>] ? cpu_stopper_thread+0x0/0x1b0
 [<ffffffff8108de16>] ? kthread+0x96/0xa0
 [<ffffffff8100c1ca>] ? child_rip+0xa/0x20
 [<ffffffff8108dd80>] ? kthread+0x0/0xa0
 [<ffffffff8100c1c0>] ? child_rip+0x0/0x20
BUG: soft lockup - CPU#0 stuck for 67s! [migration/0:5]
Modules linked in: jffs2(+) ext2(+) zlib_deflate stap_42a0620f5f7fd9b83fd77b5a99e4b3d7__2646(U) ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat
BUG: soft lockup - CPU#1 stuck for 67s! [migration/1:8]
Modules linked in: jffs2(+) ext2(+) zlib_deflate stap_42a0620f5f7fd9b83fd77b5a99e4b3d7__2646(U) ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat xt_CHECKSUM iptable_mangle bridge stp llc autofs4 nfs lockd fscache(T) nfs_acl auth_rpcgss sunrpc xt_physdev ipt_REJECT nf_conntrack_ipv4 nf_defrag_ipv4 iptable_filter ip_tables ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables ipv6 dm_mirror dm_region_hash dm_log vhost_net macvtap macvlan tun uinput sg microcode virtio_balloon snd_intel8x0 snd_ac97_codec ac97_bus snd_seq snd_seq_device snd_pcm snd_timer snd soundcore snd_page_alloc virtio_net i2c_piix4 i2c_core ext4 mbcache jbd2 virtio_blk sr_mod cdrom virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mod [last unloaded: speedstep_lib]
CPU 1:
Modules linked in: jffs2(+) ext2(+) zlib_deflate stap_42a0620f5f7fd9b83fd77b5a99e4b3d7__2646(U) ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat xt_CHECKSUM iptable_mangle bridge stp llc autofs4 nfs lockd fscache(T) nfs_acl auth_rpcgss sunrpc xt_physdev ipt_REJECT nf_conntrack_ipv4 nf_defrag_ipv4 iptable_filter ip_tables ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables ipv6 dm_mirror dm_region_hash dm_log vhost_net macvtap macvlan tun uinput sg microcode virtio_balloon snd_intel8x0 snd_ac97_codec ac97_bus snd_seq snd_seq_device snd_pcm snd_timer snd soundcore snd_page_alloc virtio_net i2c_piix4 i2c_core ext4 mbcache jbd2 virtio_blk sr_mod cdrom virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mod [last unloaded: speedstep_lib]
Pid: 8, comm: migration/1 Tainted: G           ---------------- T 2.6.32-131.12.1.el6.x86_64 #1 Bochs
RIP: 0010:[<ffffffff810c36f4>]  [<ffffffff810c36f4>] stop_machine_cpu_stop+0x74/0xe0
RSP: 0000:ffff88009cbd5dd0  EFLAGS: 00000293
RAX: 0000000000000001 RBX: ffff88009cbd5df0 RCX: ffff8800022911e8
RDX: 0000000000000000 RSI: ffff880099994100 RDI: ffff880099079c18
RBP: ffffffff8100bc8e R08: ffff88009cbd4000 R09: 0000000000000001
R10: 00000000ffffffff R11: 0000000000000000 R12: ffff88009cbd5d50
R13: ffffffff814dad57 R14: ffff88009cbd5df0 R15: ffff88009ac86f00
FS:  0000000000000000(0000) GS:ffff880002280000(0000) knlGS:0000000000000000
CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
CR2: 00000031ae41b397 CR3: 0000000037335000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Call Trace:
 [<ffffffff810c3680>] ? stop_machine_cpu_stop+0x0/0xe0
 [<ffffffff810c35aa>] ? cpu_stopper_thread+0xda/0x1b0
 [<ffffffff814dad57>] ? thread_return+0x4e/0x777
 [<ffffffff8105dc32>] ? default_wake_function+0x12/0x20
 [<ffffffff810c34d0>] ? cpu_stopper_thread+0x0/0x1b0
 [<ffffffff8108de16>] ? kthread+0x96/0xa0
 [<ffffffff8100c1ca>] ? child_rip+0xa/0x20
 [<ffffffff8108dd80>] ? kthread+0x0/0xa0
 [<ffffffff8100c1c0>] ? child_rip+0x0/0x20
BUG: soft lockup - CPU#2 stuck for 67s! [modprobe:2652]
Modules linked in: jffs2(+) ext2(+) zlib_deflate stap_42a0620f5f7fd9b83fd77b5a99e4b3d7__2646(U) ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat xt_CHECKSUM iptable_mangle bridge stp llc autofs4 nfs lockd fscache(T) nfs_acl auth_rpcgss sunrpc xt_physdev ipt_REJECT nf_conntrack_ipv4 nf_defrag_ipv4 iptable_filter ip_tables ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables ipv6 dm_mirror dm_region_hash dm_log vhost_net macvtap macvlan tun uinput sg microcode virtio_balloon snd_intel8x0 snd_ac97_codec ac97_bus snd_seq snd_seq_device snd_pcm snd_timer snd soundcore snd_page_alloc virtio_net i2c_piix4 i2c_core ext4 mbcache jbd2 virtio_blk sr_mod cdrom virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mod [last unloaded: speedstep_lib]
CPU 2:
Modules linked in: jffs2(+) ext2(+) zlib_deflate stap_42a0620f5f7fd9b83fd77b5a99e4b3d7__2646(U) ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat xt_CHECKSUM iptable_mangle bridge stp llc autofs4 nfs lockd fscache(T) nfs_acl auth_rpcgss sunrpc xt_physdev ipt_REJECT nf_conntrack_ipv4 nf_defrag_ipv4 iptable_filter ip_tables ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables ipv6 dm_mirror dm_region_hash dm_log vhost_net macvtap macvlan tun uinput sg microcode virtio_balloon snd_intel8x0 snd_ac97_codec ac97_bus snd_seq snd_seq_device snd_pcm snd_timer snd soundcore snd_page_alloc virtio_net i2c_piix4 i2c_core ext4 mbcache jbd2 virtio_blk sr_mod cdrom virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mod [last unloaded: speedstep_lib]
Pid: 2652, comm: modprobe Tainted: G           ---------------- T 2.6.32-131.12.1.el6.x86_64 #1 Bochs
RIP: 0010:[<ffffffff810a4536>]  [<ffffffff810a4536>] smp_call_function_many+0x1b6/0x210
RSP: 0018:ffff880098ccbcc8  EFLAGS: 00000202
RAX: 0000000000000011 RBX: ffff880098ccbd08 RCX: 0000000000000004
RDX: 0000000000000010 RSI: 0000000000000000 RDI: 0000000000000286
RBP: ffffffff8100bc8e R08: 0000000000000000 R09: ffff880037bc0c00
R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000002
R13: ffff880098ccbc78 R14: 0000000000000000 R15: 0000000000000020
FS:  00007f5f3eebf700(0000) GS:ffff880002300000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00000000024ef0a8 CR3: 0000000099006000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Call Trace:
 [<ffffffff810a451a>] ? smp_call_function_many+0x19a/0x210
 [<ffffffff81159560>] ? do_ccupdate_local+0x0/0x40
 [<ffffffff810a45b2>] ? smp_call_function+0x22/0x30
 [<ffffffff8106f2f4>] ? on_each_cpu+0x24/0x50
 [<ffffffff8115c53f>] ? do_tune_cpucache+0x12f/0x630
 [<ffffffff8115cc1b>] ? enable_cpucache+0x3b/0xf0
 [<ffffffff814c575f>] ? setup_cpu_cache+0x22f/0x340
 [<ffffffff8115b1b2>] ? kmem_cache_alloc+0x182/0x190
 [<ffffffff8115d8aa>] ? kmem_cache_create+0x3fa/0x580
 [<ffffffffa008c184>] ? init_jffs2_fs+0x0/0xc7 [jffs2]
 [<ffffffffa0362510>] ? jffs2_i_init_once+0x0/0x40 [jffs2]
 [<ffffffffa008c184>] ? init_jffs2_fs+0x0/0xc7 [jffs2]
 [<ffffffffa008c1ba>] ? init_jffs2_fs+0x36/0xc7 [jffs2]
 [<ffffffff8100204c>] ? do_one_initcall+0x3c/0x1d0
 [<ffffffff810aca7f>] ? sys_init_module+0xdf/0x250
 [<ffffffff8100b172>] ? system_call_fastpath+0x16/0x1b
 xt_CHECKSUM iptable_mangle bridge stp llc autofs4 nfs lockd fscache(T) nfs_acl auth_rpcgss sunrpc xt_physdev ipt_REJECT nf_conntrack_ipv4 nf_defrag_ipv4 iptable_filter ip_tables ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables ipv6 dm_mirror dm_region_hash dm_log vhost_net macvtap macvlan tun uinput sg microcode virtio_balloon snd_intel8x0 snd_ac97_codec ac97_bus snd_seq snd_seq_device snd_pcm snd_timer snd soundcore snd_page_alloc virtio_net i2c_piix4 i2c_core ext4 mbcache jbd2 virtio_blk sr_mod cdrom virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mod [last unloaded: speedstep_lib]
CPU 0:
Modules linked in: jffs2(+) ext2(+) zlib_deflate stap_42a0620f5f7fd9b83fd77b5a99e4b3d7__2646(U) ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat xt_CHECKSUM iptable_mangle bridge stp llc autofs4 nfs lockd fscache(T) nfs_acl auth_rpcgss sunrpc xt_physdev ipt_REJECT nf_conntrack_ipv4 nf_defrag_ipv4 iptable_filter ip_tables ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables ipv6 dm_mirror dm_region_hash dm_log vhost_net macvtap macvlan tun uinput sg microcode virtio_balloon snd_intel8x0 snd_ac97_codec ac97_bus snd_seq snd_seq_device snd_pcm snd_timer snd soundcore snd_page_alloc virtio_net i2c_piix4 i2c_core ext4 mbcache jbd2 virtio_blk sr_mod cdrom virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mod [last unloaded: speedstep_lib]
Pid: 5, comm: migration/0 Tainted: G           ---------------- T 2.6.32-131.12.1.el6.x86_64 #1 Bochs
RIP: 0010:[<ffffffff810c36f4>]  [<ffffffff810c36f4>] stop_machine_cpu_stop+0x74/0xe0
RSP: 0018:ffff88009cbcbdd0  EFLAGS: 00000293
RAX: 0000000000000001 RBX: ffff88009cbcbdf0 RCX: ffff8800022111e8
RDX: 0000000000000000 RSI: ffff8800953d4ac0 RDI: ffff880099079c18
RBP: ffffffff8100bc8e R08: ffff88009cbca000 R09: 0000000000000001
R10: 00000000ffffffff R11: 0000000000000000 R12: ffff88009cbcbd50
R13: ffffffff814dad57 R14: ffff88009cbcbdf0 R15: ffff880037763200
FS:  0000000000000000(0000) GS:ffff880002200000(0000) knlGS:0000000000000000
CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
CR2: 00000031af203088 CR3: 0000000098cc3000 CR4: 00000000000006f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Call Trace:
 [<ffffffff810c3680>] ? stop_machine_cpu_stop+0x0/0xe0
 [<ffffffff810c35aa>] ? cpu_stopper_thread+0xda/0x1b0
 [<ffffffff814dad57>] ? thread_return+0x4e/0x777
 [<ffffffff8105dc32>] ? default_wake_function+0x12/0x20
 [<ffffffff810c34d0>] ? cpu_stopper_thread+0x0/0x1b0
 [<ffffffff8108de16>] ? kthread+0x96/0xa0
 [<ffffffff8100c1ca>] ? child_rip+0xa/0x20
 [<ffffffff8108dd80>] ? kthread+0x0/0xa0
 [<ffffffff8100c1c0>] ? child_rip+0x0/0x20
Comment 1 Mark Wielaard 2011-09-16 18:40:08 UTC
I tried to see if this was module insert/remove related.
The following gets us a panic, although this might be unrelated.

# cd /lib/modules/`uname -r`
# for i in `find * -name \*.ko;`; do insmod $i & done

test_nx: stack was executable
test_nx: heap was executable
wacom driver registered
telclk_interrup = 0xf non-mcpbl0010 hw.
i2c-parport-light: adapter type unspecified
i2c-stub: Please specify a chip address
Error: Driver 'pcspkr' is already registered, aborting...
acpiphp_ibm: ibm_acpiphp_init: acpi_walk_namespace failed
i2c i2c-0: failed,FFFF,FF,FFFF
i2c i2c-0: failed,FFFF,FF,FFFF
i2c i2c-0: Couldn't detect a adt7475 part at 0x2e
Failed to ioremap_nocache
w83697ug/uf WDT: No W83697UG/UF could be found
BUG: unable to handle kernel NULL pointer dereference at (null)
IP: [<ffffffff81269aab>] strcmp+0xb/0x30
PGD 800db067 PUD 800fa067 PMD 0 
Oops: 0000 [#1] SMP 
last sysfs file: /sys/module/lirc_dev/initstate
CPU 3 
Modules linked in: sbc_fitpc2_wdt(+) machzwd(-) idmouse appledisplay usbsevseg usblcd cdc_wdm usblp ibmasr(-) w83697ug_wdt(-) xhci_hcd wm8350_wdt sl811_hcd umc w83877f_wdt it8712f_wdt(-) wm831x_wdt vstusb mtdconcat
SoftDog: cannot register miscdev on minor=130 (err=-16)
 mtdchar qinfo_probe nand_ecc ftl redboot map_rom cfi_util ts5500_flash(-)
WDT driver for the Winbond(TM) W83627HF/THF/HG Super I/O chip initialising.
w83627hf/thf/hg WDT: Watchdog already running. Resetting timeout to 60 sec
w83627hf/thf/hg WDT: cannot register miscdev on minor=130 (err=-16)
ipw: Unknown symbol usb_serial_port_softint
ipw: Unknown symbol usb_serial_disconnect
ipw: Unknown symbol usb_serial_probe
ipw: Unknown symbol usb_serial_register
ipw: Unknown symbol usb_serial_deregister
visor: Unknown symbol usb_serial_port_softint
visor: Unknown symbol usb_serial_disconnect
visor: Unknown symbol usb_serial_probe
visor: Unknown symbol usb_serial_register
visor: Unknown symbol usb_serial_deregister
sierra: Unknown symbol usb_serial_port_softint
sierra: Unknown symbol usb_serial_disconnect
sierra: Unknown symbol usb_serial_suspend
sierra: Unknown symbol usb_serial_probe
sierra: Unknown symbol usb_serial_register
sierra: Unknown symbol usb_serial_deregister
sierra: Unknown symbol usb_serial_resume
 map_ram gen_probe pci sisusbvga iTCO_vendor_support netsc520(-) ck804xrom(-) mtdoops i1480_dfu_usb usbled emi62 nand_ids
ark3116: Unknown symbol usb_serial_disconnect
ark3116: Unknown symbol usb_serial_generic_open
ark3116: Unknown symbol usb_serial_probe
ark3116: Unknown symbol usb_serial_register
ark3116: Unknown symbol usb_serial_deregister
usbcore: registered new interface driver microtekX6
pl2303: Unknown symbol usb_serial_port_softint
pl2303: Unknown symbol usb_serial_handle_sysrq_char
pl2303: Unknown symbol usb_serial_disconnect
pl2303: Unknown symbol usb_serial_suspend
pl2303: Unknown symbol usb_serial_handle_break
pl2303: Unknown symbol usb_serial_probe
pl2303: Unknown symbol usb_serial_register
pl2303: Unknown symbol usb_serial_deregister
pl2303: Unknown symbol usb_serial_resume
qcserial: Unknown symbol usb_wwan_write
qcserial: Unknown symbol usb_wwan_suspend
qcserial: Unknown symbol usb_wwan_close
qcserial: Unknown symbol usb_wwan_release
qcserial: Unknown symbol usb_serial_disconnect
qcserial: Unknown symbol usb_wwan_chars_in_buffer
qcserial: Unknown symbol usb_serial_suspend
qcserial: Unknown symbol usb_wwan_write_room
qcserial: Unknown symbol usb_wwan_disconnect
qcserial: Unknown symbol usb_wwan_startup
qcserial: Unknown symbol usb_wwan_resume
qcserial: Unknown symbol usb_wwan_open
qcserial: Unknown symbol usb_serial_probe
qcserial: Unknown symbol usb_serial_register
qcserial: Unknown symbol usb_serial_deregister
qcserial: Unknown symbol usb_serial_resume
belkin_sa: Unknown symbol usb_serial_disconnect
belkin_sa: Unknown symbol usb_serial_probe
belkin_sa: Unknown symbol usb_serial_register
belkin_sa: Unknown symbol usb_serial_deregister
 mtdblock_ro scb2_flash ricoh_mmc mtdram map_absent ar7part block2mtd lpddr_cmds inftl mtd_blkdevs wm8350 wm831x dell_wmi adt7475 asb100 ltc4215 f75375s i5k_amb lm75 lis3lv02d g760a sis5595 lm80 k10temp ibmpex ds1621 hwmon_vid asus_atk0110 gru adm1029 cciss hwmon i2c_amd8111 i2c_amd756 tsl2550 i2c_viapro i2c_voodoo3 i2c_tiny_usb i2c_via i2c_sis96x ir_lirc_codec i2c_scmi i2c_simtec i2c_pca_platform i2c_isch i2c_dev pcf8591 ir_jvc_decoder twofish_x86_64 keyspan_remote ir_rc5_decoder camellia des_generic bcm5974 synaptics_i2c atlas_btns kbtab appletouch i2c_algo_bit i2c_algo_pca rc_apac_viewcomp acecad yealink sermouse dvb_core cyclades ipmi_msghandler rc_avermedia_m733a_rm_k6 rc_avermedia_m135a cryptoloop dvb_usb_af9005_remote kvm virtio_console xt_statistic aiptek ad7879 isl29003 rc_avermedia ics932s401 memstick tifm_7xx1 ioc4 ab3100_core eeprom sm501 floppy wm8400_core tifm_core at24 cb710 enclosure cls_basic snd_usb_lib hpwdt rc_adstech_dvb_t_pci tua6100 snd_ice1724 elo snd_hda_codec_conexant snd_indigodj sx8 isl6405 isl6421 snd_seq_oss max6875 hpilo eeprom_93cx6 mfd_core soc_camera_platform snd_usb_caiaq rc_lirc snd_rme96 usbtouchscreen fujitsu_ts max17040_battery airo fujitsu_laptop panasonic_laptop wmi topstar_laptop bq27x00_battery aer_inject zd1201 rc_alink_dtu_m snd_emu10k1 snd_ens1371 pps_core inexio mtouch touchright touchit213 tsc2007 penmount mcs5000_ts gunze ati_remote ir_sony_decoder videodev gtco serio_raw ati_remote2 vsxxxaa eeti_ts opencores_kbd max7359_keypad v4l2_compat_ioctl32 v4l2_int_device videobuf_vmalloc videobuf_dma_sg ir_rc5_sz_decoder snd_ctxfi videobuf_core snd_lx6464es snd_mixart snd_atiixp snd_pcxhr pktcdvd rc_ati_tv_wonder_hd_600 rc_avermedia_a16d rc_asus_pc39 wacom_w8001 rc_anysee snd_hda_codec fakephp touchwin ir_rc6_decoder i2c_nforce2 i2c_i801 nls_cp861 nls_cp775 lxt powermate adp5588_keys input_polldev soc_mediabus ir_kbd_i2c intel_ips snd_cs46xx snd_darla24 snd_ca0106 snd_hdspm snd_korg1212 rc_dntv_live_dvb_t rc_digittrade rc_cinergy_1400 rc_fusionhdtv_mce lirc_dev mceusb ene_ir ir_nec_decoder streamzap winbond_cir snd_hdsp ttusbdecfe snd_indigoiox snd_indigodjx snd_mona snd_mia rsrc_nonstatic btcx_risc nls_cp855 tveeprom act_ipt rc_msi_digivox_iii rc_msi_digivox_ii rc_iodata_bctv7e rc_terratec_slim rc_imon_pad rc_total_media_in_hand rc_videomate_m1f rc_hauppauge_new rc_kworld_315u rc_nebula rc_real_audio_220_32_keys rc_tevii_nec rc_terratec_cinergy_xs rc_tbs_nec rc_core dib3000mb snd_echo3g snd_layla24 nxt200x s5h1411 si21xx tda1004x s921 s5h1420 rtc_m48t35 lgs8gxx mb86a20s dib0070 cx24116 bcm3510 dibx000_common dib0090 ec100 cx24123 dvb_pll l64781 tda8083 ves1x93 zl10353 isl6423 af9013 tda18271 tda10021 mt352 stb6100 stv0297 sp887x sp8870 tda10023 lnbp21 edd tuner_xc2028 mt312 cx24110 cx24113 cx22702 cx22700 itd1000 atbm8830 pata_hpt3x2n scsi_wait_scan sata_inic162x dcdbas snd_rawmidi snd_sb_common snd_aloop max2165 snd_ali5451 tea5767 mt2131 mt20xx mc44s803 tda18218 tda9887 qt1010 mxl5005s tuner_simple mxl5007t tda827x tda8290 tea5761 mt2060 raid0 mt2266 xc5000 tuner_types snd_gina20 snd_darla20 snd_vx_lib hid_wacom ix2505v stb6000 ds3000 tda10086 tda8261 ves1820 zl10036 s5h1409 tda826x stv6110 nxt6000 zl10039 stv090x tda10048 stv6110x stv0288 sbs twofish_common seed serpent sbshc cpufreq_powersave hed erst_dbg virtio_rng timeriomem_rng hangcheck_timer tpm_infineon cpufreq_ondemand dell_rbu cm4000_cs bcm203x snd_rme9652 snd_hrtimer snd_indigo snd_indigoio snd_gina24 ttpci_eeprom cpufreq_stats cm4040_cs n_hdlc cpufreq_conservative snd_bt87x snd_seq_midi_emul snd_seq_midi_event snd_ice17xx_ak4xxx snd_cs8427 snd_cs5535audio snd_ak4114 snd_pcsp snd_pt2258 snd_ak4xxx_adda snd_i2c snd_dummy snd_seq_dummy snd_hwdep snd_intel8x0m snd_rme32 snd_maestro3 snd_ad1889 snd_atiixp_modem snd_via82xx_modem michael_mic sha512_generic xor wp512 tcrypt(+) sha256_generic arc4 ecb gf128mul cast6 fcrypt tgr192 authenc seqiv cast5 rmd128 tea cts rmd160 ccm rmd320 snd_util_mem oprofile khazad pcbc rmd256 vmac gcm xcbc cryptd anubis test_nx blowfish mce_inject freq_table speedstep_lib ansi_cprng(+) md4 cbc ctr salsa20_x86_64 crypto_null async_tx fpu ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat xt_CHECKSUM iptable_mangle bridge stp llc autofs4 nfs lockd fscache(T) nfs_acl auth_rpcgss sunrpc xt_physdev ipt_REJECT nf_conntrack_ipv4 nf_defrag_ipv4 iptable_filter ip_tables ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables ipv6 dm_mirror dm_region_hash dm_log vhost_net macvtap macvlan tun uinput sg microcode virtio_balloon snd_intel8x0 snd_ac97_codec ac97_bus snd_seq snd_seq_device snd_pcm snd_timer snd soundcore snd_page_alloc virtio_net i2c_piix4 i2c_core ext4 mbcache jbd2 virtio_blk sr_mod cdrom virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mod [last unloaded: speedstep_lib]

Modules linked in: microtek w83627hf_wdt(-) softdog(-) whci sbc_fitpc2_wdt(+) machzwd(-) idmouse appledisplay usbsevseg usblcd cdc_wdm usblp ibmasr(-) w83697ug_wdt(-) xhci_hcd wm8350_wdt sl811_hcd umc w83877f_wdt it8712f_wdt(-) wm831x_wdt vstusb mtdconcat mtdchar qinfo_probe nand_ecc ftl redboot map_rom cfi_util ts5500_flash(-) map_ram gen_probe pci sisusbvga iTCO_vendor_support netsc520(-) ck804xrom(-) mtdoops i1480_dfu_usb usbled emi62 nand_ids mtdblock_ro scb2_flash ricoh_mmc mtdram map_absent ar7part block2mtd lpddr_cmds inftl mtd_blkdevs wm8350 wm831x dell_wmi adt7475 asb100 ltc4215 f75375s i5k_amb lm75 lis3lv02d g760a sis5595 lm80 k10temp ibmpex ds1621 hwmon_vid asus_atk0110 gru adm1029 cciss hwmon i2c_amd8111 i2c_amd756 tsl2550 i2c_viapro i2c_voodoo3 i2c_tiny_usb i2c_via i2c_sis96x ir_lirc_codec i2c_scmi i2c_simtec i2c_pca_platform i2c_isch i2c_dev pcf8591 ir_jvc_decoder twofish_x86_64 keyspan_remote ir_rc5_decoder camellia des_generic bcm5974 synaptics_i2c atlas_btns kbtab appletouch i2c_algo_bit i2c_algo_pca rc_apac_viewcomp acecad yealink sermouse dvb_core cyclades ipmi_msghandler rc_avermedia_m733a_rm_k6 rc_avermedia_m135a cryptoloop dvb_usb_af9005_remote kvm virtio_console xt_statistic aiptek ad7879 isl29003 rc_avermedia ics932s401 memstick tifm_7xx1 ioc4 ab3100_core eeprom sm501 floppy wm8400_core tifm_core at24 cb710 enclosure cls_basic snd_usb_lib hpwdt rc_adstech_dvb_t_pci tua6100 snd_ice1724 elo snd_hda_codec_conexant snd_indigodj sx8 isl6405 isl6421 snd_seq_oss max6875 hpilo eeprom_93cx6 mfd_core soc_camera_platform snd_usb_caiaq rc_lirc snd_rme96 usbtouchscreen fujitsu_ts max17040_battery airo fujitsu_laptop panasonic_laptop wmi topstar_laptop bq27x00_battery aer_inject zd1201 rc_alink_dtu_m snd_emu10k1 snd_ens1371 pps_core inexio mtouch touchright touchit213 tsc2007 penmount mcs5000_ts gunze ati_remote ir_sony_decoder videodev gtco serio_raw ati_remote2 vsxxxaa eeti_ts opencores_kbd max7359_keypad v4l2_compat_ioctl32 v4l2_int_device videobuf_vmalloc videobuf_dma_sg ir_rc5_sz_decoder snd_ctxfi videobuf_core snd_lx6464es snd_mixart snd_atiixp snd_pcxhr pktcdvd rc_ati_tv_wonder_hd_600 rc_avermedia_a16d rc_asus_pc39 wacom_w8001 rc_anysee snd_hda_codec fakephp touchwin ir_rc6_decoder i2c_nforce2 i2c_i801 nls_cp861 nls_cp775 lxt powermate adp5588_keys input_polldev soc_mediabus ir_kbd_i2c intel_ips snd_cs46xx snd_darla24 snd_ca0106 snd_hdspm snd_korg1212 rc_dntv_live_dvb_t rc_digittrade rc_cinergy_1400 rc_fusionhdtv_mce lirc_dev mceusb ene_ir ir_nec_decoder streamzap winbond_cir snd_hdsp ttusbdecfe snd_indigoiox snd_indigodjx snd_mona snd_mia rsrc_nonstatic btcx_risc nls_cp855 tveeprom act_ipt rc_msi_digivox_iii rc_msi_digivox_ii rc_iodata_bctv7e rc_terratec_slim rc_imon_pad rc_total_media_in_hand rc_videomate_m1f rc_hauppauge_new rc_kworld_315u rc_nebula rc_real_audio_220_32_keys rc_tevii_nec rc_terratec_cinergy_xs rc_tbs_nec rc_core dib3000mb snd_echo3g snd_layla24 nxt200x s5h1411 si21xx tda1004x s921 s5h1420 rtc_m48t35 lgs8gxx mb86a20s dib0070 cx24116 bcm3510 dibx000_common dib0090 ec100 cx24123 dvb_pll l64781 tda8083 ves1x93 zl10353 isl6423 af9013 tda18271 tda10021 mt352 stb6100 stv0297 sp887x sp8870 tda10023 lnbp21 edd tuner_xc2028 mt312 cx24110 cx24113 cx22702 cx22700 itd1000 atbm8830 pata_hpt3x2n scsi_wait_scan sata_inic162x dcdbas snd_rawmidi snd_sb_common snd_aloop max2165 snd_ali5451 tea5767 mt2131 mt20xx mc44s803 tda18218 tda9887 qt1010 mxl5005s tuner_simple mxl5007t tda827x tda8290 tea5761 mt2060 raid0 mt2266 xc5000 tuner_types snd_gina20 snd_darla20 snd_vx_lib hid_wacom ix2505v stb6000 ds3000 tda10086 tda8261 ves1820 zl10036 s5h1409 tda826x stv6110 nxt6000 zl10039 stv090x tda10048 stv6110x stv0288 sbs twofish_common seed serpent sbshc cpufreq_powersave hed erst_dbg virtio_rng timeriomem_rng hangcheck_timer tpm_infineon cpufreq_ondemand dell_rbu cm4000_cs bcm203x snd_rme9652 snd_hrtimer snd_indigo snd_indigoio snd_gina24 ttpci_eeprom cpufreq_stats cm4040_cs n_hdlc cpufreq_conservative snd_bt87x snd_seq_midi_emul snd_seq_midi_event snd_ice17xx_ak4xxx snd_cs8427 snd_cs5535audio snd_ak4114 snd_pcsp snd_pt2258 snd_ak4xxx_adda snd_i2c snd_dummy snd_seq_dummy snd_hwdep snd_intel8x0m snd_rme32 snd_maestro3 snd_ad1889 snd_atiixp_modem snd_via82xx_modem michael_mic sha512_generic xor wp512 tcrypt(+) sha256_generic arc4 ecb gf128mul cast6 fcrypt tgr192 authenc seqiv cast5 rmd128 tea cts rmd160 ccm rmd320 snd_util_mem oprofile khazad pcbc rmd256 vmac gcm xcbc cryptd anubis test_nx blowfish mce_inject freq_table speedstep_lib ansi_cprng(+) md4 cbc ctr salsa20_x86_64 crypto_null async_tx fpu ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat xt_CHECKSUM iptable_mangle bridge stp llc autofs4 nfs lockd fscache(T) nfs_acl auth_rpcgss sunrpc xt_physdev ipt_REJECT nf_conntrack_ipv4 nf_defrag_ipv4 iptable_filter ip_tables ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables ipv6 dm_mirror dm_region_hash dm_log vhost_net macvtap macvlan tun uinput sg microcode virtio_balloon snd_intel8x0 snd_ac97_codec ac97_bus snd_seq snd_seq_device snd_pcm snd_timer snd soundcore snd_page_alloc virtio_net i2c_piix4 i2c_core ext4 mbcache jbd2 virtio_blk sr_mod cdrom virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mod [last unloaded: speedstep_lib]
Pid: 3166, comm: insmod Tainted: G           ---------------- T 2.6.32-131.12.1.el6.x86_64 #1 Bochs
RIP: 0010:[<ffffffff81269aab>]  [<ffffffff81269aab>] strcmp+0xb/0x30
RSP: 0018:ffff8800800f7ef8  EFLAGS: 00010292
RAX: 0000000000000053 RBX: 0000000000000000 RCX: 0000000000000000
RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffffffffa23d66e8
RBP: ffff8800800f7ef8 R08: 0000000000000000 R09: 0000000000000001
R10: ffff88000240a3c0 R11: 0000000000000000 R12: ffffffffa23d9000
R13: 000000000162a010 R14: 000000000162a030 R15: 0000000000000003
FS:  00007fc032497700(0000) GS:ffff880002380000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000000 CR3: 00000000800d8000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process insmod (pid: 3166, threadinfo ffff8800800f6000, task ffff8800800e6b40)
Stack:
 ffff8800800f7f18 ffffffffa23d9022 ffff8800800f7f18 0000000000000000
<0> ffff8800800f7f48 ffffffff8100204c 000000000162a030 ffffffffa23d6860
<0> 00000000000031b0 000000000162a010 ffff8800800f7f78 ffffffff810aca7f
Call Trace:
 [<ffffffffa23d9022>] fitpc2_wdt_init+0x22/0x146 [sbc_fitpc2_wdt]
 [<ffffffff8100204c>] do_one_initcall+0x3c/0x1d0
 [<ffffffff810aca7f>] sys_init_module+0xdf/0x250
 [<ffffffff8100b172>] system_call_fastpath+0x16/0x1b
Code: 84 ff 40 88 39 74 0d 48 83 c1 01 48 83 ea 01 75 e7 c6 01 00 c9 c3 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 0f 1f 40 00 0f b6 07 <0f> b6 16 48 83 c7 01 48 83 c6 01 38 d0 75 0e 84 c0 75 ea 31 c0 
RIP  [<ffffffff81269aab>] strcmp+0xb/0x30
 RSP <ffff8800800f7ef8>
CR2: 0000000000000000
hrtimer: interrupt took 15909546 ns
---[ end trace c66712db45fd58fc ]---
Kernel panic - not syncing: Fatal exception
Pid: 3166, comm: insmod Tainted: G      D    ---------------- T 2.6.32-131.12.1.el6.x86_64 #1
Call Trace:
 [<ffffffff814da648>] ? panic+0x78/0x143
 [<ffffffff814de694>] ? oops_end+0xe4/0x100
 [<ffffffff81040c9b>] ? no_context+0xfb/0x260
 [<ffffffff81040f25>] ? __bad_area_nosemaphore+0x125/0x1e0
 [<ffffffff8104104e>] ? bad_area+0x4e/0x60
 [<ffffffff81041773>] ? __do_page_fault+0x3c3/0x480
 [<ffffffff8104a240>] ? tg_nop+0x0/0x10
 [<ffffffff8105ff50>] ? tg_shares_up+0x0/0x2a0
 [<ffffffff8104f88a>] ? walk_tg_tree+0x6a/0xb0
 [<ffffffff810370f8>] ? pvclock_clocksource_read+0x58/0xd0
 [<ffffffffa23d6007>] ? wdt_send_data+0x7/0x60 [sbc_fitpc2_wdt]
 [<ffffffff814e067e>] ? do_page_fault+0x3e/0xa0
 [<ffffffffa23d9000>] ? fitpc2_wdt_init+0x0/0x146 [sbc_fitpc2_wdt]
 [<ffffffff814dda05>] ? page_fault+0x25/0x30
 [<ffffffffa23d9000>] ? fitpc2_wdt_init+0x0/0x146 [sbc_fitpc2_wdt]
 [<ffffffff81269aab>] ? strcmp+0xb/0x30
 [<ffffffff810943c5>] ? __blocking_notifier_call_chain+0x65/0x80
 [<ffffffffa23d9022>] ? fitpc2_wdt_init+0x22/0x146 [sbc_fitpc2_wdt]
 [<ffffffff8100204c>] ? do_one_initcall+0x3c/0x1d0
 [<ffffffff810aca7f>] ? sys_init_module+0xdf/0x250
 [<ffffffff8100b172>] ? system_call_fastpath+0x16/0x1b
Comment 2 Mark Wielaard 2011-09-23 11:52:17 UTC
There was a workaround committed which looked like it was perfect:

commit 01cfbeb43559e5a93eda648fc81d4aa102e6cb13
Author: Frank Ch. Eigler <fche@redhat.com>
Date:   Fri Sep 16 17:34:14 2011 -0400

    PR13193 wallpaper
    
    The reported stuck process bug (whether it's in the kernel or
    in stap), is apparently triggered by concurrent modprobes.  Change
    the testsuite to avoid this practice, to wallpaper over the problem
    temporarily.

Unfortunately I just got another issue with this testcase, this time on i386 f15:

[157004.483057] stap_6ab51e385550b0fd6d843037b0d7d38c_9609: systemtap: 1.7/0.148.90, base: f8aca000, memory: 34data/20text/2ctx/10net/16alloc kb, probes: 2
[157008.989239] stap_a40026234a1787c83a1dafb3aae558ca__9841: systemtap: 1.7/0.148.90, base: f8b24000, memory: 61data/24text/15ctx/10net/163alloc kb, probes: 10
[157013.346702] stap_190d377c71fa6a45f476ce480c4912a1_10067: systemtap: 1.7/0.148.90, base: f8ad9000, memory: 51data/24text/15ctx/10net/163alloc kb, probes: 6
[157018.205175] stap_1aa19d6453909e68c4758c58d1d67725_10296: systemtap: 1.7/0.148.90, base: f8b61000, memory: 143data/15text/0ctx/10net/64alloc kb, probes: 396
[157018.227386] Warning: found a stray unused aggrprobe@f8942011
[157018.227398] ------------[ cut here ]------------
[157018.227453] kernel BUG at kernel/kprobes.c:707!
[157018.227498] invalid opcode: 0000 [#1] SMP 
[157018.227543] Modules linked in: jffs2(+) stap_1aa19d6453909e68c4758c58d1d67725_10296 uprobes nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack stap_e9739cce4a6f43356403eefe748313cc_565 stap_06d4707db0c0eb2d86b639a84f948296_9487 stap_e0d0211e1b7bbf8f7df399c1665c8493_565 mtd zlib_deflate tcp_lp tun fuse ebtable_nat ebtables xt_CHECKSUM bridge stp llc ppdev parport_pc lp parport sunrpc cpufreq_ondemand acpi_cpufreq mperf bnep bluetooth xt_physdev ip6t_REJECT snd_hda_codec_analog snd_hda_intel snd_hda_codec snd_hwdep snd_seq snd_seq_device iTCO_wdt snd_pcm arc4 iwl3945 snd_timer iwl_legacy mac80211 iTCO_vendor_support thinkpad_acpi snd soundcore cfg80211 e1000e i2c_i801 joydev microcode snd_page_alloc rfkill virtio_net kvm_intel kvm uinput ipv6 xts gf128mul dm_crypt yenta_socket i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: stap_190d377c71fa6a45f476ce480c4912a1_10067]
[157018.228133] 
[157018.228133] Pid: 10302, comm: modprobe Not tainted 2.6.40.4-5.fc15.i686 #1 LENOVO 195144G/195144G
[157018.228133] EIP: 0060:[<c07ee524>] EFLAGS: 00010246 CPU: 0
[157018.228133] EIP is at register_kprobe+0x1ca/0x3a5
[157018.228133] EAX: 00000047 EBX: f8b827fc ECX: 00000046 EDX: 00000000
[157018.228133] ESI: e9856420 EDI: 00000000 EBP: d1693eb0 ESP: d1693e90
[157018.228133]  DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068
[157018.228133] Process modprobe (pid: 10302, ti=d1692000 task=dba09940 task.ti=d1692000)
[157018.228133] Stack:
[157018.228133]  c0939f72 f8942011 f8b6f1fe f8b82804 f8bfa5c4 f8b827fc f8b6f1f0 00002100
[157018.228133]  d1693ed4 f8b6370e f6173a40 00000246 d1693ee4 000000c0 f8b6d0e4 00000000
[157018.228133]  00000001 d1693ef0 c07ed017 f8bfa5c4 c0a3b7ec c0a398c0 ffffffff 00000460
[157018.228133] Call Trace:
[157018.228133]  [<f8942011>] ? jffs2_compressors_init+0x11/0x11 [jffs2]
[157018.228133]  [<f8b6370e>] _stp_module_notifier+0x1a0/0x26e [stap_1aa19d6453909e68c4758c58d1d67725_10296]
[157018.228133]  [<c07ed017>] notifier_call_chain+0x2b/0x4d
[157018.228133]  [<c04582e6>] __blocking_notifier_call_chain+0x40/0x52
[157018.228133]  [<c0458317>] blocking_notifier_call_chain+0x1f/0x21
[157018.228133]  [<c0469ea0>] sys_init_module+0x1384/0x15af
[157018.228133]  [<c07e9f54>] syscall_call+0x7/0xb
[157018.228133] Code: 39 46 40 75 13 8b 46 14 c7 04 24 72 9f 93 c0 89 44 24 04 e8 08 41 ff ff 83 66 3c fd 81 7e 20 9c d1 7e c0 75 06 83 7e 50 00 75 02 <0f> 0b 89 f0 e8 1c f5 ff ff f6 46 3c 01 74 22 89 f0 e8 e6 da ff 
[157018.228133] EIP: [<c07ee524>] register_kprobe+0x1ca/0x3a5 SS:ESP 0068:d1693e90
[157018.293049] ---[ end trace f0f649cc63eb195a ]---
Comment 3 Frank Ch. Eigler 2011-11-06 16:09:12 UTC
cc:ing Masami because of the optimized-kprobes kernel warnings.
Comment 4 Frank Ch. Eigler 2011-11-06 17:22:37 UTC
With echo 0 > /proc/sys/debug/kprobes-optimization, the test, executed
in a tight loop with even the "sleep 3;" part removed from bz6503.exp,
runs fine for hours.

# while true; do make installcheck RUNTESTFLAGS=bz6503.exp; done

With echo 1, the test loop fails within a minute or two, on this VM:
Linux vm-f15-64 2.6.40.6-0.fc15.x86_64 #1 SMP Tue Oct 4 00:39:50 UTC 2011 x86_64 x86_64 x86_64 GNU/Linux
2 VCPU, 2GB RAM.

I strongly suspect this is a kernel bug that we'll just have to work
around somehow, perhaps by teaching staprun to turn off kprobes optimization.
Comment 5 Masami Hiramatsu 2011-11-10 14:28:08 UTC
(In reply to comment #4)
> I strongly suspect this is a kernel bug that we'll just have to work
> around somehow, perhaps by teaching staprun to turn off kprobes optimization.

Hmm, it seems that an aggr probe(which wasn't optimized) was gone and child probes were also gone, but the aggr probe was not released and on the kprobe's hash table.

BTW, I'd like to get more information:
 - does this happen on x86-64 too?
 - does this happen on upstream kernel too?
 - what the actual version of systemtap did you use?
 - is there any C code of stap code?

I'm not sure why systemtap tries to put a probe on the "TAIL" of a function in .init.text...

[157018.227386] Warning: found a stray unused aggrprobe@f8942011
[157018.228133]  [<f8942011>] ? jffs2_compressors_init+0x11/0x11 [jffs2]
Comment 6 Frank Ch. Eigler 2012-02-21 15:11:41 UTC
commit cd125e94 included kprobes-optimization disabling.
Once lkml upstream fixes the associated problem, and we find
a way of identifying the fixed releases, it can be removed,
or made further conditional.  (Note that one can test stap 1.7
against possible kernel fixes with env STAP_PR13193_OVERRIDE=1.)