This happens on both "real hardware" with x86_64 f14 (2.6.35.14-96.fc14.x86_64) and under kvm with x86_64 rhel6 (2.6.32-131.12.1.el6.x86_64). The following is the result of running make installcheck RUNTESTFLAGS=bz6503.exp on the rhel6 kvm instance, with echo 9 > /proc/sysrq-trigger and adding -DDEBUG_SYMBOLS to the testcase: SysRq : Changing Loglevel Loglevel set to 9 stap_42a0620f5f7fd9b83fd77b5a99e4b3d7__2646: systemtap: 1.7/0.152, base: ffffffffa0507000, memory: 221data/20text/2ctx/10net/106alloc kb, probes: 373 JFFS2 version 2.2. (NAND) (SUMMARY) © 2001-2006 Red Hat, Inc. BUG: soft lockup - CPU#0 stuck for 67s! [migration/0:5] Modules linked in: jffs2(+) ext2(+) zlib_deflate stap_42a0620f5f7fd9b83fd77b5a99e4b3d7__2646(U) ebtable_nat ebtables ipt_MASQUERADE iptable_nat BUG: soft lockup - CPU#1 stuck for 67s! [migration/1:8] Modules linked in: jffs2(+) ext2(+) zlib_deflate stap_42a0620f5f7fd9b83fd77b5a99e4b3d7__2646(U) ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat xt_CHECKSUM iptable_mangle bridge stp llc autofs4 nfs lockd fscache(T) nfs_acl auth_rpcgss sunrpc xt_physdev ipt_REJECT nf_conntrack_ipv4 nf_defrag_ipv4 iptable_filter ip_tables ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables ipv6 dm_mirror dm_region_hash dm_log vhost_net macvtap macvlan tun uinput sg microcode virtio_balloon snd_intel8x0 snd_ac97_codec ac97_bus snd_seq snd_seq_device snd_pcm snd_timer snd soundcore snd_page_alloc virtio_net i2c_piix4 i2c_core ext4 mbcache jbd2 virtio_blk sr_mod cdrom virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mod [last unloaded: speedstep_lib] CPU 1: Modules linked in: jffs2(+) ext2(+) zlib_deflate stap_42a0620f5f7fd9b83fd77b5a99e4b3d7__2646(U) ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat xt_CHECKSUM iptable_mangle bridge stp llc autofs4 nfs lockd fscache(T) nfs_acl auth_rpcgss sunrpc xt_physdev ipt_REJECT nf_conntrack_ipv4 nf_defrag_ipv4 iptable_filter ip_tables ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables ipv6 dm_mirror dm_region_hash dm_log vhost_net macvtap macvlan tun uinput sg microcode virtio_balloon snd_intel8x0 snd_ac97_codec ac97_bus snd_seq snd_seq_device snd_pcm snd_timer snd soundcore snd_page_alloc virtio_net i2c_piix4 i2c_core ext4 mbcache jbd2 virtio_blk sr_mod cdrom virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mod [last unloaded: speedstep_lib] Pid: 8, comm: migration/1 Tainted: G ---------------- T 2.6.32-131.12.1.el6.x86_64 #1 Bochs RIP: 0010:[<ffffffff810c36f4>] [<ffffffff810c36f4>] stop_machine_cpu_stop+0x74/0xe0 RSP: 0000:ffff88009cbd5dd0 EFLAGS: 00000293 RAX: 0000000000000001 RBX: ffff88009cbd5df0 RCX: ffff8800022911e8 RDX: 0000000000000000 RSI: ffff880099994100 RDI: ffff880099079c18 RBP: ffffffff8100bc8e R08: ffff88009cbd4000 R09: 0000000000000001 R10: 00000000ffffffff R11: 0000000000000000 R12: ffff88009cbd5d50 R13: ffffffff814dad57 R14: ffff88009cbd5df0 R15: ffff88009ac86f00 FS: 0000000000000000(0000) GS:ffff880002280000(0000) knlGS:0000000000000000 CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b CR2: 00000031ae41b397 CR3: 0000000037335000 CR4: 00000000000006e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Call Trace: [<ffffffff810c3680>] ? stop_machine_cpu_stop+0x0/0xe0 [<ffffffff810c35aa>] ? cpu_stopper_thread+0xda/0x1b0 [<ffffffff814dad57>] ? thread_return+0x4e/0x777 [<ffffffff8105dc32>] ? default_wake_function+0x12/0x20 [<ffffffff810c34d0>] ? cpu_stopper_thread+0x0/0x1b0 [<ffffffff8108de16>] ? kthread+0x96/0xa0 [<ffffffff8100c1ca>] ? child_rip+0xa/0x20 [<ffffffff8108dd80>] ? kthread+0x0/0xa0 [<ffffffff8100c1c0>] ? child_rip+0x0/0x20 BUG: soft lockup - CPU#2 stuck for 67s! [modprobe:2652] Modules linked in: jffs2(+) ext2(+) zlib_deflate stap_42a0620f5f7fd9b83fd77b5a99e4b3d7__2646(U) ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat xt_CHECKSUM iptable_mangle bridge stp llc autofs4 nfs lockd fscache(T) nfs_acl auth_rpcgss sunrpc xt_physdev ipt_REJECT nf_conntrack_ipv4 nf_defrag_ipv4 iptable_filter ip_tables ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables ipv6 dm_mirror dm_region_hash dm_log vhost_net macvtap macvlan tun uinput sg microcode virtio_balloon snd_intel8x0 snd_ac97_codec ac97_bus snd_seq snd_seq_device snd_pcm snd_timer snd soundcore snd_page_alloc virtio_net i2c_piix4 i2c_core ext4 mbcache jbd2 virtio_blk sr_mod cdrom virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mod [last unloaded: speedstep_lib] CPU 2: Modules linked in: jffs2(+) ext2(+) zlib_deflate stap_42a0620f5f7fd9b83fd77b5a99e4b3d7__2646(U) ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat xt_CHECKSUM iptable_mangle bridge stp llc autofs4 nfs lockd fscache(T) nfs_acl auth_rpcgss sunrpc xt_physdev ipt_REJECT nf_conntrack_ipv4 nf_defrag_ipv4 iptable_filter ip_tables ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables ipv6 dm_mirror dm_region_hash dm_log vhost_net macvtap macvlan tun uinput sg microcode virtio_balloon snd_intel8x0 snd_ac97_codec ac97_bus snd_seq snd_seq_device snd_pcm snd_timer snd soundcore snd_page_alloc virtio_net i2c_piix4 i2c_core ext4 mbcache jbd2 virtio_blk sr_mod cdrom virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mod [last unloaded: speedstep_lib] Pid: 2652, comm: modprobe Tainted: G ---------------- T 2.6.32-131.12.1.el6.x86_64 #1 Bochs RIP: 0010:[<ffffffff810a4536>] [<ffffffff810a4536>] smp_call_function_many+0x1b6/0x210 RSP: 0018:ffff880098ccbcc8 EFLAGS: 00000202 RAX: 0000000000000011 RBX: ffff880098ccbd08 RCX: 0000000000000004 RDX: 0000000000000010 RSI: 0000000000000000 RDI: 0000000000000286 RBP: ffffffff8100bc8e R08: 0000000000000000 R09: ffff880037bc0c00 R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000002 R13: ffff880098ccbc78 R14: 0000000000000000 R15: 0000000000000020 FS: 00007f5f3eebf700(0000) GS:ffff880002300000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00000000024ef0a8 CR3: 0000000099006000 CR4: 00000000000006e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Call Trace: [<ffffffff810a451a>] ? smp_call_function_many+0x19a/0x210 [<ffffffff81159560>] ? do_ccupdate_local+0x0/0x40 [<ffffffff810a45b2>] ? smp_call_function+0x22/0x30 [<ffffffff8106f2f4>] ? on_each_cpu+0x24/0x50 [<ffffffff8115c53f>] ? do_tune_cpucache+0x12f/0x630 [<ffffffff8115cc1b>] ? enable_cpucache+0x3b/0xf0 [<ffffffff814c575f>] ? setup_cpu_cache+0x22f/0x340 [<ffffffff8115b1b2>] ? kmem_cache_alloc+0x182/0x190 [<ffffffff8115d8aa>] ? kmem_cache_create+0x3fa/0x580 [<ffffffffa008c184>] ? init_jffs2_fs+0x0/0xc7 [jffs2] [<ffffffffa0362510>] ? jffs2_i_init_once+0x0/0x40 [jffs2] [<ffffffffa008c184>] ? init_jffs2_fs+0x0/0xc7 [jffs2] [<ffffffffa008c1ba>] ? init_jffs2_fs+0x36/0xc7 [jffs2] [<ffffffff8100204c>] ? do_one_initcall+0x3c/0x1d0 [<ffffffff810aca7f>] ? sys_init_module+0xdf/0x250 [<ffffffff8100b172>] ? system_call_fastpath+0x16/0x1b nf_nat xt_CHECKSUM iptable_mangle bridge stp llc autofs4 nfs lockd fscache(T) nfs_acl auth_rpcgss sunrpc xt_physdev ipt_REJECT nf_conntrack_ipv4 nf_defrag_ipv4 iptable_filter ip_tables ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables ipv6 dm_mirror dm_region_hash dm_log vhost_net macvtap macvlan tun uinput sg microcode virtio_balloon snd_intel8x0 snd_ac97_codec ac97_bus snd_seq snd_seq_device snd_pcm snd_timer snd soundcore snd_page_alloc virtio_net i2c_piix4 i2c_core ext4 mbcache jbd2 virtio_blk sr_mod cdrom virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mod [last unloaded: speedstep_lib] CPU 0: Modules linked in: jffs2(+) ext2(+) zlib_deflate stap_42a0620f5f7fd9b83fd77b5a99e4b3d7__2646(U) ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat xt_CHECKSUM iptable_mangle bridge stp llc autofs4 nfs lockd fscache(T) nfs_acl auth_rpcgss sunrpc xt_physdev ipt_REJECT nf_conntrack_ipv4 nf_defrag_ipv4 iptable_filter ip_tables ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables ipv6 dm_mirror dm_region_hash dm_log vhost_net macvtap macvlan tun uinput sg microcode virtio_balloon snd_intel8x0 snd_ac97_codec ac97_bus snd_seq snd_seq_device snd_pcm snd_timer snd soundcore snd_page_alloc virtio_net i2c_piix4 i2c_core ext4 mbcache jbd2 virtio_blk sr_mod cdrom virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mod [last unloaded: speedstep_lib] Pid: 5, comm: migration/0 Tainted: G ---------------- T 2.6.32-131.12.1.el6.x86_64 #1 Bochs RIP: 0010:[<ffffffff810c36f8>] [<ffffffff810c36f8>] stop_machine_cpu_stop+0x78/0xe0 RSP: 0018:ffff88009cbcbdd0 EFLAGS: 00000293 RAX: 0000000000000001 RBX: ffff88009cbcbdf0 RCX: ffff8800022111e8 RDX: 0000000000000000 RSI: ffff8800953d4ac0 RDI: ffff880099079c18 RBP: ffffffff8100bc8e R08: ffff88009cbca000 R09: 0000000000000001 R10: 00000000ffffffff R11: 0000000000000000 R12: ffff88009cbcbd50 R13: ffffffff814dad57 R14: ffff88009cbcbdf0 R15: ffff880037763200 FS: 0000000000000000(0000) GS:ffff880002200000(0000) knlGS:0000000000000000 CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b CR2: 00000031af203088 CR3: 0000000098cc3000 CR4: 00000000000006f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Call Trace: [<ffffffff810c3680>] ? stop_machine_cpu_stop+0x0/0xe0 [<ffffffff810c35aa>] ? cpu_stopper_thread+0xda/0x1b0 [<ffffffff814dad57>] ? thread_return+0x4e/0x777 [<ffffffff8105dc32>] ? default_wake_function+0x12/0x20 [<ffffffff810c34d0>] ? cpu_stopper_thread+0x0/0x1b0 [<ffffffff8108de16>] ? kthread+0x96/0xa0 [<ffffffff8100c1ca>] ? child_rip+0xa/0x20 [<ffffffff8108dd80>] ? kthread+0x0/0xa0 [<ffffffff8100c1c0>] ? child_rip+0x0/0x20 BUG: soft lockup - CPU#0 stuck for 67s! [migration/0:5] Modules linked in: jffs2(+) ext2(+) zlib_deflate stap_42a0620f5f7fd9b83fd77b5a99e4b3d7__2646(U) ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat BUG: soft lockup - CPU#1 stuck for 67s! [migration/1:8] Modules linked in: jffs2(+) ext2(+) zlib_deflate stap_42a0620f5f7fd9b83fd77b5a99e4b3d7__2646(U) ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat xt_CHECKSUM iptable_mangle bridge stp llc autofs4 nfs lockd fscache(T) nfs_acl auth_rpcgss sunrpc xt_physdev ipt_REJECT nf_conntrack_ipv4 nf_defrag_ipv4 iptable_filter ip_tables ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables ipv6 dm_mirror dm_region_hash dm_log vhost_net macvtap macvlan tun uinput sg microcode virtio_balloon snd_intel8x0 snd_ac97_codec ac97_bus snd_seq snd_seq_device snd_pcm snd_timer snd soundcore snd_page_alloc virtio_net i2c_piix4 i2c_core ext4 mbcache jbd2 virtio_blk sr_mod cdrom virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mod [last unloaded: speedstep_lib] CPU 1: Modules linked in: jffs2(+) ext2(+) zlib_deflate stap_42a0620f5f7fd9b83fd77b5a99e4b3d7__2646(U) ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat xt_CHECKSUM iptable_mangle bridge stp llc autofs4 nfs lockd fscache(T) nfs_acl auth_rpcgss sunrpc xt_physdev ipt_REJECT nf_conntrack_ipv4 nf_defrag_ipv4 iptable_filter ip_tables ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables ipv6 dm_mirror dm_region_hash dm_log vhost_net macvtap macvlan tun uinput sg microcode virtio_balloon snd_intel8x0 snd_ac97_codec ac97_bus snd_seq snd_seq_device snd_pcm snd_timer snd soundcore snd_page_alloc virtio_net i2c_piix4 i2c_core ext4 mbcache jbd2 virtio_blk sr_mod cdrom virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mod [last unloaded: speedstep_lib] Pid: 8, comm: migration/1 Tainted: G ---------------- T 2.6.32-131.12.1.el6.x86_64 #1 Bochs RIP: 0010:[<ffffffff810c36f4>] [<ffffffff810c36f4>] stop_machine_cpu_stop+0x74/0xe0 RSP: 0000:ffff88009cbd5dd0 EFLAGS: 00000293 RAX: 0000000000000001 RBX: ffff88009cbd5df0 RCX: ffff8800022911e8 RDX: 0000000000000000 RSI: ffff880099994100 RDI: ffff880099079c18 RBP: ffffffff8100bc8e R08: ffff88009cbd4000 R09: 0000000000000001 R10: 00000000ffffffff R11: 0000000000000000 R12: ffff88009cbd5d50 R13: ffffffff814dad57 R14: ffff88009cbd5df0 R15: ffff88009ac86f00 FS: 0000000000000000(0000) GS:ffff880002280000(0000) knlGS:0000000000000000 CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b CR2: 00000031ae41b397 CR3: 0000000037335000 CR4: 00000000000006e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Call Trace: [<ffffffff810c3680>] ? stop_machine_cpu_stop+0x0/0xe0 [<ffffffff810c35aa>] ? cpu_stopper_thread+0xda/0x1b0 [<ffffffff814dad57>] ? thread_return+0x4e/0x777 [<ffffffff8105dc32>] ? default_wake_function+0x12/0x20 [<ffffffff810c34d0>] ? cpu_stopper_thread+0x0/0x1b0 [<ffffffff8108de16>] ? kthread+0x96/0xa0 [<ffffffff8100c1ca>] ? child_rip+0xa/0x20 [<ffffffff8108dd80>] ? kthread+0x0/0xa0 [<ffffffff8100c1c0>] ? child_rip+0x0/0x20 BUG: soft lockup - CPU#2 stuck for 67s! [modprobe:2652] Modules linked in: jffs2(+) ext2(+) zlib_deflate stap_42a0620f5f7fd9b83fd77b5a99e4b3d7__2646(U) ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat xt_CHECKSUM iptable_mangle bridge stp llc autofs4 nfs lockd fscache(T) nfs_acl auth_rpcgss sunrpc xt_physdev ipt_REJECT nf_conntrack_ipv4 nf_defrag_ipv4 iptable_filter ip_tables ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables ipv6 dm_mirror dm_region_hash dm_log vhost_net macvtap macvlan tun uinput sg microcode virtio_balloon snd_intel8x0 snd_ac97_codec ac97_bus snd_seq snd_seq_device snd_pcm snd_timer snd soundcore snd_page_alloc virtio_net i2c_piix4 i2c_core ext4 mbcache jbd2 virtio_blk sr_mod cdrom virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mod [last unloaded: speedstep_lib] CPU 2: Modules linked in: jffs2(+) ext2(+) zlib_deflate stap_42a0620f5f7fd9b83fd77b5a99e4b3d7__2646(U) ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat xt_CHECKSUM iptable_mangle bridge stp llc autofs4 nfs lockd fscache(T) nfs_acl auth_rpcgss sunrpc xt_physdev ipt_REJECT nf_conntrack_ipv4 nf_defrag_ipv4 iptable_filter ip_tables ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables ipv6 dm_mirror dm_region_hash dm_log vhost_net macvtap macvlan tun uinput sg microcode virtio_balloon snd_intel8x0 snd_ac97_codec ac97_bus snd_seq snd_seq_device snd_pcm snd_timer snd soundcore snd_page_alloc virtio_net i2c_piix4 i2c_core ext4 mbcache jbd2 virtio_blk sr_mod cdrom virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mod [last unloaded: speedstep_lib] Pid: 2652, comm: modprobe Tainted: G ---------------- T 2.6.32-131.12.1.el6.x86_64 #1 Bochs RIP: 0010:[<ffffffff810a4536>] [<ffffffff810a4536>] smp_call_function_many+0x1b6/0x210 RSP: 0018:ffff880098ccbcc8 EFLAGS: 00000202 RAX: 0000000000000011 RBX: ffff880098ccbd08 RCX: 0000000000000004 RDX: 0000000000000010 RSI: 0000000000000000 RDI: 0000000000000286 RBP: ffffffff8100bc8e R08: 0000000000000000 R09: ffff880037bc0c00 R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000002 R13: ffff880098ccbc78 R14: 0000000000000000 R15: 0000000000000020 FS: 00007f5f3eebf700(0000) GS:ffff880002300000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00000000024ef0a8 CR3: 0000000099006000 CR4: 00000000000006e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Call Trace: [<ffffffff810a451a>] ? smp_call_function_many+0x19a/0x210 [<ffffffff81159560>] ? do_ccupdate_local+0x0/0x40 [<ffffffff810a45b2>] ? smp_call_function+0x22/0x30 [<ffffffff8106f2f4>] ? on_each_cpu+0x24/0x50 [<ffffffff8115c53f>] ? do_tune_cpucache+0x12f/0x630 [<ffffffff8115cc1b>] ? enable_cpucache+0x3b/0xf0 [<ffffffff814c575f>] ? setup_cpu_cache+0x22f/0x340 [<ffffffff8115b1b2>] ? kmem_cache_alloc+0x182/0x190 [<ffffffff8115d8aa>] ? kmem_cache_create+0x3fa/0x580 [<ffffffffa008c184>] ? init_jffs2_fs+0x0/0xc7 [jffs2] [<ffffffffa0362510>] ? jffs2_i_init_once+0x0/0x40 [jffs2] [<ffffffffa008c184>] ? init_jffs2_fs+0x0/0xc7 [jffs2] [<ffffffffa008c1ba>] ? init_jffs2_fs+0x36/0xc7 [jffs2] [<ffffffff8100204c>] ? do_one_initcall+0x3c/0x1d0 [<ffffffff810aca7f>] ? sys_init_module+0xdf/0x250 [<ffffffff8100b172>] ? system_call_fastpath+0x16/0x1b xt_CHECKSUM iptable_mangle bridge stp llc autofs4 nfs lockd fscache(T) nfs_acl auth_rpcgss sunrpc xt_physdev ipt_REJECT nf_conntrack_ipv4 nf_defrag_ipv4 iptable_filter ip_tables ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables ipv6 dm_mirror dm_region_hash dm_log vhost_net macvtap macvlan tun uinput sg microcode virtio_balloon snd_intel8x0 snd_ac97_codec ac97_bus snd_seq snd_seq_device snd_pcm snd_timer snd soundcore snd_page_alloc virtio_net i2c_piix4 i2c_core ext4 mbcache jbd2 virtio_blk sr_mod cdrom virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mod [last unloaded: speedstep_lib] CPU 0: Modules linked in: jffs2(+) ext2(+) zlib_deflate stap_42a0620f5f7fd9b83fd77b5a99e4b3d7__2646(U) ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat xt_CHECKSUM iptable_mangle bridge stp llc autofs4 nfs lockd fscache(T) nfs_acl auth_rpcgss sunrpc xt_physdev ipt_REJECT nf_conntrack_ipv4 nf_defrag_ipv4 iptable_filter ip_tables ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables ipv6 dm_mirror dm_region_hash dm_log vhost_net macvtap macvlan tun uinput sg microcode virtio_balloon snd_intel8x0 snd_ac97_codec ac97_bus snd_seq snd_seq_device snd_pcm snd_timer snd soundcore snd_page_alloc virtio_net i2c_piix4 i2c_core ext4 mbcache jbd2 virtio_blk sr_mod cdrom virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mod [last unloaded: speedstep_lib] Pid: 5, comm: migration/0 Tainted: G ---------------- T 2.6.32-131.12.1.el6.x86_64 #1 Bochs RIP: 0010:[<ffffffff810c36f4>] [<ffffffff810c36f4>] stop_machine_cpu_stop+0x74/0xe0 RSP: 0018:ffff88009cbcbdd0 EFLAGS: 00000293 RAX: 0000000000000001 RBX: ffff88009cbcbdf0 RCX: ffff8800022111e8 RDX: 0000000000000000 RSI: ffff8800953d4ac0 RDI: ffff880099079c18 RBP: ffffffff8100bc8e R08: ffff88009cbca000 R09: 0000000000000001 R10: 00000000ffffffff R11: 0000000000000000 R12: ffff88009cbcbd50 R13: ffffffff814dad57 R14: ffff88009cbcbdf0 R15: ffff880037763200 FS: 0000000000000000(0000) GS:ffff880002200000(0000) knlGS:0000000000000000 CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b CR2: 00000031af203088 CR3: 0000000098cc3000 CR4: 00000000000006f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Call Trace: [<ffffffff810c3680>] ? stop_machine_cpu_stop+0x0/0xe0 [<ffffffff810c35aa>] ? cpu_stopper_thread+0xda/0x1b0 [<ffffffff814dad57>] ? thread_return+0x4e/0x777 [<ffffffff8105dc32>] ? default_wake_function+0x12/0x20 [<ffffffff810c34d0>] ? cpu_stopper_thread+0x0/0x1b0 [<ffffffff8108de16>] ? kthread+0x96/0xa0 [<ffffffff8100c1ca>] ? child_rip+0xa/0x20 [<ffffffff8108dd80>] ? kthread+0x0/0xa0 [<ffffffff8100c1c0>] ? child_rip+0x0/0x20
I tried to see if this was module insert/remove related. The following gets us a panic, although this might be unrelated. # cd /lib/modules/`uname -r` # for i in `find * -name \*.ko;`; do insmod $i & done test_nx: stack was executable test_nx: heap was executable wacom driver registered telclk_interrup = 0xf non-mcpbl0010 hw. i2c-parport-light: adapter type unspecified i2c-stub: Please specify a chip address Error: Driver 'pcspkr' is already registered, aborting... acpiphp_ibm: ibm_acpiphp_init: acpi_walk_namespace failed i2c i2c-0: failed,FFFF,FF,FFFF i2c i2c-0: failed,FFFF,FF,FFFF i2c i2c-0: Couldn't detect a adt7475 part at 0x2e Failed to ioremap_nocache w83697ug/uf WDT: No W83697UG/UF could be found BUG: unable to handle kernel NULL pointer dereference at (null) IP: [<ffffffff81269aab>] strcmp+0xb/0x30 PGD 800db067 PUD 800fa067 PMD 0 Oops: 0000 [#1] SMP last sysfs file: /sys/module/lirc_dev/initstate CPU 3 Modules linked in: sbc_fitpc2_wdt(+) machzwd(-) idmouse appledisplay usbsevseg usblcd cdc_wdm usblp ibmasr(-) w83697ug_wdt(-) xhci_hcd wm8350_wdt sl811_hcd umc w83877f_wdt it8712f_wdt(-) wm831x_wdt vstusb mtdconcat SoftDog: cannot register miscdev on minor=130 (err=-16) mtdchar qinfo_probe nand_ecc ftl redboot map_rom cfi_util ts5500_flash(-) WDT driver for the Winbond(TM) W83627HF/THF/HG Super I/O chip initialising. w83627hf/thf/hg WDT: Watchdog already running. Resetting timeout to 60 sec w83627hf/thf/hg WDT: cannot register miscdev on minor=130 (err=-16) ipw: Unknown symbol usb_serial_port_softint ipw: Unknown symbol usb_serial_disconnect ipw: Unknown symbol usb_serial_probe ipw: Unknown symbol usb_serial_register ipw: Unknown symbol usb_serial_deregister visor: Unknown symbol usb_serial_port_softint visor: Unknown symbol usb_serial_disconnect visor: Unknown symbol usb_serial_probe visor: Unknown symbol usb_serial_register visor: Unknown symbol usb_serial_deregister sierra: Unknown symbol usb_serial_port_softint sierra: Unknown symbol usb_serial_disconnect sierra: Unknown symbol usb_serial_suspend sierra: Unknown symbol usb_serial_probe sierra: Unknown symbol usb_serial_register sierra: Unknown symbol usb_serial_deregister sierra: Unknown symbol usb_serial_resume map_ram gen_probe pci sisusbvga iTCO_vendor_support netsc520(-) ck804xrom(-) mtdoops i1480_dfu_usb usbled emi62 nand_ids ark3116: Unknown symbol usb_serial_disconnect ark3116: Unknown symbol usb_serial_generic_open ark3116: Unknown symbol usb_serial_probe ark3116: Unknown symbol usb_serial_register ark3116: Unknown symbol usb_serial_deregister usbcore: registered new interface driver microtekX6 pl2303: Unknown symbol usb_serial_port_softint pl2303: Unknown symbol usb_serial_handle_sysrq_char pl2303: Unknown symbol usb_serial_disconnect pl2303: Unknown symbol usb_serial_suspend pl2303: Unknown symbol usb_serial_handle_break pl2303: Unknown symbol usb_serial_probe pl2303: Unknown symbol usb_serial_register pl2303: Unknown symbol usb_serial_deregister pl2303: Unknown symbol usb_serial_resume qcserial: Unknown symbol usb_wwan_write qcserial: Unknown symbol usb_wwan_suspend qcserial: Unknown symbol usb_wwan_close qcserial: Unknown symbol usb_wwan_release qcserial: Unknown symbol usb_serial_disconnect qcserial: Unknown symbol usb_wwan_chars_in_buffer qcserial: Unknown symbol usb_serial_suspend qcserial: Unknown symbol usb_wwan_write_room qcserial: Unknown symbol usb_wwan_disconnect qcserial: Unknown symbol usb_wwan_startup qcserial: Unknown symbol usb_wwan_resume qcserial: Unknown symbol usb_wwan_open qcserial: Unknown symbol usb_serial_probe qcserial: Unknown symbol usb_serial_register qcserial: Unknown symbol usb_serial_deregister qcserial: Unknown symbol usb_serial_resume belkin_sa: Unknown symbol usb_serial_disconnect belkin_sa: Unknown symbol usb_serial_probe belkin_sa: Unknown symbol usb_serial_register belkin_sa: Unknown symbol usb_serial_deregister mtdblock_ro scb2_flash ricoh_mmc mtdram map_absent ar7part block2mtd lpddr_cmds inftl mtd_blkdevs wm8350 wm831x dell_wmi adt7475 asb100 ltc4215 f75375s i5k_amb lm75 lis3lv02d g760a sis5595 lm80 k10temp ibmpex ds1621 hwmon_vid asus_atk0110 gru adm1029 cciss hwmon i2c_amd8111 i2c_amd756 tsl2550 i2c_viapro i2c_voodoo3 i2c_tiny_usb i2c_via i2c_sis96x ir_lirc_codec i2c_scmi i2c_simtec i2c_pca_platform i2c_isch i2c_dev pcf8591 ir_jvc_decoder twofish_x86_64 keyspan_remote ir_rc5_decoder camellia des_generic bcm5974 synaptics_i2c atlas_btns kbtab appletouch i2c_algo_bit i2c_algo_pca rc_apac_viewcomp acecad yealink sermouse dvb_core cyclades ipmi_msghandler rc_avermedia_m733a_rm_k6 rc_avermedia_m135a cryptoloop dvb_usb_af9005_remote kvm virtio_console xt_statistic aiptek ad7879 isl29003 rc_avermedia ics932s401 memstick tifm_7xx1 ioc4 ab3100_core eeprom sm501 floppy wm8400_core tifm_core at24 cb710 enclosure cls_basic snd_usb_lib hpwdt rc_adstech_dvb_t_pci tua6100 snd_ice1724 elo snd_hda_codec_conexant snd_indigodj sx8 isl6405 isl6421 snd_seq_oss max6875 hpilo eeprom_93cx6 mfd_core soc_camera_platform snd_usb_caiaq rc_lirc snd_rme96 usbtouchscreen fujitsu_ts max17040_battery airo fujitsu_laptop panasonic_laptop wmi topstar_laptop bq27x00_battery aer_inject zd1201 rc_alink_dtu_m snd_emu10k1 snd_ens1371 pps_core inexio mtouch touchright touchit213 tsc2007 penmount mcs5000_ts gunze ati_remote ir_sony_decoder videodev gtco serio_raw ati_remote2 vsxxxaa eeti_ts opencores_kbd max7359_keypad v4l2_compat_ioctl32 v4l2_int_device videobuf_vmalloc videobuf_dma_sg ir_rc5_sz_decoder snd_ctxfi videobuf_core snd_lx6464es snd_mixart snd_atiixp snd_pcxhr pktcdvd rc_ati_tv_wonder_hd_600 rc_avermedia_a16d rc_asus_pc39 wacom_w8001 rc_anysee snd_hda_codec fakephp touchwin ir_rc6_decoder i2c_nforce2 i2c_i801 nls_cp861 nls_cp775 lxt powermate adp5588_keys input_polldev soc_mediabus ir_kbd_i2c intel_ips snd_cs46xx snd_darla24 snd_ca0106 snd_hdspm snd_korg1212 rc_dntv_live_dvb_t rc_digittrade rc_cinergy_1400 rc_fusionhdtv_mce lirc_dev mceusb ene_ir ir_nec_decoder streamzap winbond_cir snd_hdsp ttusbdecfe snd_indigoiox snd_indigodjx snd_mona snd_mia rsrc_nonstatic btcx_risc nls_cp855 tveeprom act_ipt rc_msi_digivox_iii rc_msi_digivox_ii rc_iodata_bctv7e rc_terratec_slim rc_imon_pad rc_total_media_in_hand rc_videomate_m1f rc_hauppauge_new rc_kworld_315u rc_nebula rc_real_audio_220_32_keys rc_tevii_nec rc_terratec_cinergy_xs rc_tbs_nec rc_core dib3000mb snd_echo3g snd_layla24 nxt200x s5h1411 si21xx tda1004x s921 s5h1420 rtc_m48t35 lgs8gxx mb86a20s dib0070 cx24116 bcm3510 dibx000_common dib0090 ec100 cx24123 dvb_pll l64781 tda8083 ves1x93 zl10353 isl6423 af9013 tda18271 tda10021 mt352 stb6100 stv0297 sp887x sp8870 tda10023 lnbp21 edd tuner_xc2028 mt312 cx24110 cx24113 cx22702 cx22700 itd1000 atbm8830 pata_hpt3x2n scsi_wait_scan sata_inic162x dcdbas snd_rawmidi snd_sb_common snd_aloop max2165 snd_ali5451 tea5767 mt2131 mt20xx mc44s803 tda18218 tda9887 qt1010 mxl5005s tuner_simple mxl5007t tda827x tda8290 tea5761 mt2060 raid0 mt2266 xc5000 tuner_types snd_gina20 snd_darla20 snd_vx_lib hid_wacom ix2505v stb6000 ds3000 tda10086 tda8261 ves1820 zl10036 s5h1409 tda826x stv6110 nxt6000 zl10039 stv090x tda10048 stv6110x stv0288 sbs twofish_common seed serpent sbshc cpufreq_powersave hed erst_dbg virtio_rng timeriomem_rng hangcheck_timer tpm_infineon cpufreq_ondemand dell_rbu cm4000_cs bcm203x snd_rme9652 snd_hrtimer snd_indigo snd_indigoio snd_gina24 ttpci_eeprom cpufreq_stats cm4040_cs n_hdlc cpufreq_conservative snd_bt87x snd_seq_midi_emul snd_seq_midi_event snd_ice17xx_ak4xxx snd_cs8427 snd_cs5535audio snd_ak4114 snd_pcsp snd_pt2258 snd_ak4xxx_adda snd_i2c snd_dummy snd_seq_dummy snd_hwdep snd_intel8x0m snd_rme32 snd_maestro3 snd_ad1889 snd_atiixp_modem snd_via82xx_modem michael_mic sha512_generic xor wp512 tcrypt(+) sha256_generic arc4 ecb gf128mul cast6 fcrypt tgr192 authenc seqiv cast5 rmd128 tea cts rmd160 ccm rmd320 snd_util_mem oprofile khazad pcbc rmd256 vmac gcm xcbc cryptd anubis test_nx blowfish mce_inject freq_table speedstep_lib ansi_cprng(+) md4 cbc ctr salsa20_x86_64 crypto_null async_tx fpu ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat xt_CHECKSUM iptable_mangle bridge stp llc autofs4 nfs lockd fscache(T) nfs_acl auth_rpcgss sunrpc xt_physdev ipt_REJECT nf_conntrack_ipv4 nf_defrag_ipv4 iptable_filter ip_tables ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables ipv6 dm_mirror dm_region_hash dm_log vhost_net macvtap macvlan tun uinput sg microcode virtio_balloon snd_intel8x0 snd_ac97_codec ac97_bus snd_seq snd_seq_device snd_pcm snd_timer snd soundcore snd_page_alloc virtio_net i2c_piix4 i2c_core ext4 mbcache jbd2 virtio_blk sr_mod cdrom virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mod [last unloaded: speedstep_lib] Modules linked in: microtek w83627hf_wdt(-) softdog(-) whci sbc_fitpc2_wdt(+) machzwd(-) idmouse appledisplay usbsevseg usblcd cdc_wdm usblp ibmasr(-) w83697ug_wdt(-) xhci_hcd wm8350_wdt sl811_hcd umc w83877f_wdt it8712f_wdt(-) wm831x_wdt vstusb mtdconcat mtdchar qinfo_probe nand_ecc ftl redboot map_rom cfi_util ts5500_flash(-) map_ram gen_probe pci sisusbvga iTCO_vendor_support netsc520(-) ck804xrom(-) mtdoops i1480_dfu_usb usbled emi62 nand_ids mtdblock_ro scb2_flash ricoh_mmc mtdram map_absent ar7part block2mtd lpddr_cmds inftl mtd_blkdevs wm8350 wm831x dell_wmi adt7475 asb100 ltc4215 f75375s i5k_amb lm75 lis3lv02d g760a sis5595 lm80 k10temp ibmpex ds1621 hwmon_vid asus_atk0110 gru adm1029 cciss hwmon i2c_amd8111 i2c_amd756 tsl2550 i2c_viapro i2c_voodoo3 i2c_tiny_usb i2c_via i2c_sis96x ir_lirc_codec i2c_scmi i2c_simtec i2c_pca_platform i2c_isch i2c_dev pcf8591 ir_jvc_decoder twofish_x86_64 keyspan_remote ir_rc5_decoder camellia des_generic bcm5974 synaptics_i2c atlas_btns kbtab appletouch i2c_algo_bit i2c_algo_pca rc_apac_viewcomp acecad yealink sermouse dvb_core cyclades ipmi_msghandler rc_avermedia_m733a_rm_k6 rc_avermedia_m135a cryptoloop dvb_usb_af9005_remote kvm virtio_console xt_statistic aiptek ad7879 isl29003 rc_avermedia ics932s401 memstick tifm_7xx1 ioc4 ab3100_core eeprom sm501 floppy wm8400_core tifm_core at24 cb710 enclosure cls_basic snd_usb_lib hpwdt rc_adstech_dvb_t_pci tua6100 snd_ice1724 elo snd_hda_codec_conexant snd_indigodj sx8 isl6405 isl6421 snd_seq_oss max6875 hpilo eeprom_93cx6 mfd_core soc_camera_platform snd_usb_caiaq rc_lirc snd_rme96 usbtouchscreen fujitsu_ts max17040_battery airo fujitsu_laptop panasonic_laptop wmi topstar_laptop bq27x00_battery aer_inject zd1201 rc_alink_dtu_m snd_emu10k1 snd_ens1371 pps_core inexio mtouch touchright touchit213 tsc2007 penmount mcs5000_ts gunze ati_remote ir_sony_decoder videodev gtco serio_raw ati_remote2 vsxxxaa eeti_ts opencores_kbd max7359_keypad v4l2_compat_ioctl32 v4l2_int_device videobuf_vmalloc videobuf_dma_sg ir_rc5_sz_decoder snd_ctxfi videobuf_core snd_lx6464es snd_mixart snd_atiixp snd_pcxhr pktcdvd rc_ati_tv_wonder_hd_600 rc_avermedia_a16d rc_asus_pc39 wacom_w8001 rc_anysee snd_hda_codec fakephp touchwin ir_rc6_decoder i2c_nforce2 i2c_i801 nls_cp861 nls_cp775 lxt powermate adp5588_keys input_polldev soc_mediabus ir_kbd_i2c intel_ips snd_cs46xx snd_darla24 snd_ca0106 snd_hdspm snd_korg1212 rc_dntv_live_dvb_t rc_digittrade rc_cinergy_1400 rc_fusionhdtv_mce lirc_dev mceusb ene_ir ir_nec_decoder streamzap winbond_cir snd_hdsp ttusbdecfe snd_indigoiox snd_indigodjx snd_mona snd_mia rsrc_nonstatic btcx_risc nls_cp855 tveeprom act_ipt rc_msi_digivox_iii rc_msi_digivox_ii rc_iodata_bctv7e rc_terratec_slim rc_imon_pad rc_total_media_in_hand rc_videomate_m1f rc_hauppauge_new rc_kworld_315u rc_nebula rc_real_audio_220_32_keys rc_tevii_nec rc_terratec_cinergy_xs rc_tbs_nec rc_core dib3000mb snd_echo3g snd_layla24 nxt200x s5h1411 si21xx tda1004x s921 s5h1420 rtc_m48t35 lgs8gxx mb86a20s dib0070 cx24116 bcm3510 dibx000_common dib0090 ec100 cx24123 dvb_pll l64781 tda8083 ves1x93 zl10353 isl6423 af9013 tda18271 tda10021 mt352 stb6100 stv0297 sp887x sp8870 tda10023 lnbp21 edd tuner_xc2028 mt312 cx24110 cx24113 cx22702 cx22700 itd1000 atbm8830 pata_hpt3x2n scsi_wait_scan sata_inic162x dcdbas snd_rawmidi snd_sb_common snd_aloop max2165 snd_ali5451 tea5767 mt2131 mt20xx mc44s803 tda18218 tda9887 qt1010 mxl5005s tuner_simple mxl5007t tda827x tda8290 tea5761 mt2060 raid0 mt2266 xc5000 tuner_types snd_gina20 snd_darla20 snd_vx_lib hid_wacom ix2505v stb6000 ds3000 tda10086 tda8261 ves1820 zl10036 s5h1409 tda826x stv6110 nxt6000 zl10039 stv090x tda10048 stv6110x stv0288 sbs twofish_common seed serpent sbshc cpufreq_powersave hed erst_dbg virtio_rng timeriomem_rng hangcheck_timer tpm_infineon cpufreq_ondemand dell_rbu cm4000_cs bcm203x snd_rme9652 snd_hrtimer snd_indigo snd_indigoio snd_gina24 ttpci_eeprom cpufreq_stats cm4040_cs n_hdlc cpufreq_conservative snd_bt87x snd_seq_midi_emul snd_seq_midi_event snd_ice17xx_ak4xxx snd_cs8427 snd_cs5535audio snd_ak4114 snd_pcsp snd_pt2258 snd_ak4xxx_adda snd_i2c snd_dummy snd_seq_dummy snd_hwdep snd_intel8x0m snd_rme32 snd_maestro3 snd_ad1889 snd_atiixp_modem snd_via82xx_modem michael_mic sha512_generic xor wp512 tcrypt(+) sha256_generic arc4 ecb gf128mul cast6 fcrypt tgr192 authenc seqiv cast5 rmd128 tea cts rmd160 ccm rmd320 snd_util_mem oprofile khazad pcbc rmd256 vmac gcm xcbc cryptd anubis test_nx blowfish mce_inject freq_table speedstep_lib ansi_cprng(+) md4 cbc ctr salsa20_x86_64 crypto_null async_tx fpu ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat xt_CHECKSUM iptable_mangle bridge stp llc autofs4 nfs lockd fscache(T) nfs_acl auth_rpcgss sunrpc xt_physdev ipt_REJECT nf_conntrack_ipv4 nf_defrag_ipv4 iptable_filter ip_tables ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables ipv6 dm_mirror dm_region_hash dm_log vhost_net macvtap macvlan tun uinput sg microcode virtio_balloon snd_intel8x0 snd_ac97_codec ac97_bus snd_seq snd_seq_device snd_pcm snd_timer snd soundcore snd_page_alloc virtio_net i2c_piix4 i2c_core ext4 mbcache jbd2 virtio_blk sr_mod cdrom virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mod [last unloaded: speedstep_lib] Pid: 3166, comm: insmod Tainted: G ---------------- T 2.6.32-131.12.1.el6.x86_64 #1 Bochs RIP: 0010:[<ffffffff81269aab>] [<ffffffff81269aab>] strcmp+0xb/0x30 RSP: 0018:ffff8800800f7ef8 EFLAGS: 00010292 RAX: 0000000000000053 RBX: 0000000000000000 RCX: 0000000000000000 RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffffffffa23d66e8 RBP: ffff8800800f7ef8 R08: 0000000000000000 R09: 0000000000000001 R10: ffff88000240a3c0 R11: 0000000000000000 R12: ffffffffa23d9000 R13: 000000000162a010 R14: 000000000162a030 R15: 0000000000000003 FS: 00007fc032497700(0000) GS:ffff880002380000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 0000000000000000 CR3: 00000000800d8000 CR4: 00000000000006e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process insmod (pid: 3166, threadinfo ffff8800800f6000, task ffff8800800e6b40) Stack: ffff8800800f7f18 ffffffffa23d9022 ffff8800800f7f18 0000000000000000 <0> ffff8800800f7f48 ffffffff8100204c 000000000162a030 ffffffffa23d6860 <0> 00000000000031b0 000000000162a010 ffff8800800f7f78 ffffffff810aca7f Call Trace: [<ffffffffa23d9022>] fitpc2_wdt_init+0x22/0x146 [sbc_fitpc2_wdt] [<ffffffff8100204c>] do_one_initcall+0x3c/0x1d0 [<ffffffff810aca7f>] sys_init_module+0xdf/0x250 [<ffffffff8100b172>] system_call_fastpath+0x16/0x1b Code: 84 ff 40 88 39 74 0d 48 83 c1 01 48 83 ea 01 75 e7 c6 01 00 c9 c3 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 0f 1f 40 00 0f b6 07 <0f> b6 16 48 83 c7 01 48 83 c6 01 38 d0 75 0e 84 c0 75 ea 31 c0 RIP [<ffffffff81269aab>] strcmp+0xb/0x30 RSP <ffff8800800f7ef8> CR2: 0000000000000000 hrtimer: interrupt took 15909546 ns ---[ end trace c66712db45fd58fc ]--- Kernel panic - not syncing: Fatal exception Pid: 3166, comm: insmod Tainted: G D ---------------- T 2.6.32-131.12.1.el6.x86_64 #1 Call Trace: [<ffffffff814da648>] ? panic+0x78/0x143 [<ffffffff814de694>] ? oops_end+0xe4/0x100 [<ffffffff81040c9b>] ? no_context+0xfb/0x260 [<ffffffff81040f25>] ? __bad_area_nosemaphore+0x125/0x1e0 [<ffffffff8104104e>] ? bad_area+0x4e/0x60 [<ffffffff81041773>] ? __do_page_fault+0x3c3/0x480 [<ffffffff8104a240>] ? tg_nop+0x0/0x10 [<ffffffff8105ff50>] ? tg_shares_up+0x0/0x2a0 [<ffffffff8104f88a>] ? walk_tg_tree+0x6a/0xb0 [<ffffffff810370f8>] ? pvclock_clocksource_read+0x58/0xd0 [<ffffffffa23d6007>] ? wdt_send_data+0x7/0x60 [sbc_fitpc2_wdt] [<ffffffff814e067e>] ? do_page_fault+0x3e/0xa0 [<ffffffffa23d9000>] ? fitpc2_wdt_init+0x0/0x146 [sbc_fitpc2_wdt] [<ffffffff814dda05>] ? page_fault+0x25/0x30 [<ffffffffa23d9000>] ? fitpc2_wdt_init+0x0/0x146 [sbc_fitpc2_wdt] [<ffffffff81269aab>] ? strcmp+0xb/0x30 [<ffffffff810943c5>] ? __blocking_notifier_call_chain+0x65/0x80 [<ffffffffa23d9022>] ? fitpc2_wdt_init+0x22/0x146 [sbc_fitpc2_wdt] [<ffffffff8100204c>] ? do_one_initcall+0x3c/0x1d0 [<ffffffff810aca7f>] ? sys_init_module+0xdf/0x250 [<ffffffff8100b172>] ? system_call_fastpath+0x16/0x1b
There was a workaround committed which looked like it was perfect: commit 01cfbeb43559e5a93eda648fc81d4aa102e6cb13 Author: Frank Ch. Eigler <fche@redhat.com> Date: Fri Sep 16 17:34:14 2011 -0400 PR13193 wallpaper The reported stuck process bug (whether it's in the kernel or in stap), is apparently triggered by concurrent modprobes. Change the testsuite to avoid this practice, to wallpaper over the problem temporarily. Unfortunately I just got another issue with this testcase, this time on i386 f15: [157004.483057] stap_6ab51e385550b0fd6d843037b0d7d38c_9609: systemtap: 1.7/0.148.90, base: f8aca000, memory: 34data/20text/2ctx/10net/16alloc kb, probes: 2 [157008.989239] stap_a40026234a1787c83a1dafb3aae558ca__9841: systemtap: 1.7/0.148.90, base: f8b24000, memory: 61data/24text/15ctx/10net/163alloc kb, probes: 10 [157013.346702] stap_190d377c71fa6a45f476ce480c4912a1_10067: systemtap: 1.7/0.148.90, base: f8ad9000, memory: 51data/24text/15ctx/10net/163alloc kb, probes: 6 [157018.205175] stap_1aa19d6453909e68c4758c58d1d67725_10296: systemtap: 1.7/0.148.90, base: f8b61000, memory: 143data/15text/0ctx/10net/64alloc kb, probes: 396 [157018.227386] Warning: found a stray unused aggrprobe@f8942011 [157018.227398] ------------[ cut here ]------------ [157018.227453] kernel BUG at kernel/kprobes.c:707! [157018.227498] invalid opcode: 0000 [#1] SMP [157018.227543] Modules linked in: jffs2(+) stap_1aa19d6453909e68c4758c58d1d67725_10296 uprobes nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack stap_e9739cce4a6f43356403eefe748313cc_565 stap_06d4707db0c0eb2d86b639a84f948296_9487 stap_e0d0211e1b7bbf8f7df399c1665c8493_565 mtd zlib_deflate tcp_lp tun fuse ebtable_nat ebtables xt_CHECKSUM bridge stp llc ppdev parport_pc lp parport sunrpc cpufreq_ondemand acpi_cpufreq mperf bnep bluetooth xt_physdev ip6t_REJECT snd_hda_codec_analog snd_hda_intel snd_hda_codec snd_hwdep snd_seq snd_seq_device iTCO_wdt snd_pcm arc4 iwl3945 snd_timer iwl_legacy mac80211 iTCO_vendor_support thinkpad_acpi snd soundcore cfg80211 e1000e i2c_i801 joydev microcode snd_page_alloc rfkill virtio_net kvm_intel kvm uinput ipv6 xts gf128mul dm_crypt yenta_socket i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: stap_190d377c71fa6a45f476ce480c4912a1_10067] [157018.228133] [157018.228133] Pid: 10302, comm: modprobe Not tainted 2.6.40.4-5.fc15.i686 #1 LENOVO 195144G/195144G [157018.228133] EIP: 0060:[<c07ee524>] EFLAGS: 00010246 CPU: 0 [157018.228133] EIP is at register_kprobe+0x1ca/0x3a5 [157018.228133] EAX: 00000047 EBX: f8b827fc ECX: 00000046 EDX: 00000000 [157018.228133] ESI: e9856420 EDI: 00000000 EBP: d1693eb0 ESP: d1693e90 [157018.228133] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068 [157018.228133] Process modprobe (pid: 10302, ti=d1692000 task=dba09940 task.ti=d1692000) [157018.228133] Stack: [157018.228133] c0939f72 f8942011 f8b6f1fe f8b82804 f8bfa5c4 f8b827fc f8b6f1f0 00002100 [157018.228133] d1693ed4 f8b6370e f6173a40 00000246 d1693ee4 000000c0 f8b6d0e4 00000000 [157018.228133] 00000001 d1693ef0 c07ed017 f8bfa5c4 c0a3b7ec c0a398c0 ffffffff 00000460 [157018.228133] Call Trace: [157018.228133] [<f8942011>] ? jffs2_compressors_init+0x11/0x11 [jffs2] [157018.228133] [<f8b6370e>] _stp_module_notifier+0x1a0/0x26e [stap_1aa19d6453909e68c4758c58d1d67725_10296] [157018.228133] [<c07ed017>] notifier_call_chain+0x2b/0x4d [157018.228133] [<c04582e6>] __blocking_notifier_call_chain+0x40/0x52 [157018.228133] [<c0458317>] blocking_notifier_call_chain+0x1f/0x21 [157018.228133] [<c0469ea0>] sys_init_module+0x1384/0x15af [157018.228133] [<c07e9f54>] syscall_call+0x7/0xb [157018.228133] Code: 39 46 40 75 13 8b 46 14 c7 04 24 72 9f 93 c0 89 44 24 04 e8 08 41 ff ff 83 66 3c fd 81 7e 20 9c d1 7e c0 75 06 83 7e 50 00 75 02 <0f> 0b 89 f0 e8 1c f5 ff ff f6 46 3c 01 74 22 89 f0 e8 e6 da ff [157018.228133] EIP: [<c07ee524>] register_kprobe+0x1ca/0x3a5 SS:ESP 0068:d1693e90 [157018.293049] ---[ end trace f0f649cc63eb195a ]---
cc:ing Masami because of the optimized-kprobes kernel warnings.
With echo 0 > /proc/sys/debug/kprobes-optimization, the test, executed in a tight loop with even the "sleep 3;" part removed from bz6503.exp, runs fine for hours. # while true; do make installcheck RUNTESTFLAGS=bz6503.exp; done With echo 1, the test loop fails within a minute or two, on this VM: Linux vm-f15-64 2.6.40.6-0.fc15.x86_64 #1 SMP Tue Oct 4 00:39:50 UTC 2011 x86_64 x86_64 x86_64 GNU/Linux 2 VCPU, 2GB RAM. I strongly suspect this is a kernel bug that we'll just have to work around somehow, perhaps by teaching staprun to turn off kprobes optimization.
(In reply to comment #4) > I strongly suspect this is a kernel bug that we'll just have to work > around somehow, perhaps by teaching staprun to turn off kprobes optimization. Hmm, it seems that an aggr probe(which wasn't optimized) was gone and child probes were also gone, but the aggr probe was not released and on the kprobe's hash table. BTW, I'd like to get more information: - does this happen on x86-64 too? - does this happen on upstream kernel too? - what the actual version of systemtap did you use? - is there any C code of stap code? I'm not sure why systemtap tries to put a probe on the "TAIL" of a function in .init.text... [157018.227386] Warning: found a stray unused aggrprobe@f8942011 [157018.228133] [<f8942011>] ? jffs2_compressors_init+0x11/0x11 [jffs2]
commit cd125e94 included kprobes-optimization disabling. Once lkml upstream fixes the associated problem, and we find a way of identifying the fixed releases, it can be removed, or made further conditional. (Note that one can test stap 1.7 against possible kernel fixes with env STAP_PR13193_OVERRIDE=1.)