RE: Kernel panics during execution of timeouts.stp example when booted with kgdboc=kms,kbd

[<dmitry dot krivenok at emc dot com>]

Hi,
I've configured serial console on my Fedora-16 VM and below is what I got:

...
...
[   15.083666] abrtd[716]: Init complete, entering main loop
Entering kdb (current=0xffff880131281730, pid 1185) on processor 3 Oops: (null)
due to oops @ 0xffffffff81076c2b
CPU 3 <d>Modules linked in: stap_5c9aa472e68e1b50280a6ab04af46f6e__1185 nfs fscache auth_rpcgss nfs_acl lockd 8021q fcoe libfcoe libfc scsi_transport_fc scsi_tgt garp stp llc ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables ppdev vmw_balloon microcode e1000 parport_pc parport i2c_piix4 i2c_core uinput shpchp sunrpc mptspi mptscsih mptbase scsi_transport_spi [last unloaded: scsi_wait_scan]
<c>
<d>Pid: 1185, comm: stapio Not tainted 3.1.4-1.fc16.x86_64 #1<c> VMware, Inc. VMware Virtual Platform<c>/440BX Desktop Reference Platform<c>
<d>RIP: 0010:[<ffffffff81076c2b>]  [<ffffffff81076c2b>] sys_nanosleep+0x1/0x6a
<d>RSP: 0018:ffff88013068ff80  EFLAGS: 00000297
<d>RAX: 0000000000000023 RBX: ffffffffffffffff RCX: 00007fbdf15389d0
<d>RDX: 0000000000000000 RSI: 0000000000000000 RDI: 00007fffb165b740
<d>RBP: 00007fbdf273a010 R08: 00007fbdf1538700 R09: 00007fbdf1e72f50
<d>R10: 00007fbdf15389d0 R11: 0000000000000293 R12: 0000000000008002
<d>R13: 00007fbdf27403b0 R14: 00007fbdf273a008 R15: 000000000000000b
<d>FS:  00007fbdf2514700(0000) GS:ffff88013fd80000(0000) knlGS:0000000000000000
<d>CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
<d>CR2: 00007fbdf1df5340 CR3: 0000000133b3a000 CR4: 00000000000006e0
<d>DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
<d>DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process stapio (pid: 1185, threadinfo ffff88013068e000, task ffff880131281730)
<0>Stack:
<c> ffffffff814bd9c2<c> 0000000000000293<c> 00007fbdf15389d0<c> 00007fbdf1e72f50<c>
<c> 00007fbdf1538700<c> 0000000000000023<c> 00007fbdf20fd7e5<c> 0000000000000000<c>
<c> 0000000000000000<c> 00007fffb165b740<c> 0000000000000023<c> 00007fbdf1df536d<c>
<0>Call Trace:
[3]more> 
<0> [<ffffffff814bd9c2>] ? system_call_fastpath+0x16/0x1b
[3]more> 
<0>Code: ff ba fc fd ff ff 48 63 c2 48 8b 55 d8 65 48 33 14 25 28 00 00 00 74 05 e8 14 0b fe ff 48 83 c4 70 5b 41 5c 41 5d 41 5e 5d c3 cc <48> 89 e5 41 54 53 48 83 ec 10 66 66 66 66 90 49 89 fc 48 89 f3 
[3]more> 
Call Trace:
[3]more> 
 [<ffffffff814bd9c2>] ? system_call_fastpath+0x16/0x1b
[3]more> 
[3]more> 
[3]kdb>


Also, I've run "sudo stap-report > ~/tmp/stap-report.out 2>&1" and attached the output.
I'm going to investigate this problem more deeply later today and then post the results.

Thanks,
Dmitry

-----Original Message-----
From: systemtap-owner@sourceware.org [mailto:systemtap-owner@sourceware.org] On Behalf Of Mark Wielaard
Sent: Wednesday, December 07, 2011 3:20 PM
To: Krivenok, Dmitry
Cc: systemtap@sourceware.org
Subject: Re: Kernel panics during execution of timeouts.stp example when booted with kgdboc=kms,kbd

Hi,

On Tue, 2011-12-06 at 03:39 -0500, dmitry.krivenok@emc.com wrote:
> The problem is that kernel panics if you run it with "kgdboc=kms,kbd"
> parameter and then run standard timeout.stp example as follows:
> 
> $ sudo stap -v `locate timeout.stp`
> Pass 1: parsed user script and 77 library script(s) using 78192virt/22076res/2376shr kb, in 190usr/60sys/689real ms.
> Pass 2: analyzed script: 12 probe(s), 7 function(s), 5 embed(s), 10 global(s) using 202032virt/44892res/3256shr kb, in 790usr/4810sys/10056real ms.
> Pass 3: using cached /root/.systemtap/cache/dd/stap_ddd8c5ac8daf0d8644742908043722b3_12685.c
> Pass 4: using cached /root/.systemtap/cache/dd/stap_ddd8c5ac8daf0d8644742908043722b3_12685.ko
> Pass 5: starting run.
> 
> I also tried to run stap with -vvv options and the last lines I saw were:
> ...
> ...
> execing: /usr/local/libexec/systemtap/stapio -v -v -R stap_ddd8c5ac8daf0d8644742908043722b3_1_777
> stapio:parse_modpath:368 modpath="/lib/modules/3.1.1/systemtap/stap_ddd8c5ac8daf0d8644742908043722b3_1_777.ko"
> stapio:main:41 modpath="/lib/modules/3.1.1/systemtap/stap_ddd8c5ac8daf0d8644742908043722b3_1_777.ko", modname="stap_ddd8c5ac8daf0d8644742908043722b3_1_777"
> stapio:init_stapio:338 init_stapio
> stapio:init_ctl_channel:31 Opened /sys/kernel/debug/systemtap/stap_ddd8c5ac8daf0d8644742908043722b3_1_777/.cmd (3)
> stapio:stp_main_loop:529 in main loop
> stapio:init_relayfs:239 initializing relayfs
> stapio:init_relayfs:263 attempting to open /sys/kernel/debug/systemtap/stap_ddd8c5ac8daf0d8644742908043722b3_1_777/trace0
> stapio:init_relayfs:263 attempting to open /sys/kernel/debug/systemtap/stap_ddd8c5ac8daf0d8644742908043722b3_1_777/trace1
> stapio:init_relayfs:269 ncpus=1, bulkmode = 0
> stapio:init_relayfs:352 starting threads
> 
> The problem is 100% reproducible on both systems.
> I'm not sure if it's a bug in Systemtap or kernel, because I could misconfigure something.
> 
> Could someone please try to reproduce the problem and post results here?

I don't have the above setup here handy atm. But it would be helpful if
you could post the results of 'stap-report' somewhere. And even better
if you could try and get a serial console dump of the kernel panic you
are getting, see http://sourceware.org/systemtap/wiki/HowToReportBugs
and here http://sourceware.org/systemtap/wiki/DeveloperSetupTips for
some guides on setting that up.

Thanks,

Mark

Attachment: stap-report.out
Description: stap-report.out