Bug 22265 - on rawhide, we're getting a kernel panic
Summary: on rawhide, we're getting a kernel panic
Status: RESOLVED WORKSFORME
Alias: None
Product: systemtap
Classification: Unclassified
Component: runtime (show other bugs)
Version: unspecified
: P2 normal
Target Milestone: ---
Assignee: Unassigned
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2017-10-05 16:10 UTC by David Smith
Modified: 2020-02-19 21:39 UTC (History)
1 user (show)

See Also:
Host:
Target:
Build:
Last reconfirmed:

Attachments
Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this bug.
Description David Smith 2017-10-05 16:10:05 UTC 
On rawhide (4.14.0-0.rc3.git0.1.fc28.x86_64), we're getting an odd kernel panic on full testsuite runs. I can't seem to narrow it down any more than that no matter what I try.

Here are excerpts from the console output. Things start out OK, then we get lots of segfaults in stapdyn (which is understandable with the current dyninst problems on rawhide):

====
[ 1582.835099] stap_d6038e88b0e51e67a5adebe6504b82a3_5823: loading out-of-tree module taints kernel. 
[ 1582.836265] stap_d6038e88b0e51e67a5adebe6504b82a3_5823: module verification failed: signature and/or required key missing - tainting kernel 
[ 1772.111044] show_signal_msg: 62 callbacks suppressed 
[ 1772.111983] stapdyn[25606]: segfault at 58 ip 00007f199d0605bb sp 00007ffebd186500 error 4 in libsymtabAPI.so.9.3.2[7f199d01e000+123000] 
[ 1777.990599] stapdyn[26096]: segfault at 58 ip 00007ff0158455bb sp 00007ffd392579d0 error 4 in libsymtabAPI.so.9.3.2[7ff015803000+123000] 
[ 1783.283213] stapdyn[26586]: segfault at 58 ip 00007fab0dc8c5bb sp 00007ffd0e2acc50 error 4 in libsymtabAPI.so.9.3.2[7fab0dc4a000+123000] 
[ 1795.359730] stapdyn[27087]: segfault at 58 ip 00007f37e575b5bb sp 00007ffcad1fe6f0 error 4 in libsymtabAPI.so.9.3.2[7f37e5719000+123000] 
[ 1807.681140] stapdyn[27591]: segfault at 58 ip 00007f1be69715bb sp 00007ffce239e9d0 error 4 in libsymtabAPI.so.9.3.2[7f1be692f000+123000] 
[ 1964.299192] stapdyn[7435]: segfault at 58 ip 00007fb8b045b5bb sp 00007fffe80f9870 error 4 in libsymtabAPI.so.9.3.2[7fb8b0419000+123000] 
====

... I've skipped a bunch of stapdyn errors here...

Here is where things start to get wonky. The segfaults start to occur in the probed processes, not in stapdyn. Then this seems to spread to systemd (process 1). 

====
 8920.239055] stapdyn[25959]: segfault at 58 ip 00007f060df905bb sp 00007ffc96f17030 error 4 in libsymtabAPI.so.9.3.2[7f060df4e000+123000] 
[ 8920.489612] utrace_p5_multi[25964]: segfault at 0 ip           (null) sp 00007ffd169ac2d8 error 14 in utrace_p5_multi_4512[400000+1000] 
[ 8931.470029] stapdyn[26482]: segfault at 58 ip 00007f3bb9aba5bb sp 00007ffd501f3920 error 4 in libsymtabAPI.so.9.3.2[7f3bb9a78000+123000] 
[ 8931.726801] utrace_p5_multi[26487]: segfault at 0 ip           (null) sp 00007fff4fdb61f8 error 14 in utrace_p5_multi_4512[400000+1000] 
[ 8990.722785] stapdyn[30192]: segfault at 58 ip 00007fb3b68aa5bb sp 00007fffb28cbdc0 error 4 in libsymtabAPI.so.9.3.2[7fb3b6868000+123000] 
[ 9073.875930] stapdyn[3611]: segfault at 58 ip 00007f78a9e2d5bb sp 00007ffe57419980 error 4 in libsymtabAPI.so.9.3.2[7f78a9deb000+123000] 
[ 9242.226823] nr_pdflush_threads exported in /proc is scheduled for removal 
[12184.169178] stapdyn[10279]: segfault at 58 ip 00007f7bf42885bb sp 00007ffcc170f990 error 4 in libsymtabAPI.so.9.3.2[7f7bf4246000+123000] 
[12184.428844] pthreadprobes[10284]: segfault at 0 ip           (null) sp 00007fff490ad788 error 14 in pthreadprobes[400000+2000] 
[12505.244354] stapdyn[31221]: segfault at 58 ip 00007f0c949ed5bb sp 00007ffd8ee9d950 error 4 in libsymtabAPI.so.9.3.2[7f0c949ab000+123000] 
[19041.457388] traps: beah-rhts-task[862] trap stack segment ip:7f5cd038d080 sp:7fffe87a0688 error:0 in libc-2.26.90.so[7f5cd030b000+1ae000] 
[19041.459477] systemd-journal[452]: segfault at 3d ip 00007f3999a4f260 sp 00007fff1f6ff6b8 error 6 in libc-2.26.90.so[7f399994d000+1ae000] 
[19041.460709] auditd[532]: segfault at 5f ip 00007fa9ab88b260 sp 00007fff44fd6728 error 6 in libc-2.26.90.so[7fa9ab789000+1ae000] 
[19041.462895] systemd[1]: segfault at 2 ip 00007f1bd6af9260 sp 00007ffdf11eff98 error 6 in libc-2.26.90.so[7f1bd69f7000+1ae000] 
[19041.466851] systemd[1]: segfault at 1 ip 00007f1bd6af9200 sp 00007ffdf11ef788 error 6 in libc-2.26.90.so[7f1bd69f7000+1ae000] 
[19041.470302] systemd[1]: segfault at 1 ip 00007f1bd6af9200 sp 00007ffdf11eef88 error 6 in libc-2.26.90.so[7f1bd69f7000+1ae000] 
[19041.471631] systemd[1]: segfault at 1 ip 00007f1bd6af9200 sp 00007ffdf11ee788 error 6 in libc-2.26.90.so[7f1bd69f7000+1ae000] 
[19041.476197] audit: type=1701 audit(1507163333.199:695): auid=4294967295 uid=0 gid=0 ses=4294967295 subj=system_u:system_r:init_t:s0 pid=17718 comm="systemd" exe="/usr/lib/systemd/systemd" sig=11 res=1 
[19041.478055] audit: type=1701 audit(1507163333.202:696): auid=4294967295 uid=0 gid=0 ses=4294967295 subj=system_u:system_r:init_t:s0 pid=17720 comm="systemd" exe="/usr/lib/systemd/systemd" sig=11 res=1 
[19041.480178] systemd[1]: segfault at 1 ip 00007f1bd6af9200 sp 00007ffdf11edf88 error 6 in libc-2.26.90.so[7f1bd69f7000+1ae000] 
[19047.533621] show_signal_msg: 81 callbacks suppressed 
[19047.534317] systemd[1]: segfault at 1 ip 00007f1bd6af9200 sp 00007ffdf11d8f88 error 6 in libc-2.26.90.so[7f1bd69f7000+1ae000] 
[19047.537838] systemd[17766]: segfault at 7ffdf09f0ff8 ip 00007f1bd6dc0f02 sp 00007ffdf09f0ef0 error 6 in libpthread-2.26.90.so[7f1bd6daf000+19000] 
[19047.634192] systemd[17765]: segfault at 7ffdf09f0fc8 ip 0000560790523256 sp 00007ffdf09f0fd0 error 6 in systemd[560790437000+167000] 
[19047.670316] systemd[17769]: segfault at 7ffdf09f0f98 ip 00007f1bd6dc1191 sp 00007ffdf09f0f80 error 6 in libpthread-2.26.90.so[7f1bd6daf000+19000] 
[19047.678171] systemd[17767]: segfault at 7ffdf09f0fb8 ip 00007f1bd6dc0f19 sp 00007ffdf09f0f30 error 6 in libpthread-2.26.90.so[7f1bd6daf000+19000] 
[19047.809605] sssd_nss[723]: segfault at 7f6be8f41e8d ip 00007f6be8774d27 sp 00007ffc25db9b80 error 7 in libc-2.26.90.so[7f6be86f7000+1ae000] 
[19047.811408] audit: type=1701 audit(1507163339.535:727): auid=4294967295 uid=0 gid=0 ses=4294967295 subj=system_u:system_r:sssd_t:s0 pid=723 comm="sssd_nss" exe="/usr/libexec/sssd/sssd_nss" sig=11 res=1 
[19047.838607] sssd_be[584]: segfault at 7ff9e67f0700 ip 00007ff9e373ed90 sp 00007ffd5ee6eee8 error 7 in libpthread-2.26.90.so[7ff9e3735000+19000] 
[19047.841767] audit: type=1701 audit(1507163339.565:728): auid=4294967295 uid=0 gid=0 ses=4294967295 subj=system_u:system_r:sssd_t:s0 pid=584 comm="sssd_be" exe="/usr/libexec/sssd/sssd_be" sig=11 res=1 
[19047.856917] sssd[551]: segfault at 7f3f91862741 ip 00007f3f8e7b0d92 sp 00007ffce81a2d78 error 7 in libpthread-2.26.90.so[7f3f8e7a7000+19000] 
[19052.726436] kauditd_printk_skb: 59 callbacks suppressed 
[19052.726985] audit: type=1701 audit(1507163344.450:754): auid=4294967295 uid=0 gid=0 ses=4294967295 subj=system_u:system_r:init_t:s0 pid=17798 comm="systemd" exe="/usr/lib/systemd/systemd" sig=11 res=1 
[19052.742422] systemd[1]: segfault at 1 ip 00007f1bd6af9200 sp 00007ffdf11c8f88 error 6 in libc-2.26.90.so[7f1bd69f7000+1ae000] 
[19052.753497] audit: type=1701 audit(1507163344.477:755): auid=4294967295 uid=0 gid=0 ses=4294967295 subj=system_u:system_r:init_t:s0 pid=17799 comm="systemd" exe="/usr/lib/systemd/systemd" sig=11 res=1 
[19052.896612] audit: type=1701 audit(1507163344.620:756): auid=4294967295 uid=0 gid=0 ses=4294967295 subj=system_u:system_r:init_t:s0 pid=17813 comm="systemd" exe="/usr/lib/systemd/systemd" sig=11 res=1 
[19053.749655] audit: type=1701 audit(1507163345.473:757): auid=4294967295 uid=0 gid=0 ses=4294967295 subj=system_u:system_r:init_t:s0 pid=17814 comm="systemd" exe="/usr/lib/systemd/systemd" sig=11 res=1 
[19053.764691] systemd[17816]: segfault at 7ffdf09f0ff8 ip 000056079052324c sp 00007ffdf09f0ed0 error 6 in systemd[560790437000+167000] 
[19053.787390] systemd[1]: segfault at 1 ip 00007f1bd6af9200 sp 00007ffdf11c8788 error 6 in libc-2.26.90.so[7f1bd69f7000+1ae000] 
[19053.840424] audit: type=1701 audit(1507163345.564:758): auid=4294967295 uid=0 gid=0 ses=4294967295 subj=system_u:system_r:init_t:s0 pid=17811 comm="systemd" exe="/usr/lib/systemd/systemd" sig=11 res=1 
[19053.844535] systemd[17817]: segfault at 7ffdf09f0f08 ip 0000560790523256 sp 00007ffdf09f0f10 error 6 in systemd[560790437000+167000] 
[19053.862475] systemd[17820]: segfault at 7ffdf09f0fc8 ip 0000560790523256 sp 00007ffdf09f0fd0 error 6 in systemd[560790437000+167000] 
[19057.793553] show_signal_msg: 86 callbacks suppressed 
[19057.794076] systemd[17885]: segfault at 7ffdf09f0f48 ip 0000560790523256 sp 00007ffdf09f0f50 error 6 in systemd[560790437000+167000] 
[19057.812420] audit: type=1701 audit(1507163349.536:797): auid=4294967295 uid=0 gid=0 ses=4294967295 subj=system_u:system_r:init_t:s0 pid=17885 comm="systemd" exe="/usr/lib/systemd/systemd" sig=11 res=1 
[19058.455468] systemd[1]: segfault at 1 ip 00007f1bd6af9200 sp 00007ffdf11b5788 error 6 in libc-2.26.90.so[7f1bd69f7000+1ae000] 
[19058.463660] systemd[17891]: segfault at 7ffdf09f0ff8 ip 00007f1bd6dc0f02 sp 00007ffdf09f0ef0 error 6 in libpthread-2.26.90.so[7f1bd6daf000+19000] 
[19058.482145] systemd[17892]: segfault at 7ffdf09f0fb8 ip 00007f1bd6dc0f19 sp 00007ffdf09f0f30 error 6 in libpthread-2.26.90.so[7f1bd6daf000+19000] 
[19058.509725] systemd[1]: segfault at 1 ip 00007f1bd6af9200 sp 00007ffdf11b4f88 error 6 in libc-2.26.90.so[7f1bd69f7000+1ae000] 
[19058.525793] audit: type=1701 audit(1507163350.249:798): auid=4294967295 uid=0 gid=0 ses=4294967295 subj=system_u:system_r:init_t:s0 pid=17891 comm="systemd" exe="/usr/lib/systemd/systemd" sig=11 res=1 
[19058.552338] audit: type=1701 audit(1507163350.276:799): auid=4294967295 uid=0 gid=0 ses=4294967295 subj=system_u:system_r:init_t:s0 pid=17892 comm="systemd" exe="/usr/lib/systemd/systemd" sig=11 res=1 
[19058.637978] audit: type=1701 audit(1507163350.361:800): auid=4294967295 uid=0 gid=0 ses=4294967295 subj=system_u:system_r:init_t:s0 pid=17898 comm="systemd" exe="/usr/lib/systemd/systemd" sig=11 res=1 
[19058.650453] systemd[1]: segfault at 1 ip 00007f1bd6af9200 sp 00007ffdf11b4788 error 6 in libc-2.26.90.so[7f1bd69f7000+1ae000] 
[19063.107909] show_signal_msg: 86 callbacks suppressed 
[19063.108412] systemd[17966]: segfault at 7ffdf09f0fd8 ip 00007f1bd6dc1191 sp 00007ffdf09f0fc0 error 6 in libpthread-2.26.90.so[7f1bd6daf000+19000] 
[19063.236707] systemd[1]: segfault at 1 ip 00007f1bd6af9200 sp 00007ffdf11a1788 error 6 in libc-2.26.90.so[7f1bd69f7000+1ae000] 
[19063.247143] audit: type=1701 audit(1507163354.970:839): auid=4294967295 uid=0 gid=0 ses=4294967295 subj=system_u:system_r:init_t:s0 pid=17975 comm="systemd" exe="/usr/lib/systemd/systemd" sig=11 res=1 
[19063.313254] audit: type=1701 audit(1507163354.955:838): auid=4294967295 uid=0 gid=0 ses=4294967295 subj=system_u:system_r:init_t:s0 pid=17974 comm="systemd" exe="/usr/lib/systemd/systemd" sig=11 res=1 
[19063.332955] audit: type=1701 audit(1507163355.056:840): auid=4294967295 uid=0 gid=0 ses=4294967295 subj=system_u:system_r:init_t:s0 pid=17971 comm="systemd" exe="/usr/lib/systemd/systemd" sig=11 res=1 
[19063.347875] audit: type=1701 audit(1507163355.071:841): auid=4294967295 uid=0 gid=0 ses=4294967295 subj=system_u:system_r:init_t:s0 pid=17977 comm="systemd" exe="/usr/lib/systemd/systemd" sig=11 res=1 
[19063.356221] audit: type=1701 audit(1507163355.080:842): auid=4294967295 uid=0 gid=0 ses=4294967295 subj=system_u:system_r:init_t:s0 pid=17966 comm="systemd" exe="/usr/lib/systemd/systemd" sig=11 res=1 
[19063.359546] systemd[1]: segfault at 1 ip 00007f1bd6af9200 sp 00007ffdf11a0f88 error 6 in libc-2.26.90.so[7f1bd69f7000+1ae000] 
[19063.385186] audit: type=1701 audit(1507163355.108:843): auid=4294967295 uid=0 gid=0 ses=4294967295 subj=system_u:system_r:init_t:s0 pid=17982 comm="systemd" exe="/usr/lib/systemd/systemd" sig=11 res=1 
[19063.387576] systemd[1]: segfault at 1 ip 00007f1bd6af9200 sp 00007ffdf11a0788 error 6 in libc-2.26.90.so[7f1bd69f7000+1ae000] 
[19068.118130] show_signal_msg: 523 callbacks suppressed 
[19068.118828] systemd[1]: segfault at 1 ip 00007f1bd6af9200 sp 00007ffdf112c788 error 6 
[19068.119491] systemd[18419]: segfault at 7ffdf09f0fd8 ip 00007f1bd6dc1191 sp 00007ffdf09f0fc0 error 6 
[19068.129564] audit: type=1701 audit(1507163359.853:1067): auid=4294967295 uid=0 gid=0 ses=4294967295 subj=system_u:system_r:init_t:s0 pid=18423 comm="systemd" exe="/usr/lib/systemd/systemd" sig=11 res=1 
[19068.147892] audit: type=1701 audit(1507163359.871:1068): auid=4294967295 uid=0 gid=0 ses=4294967295 subj=system_u:system_r:init_t:s0 pid=18422 comm="systemd" exe="/usr/lib/systemd/systemd" sig=11 res=1 
[19068.161665] audit: type=1701 audit(1507163359.885:1069): auid=4294967295 uid=0 gid=0 ses=4294967295 subj=system_u:system_r:init_t:s0 pid=18429 comm="systemd" exe="/usr/lib/systemd/systemd" sig=11 res=1 
[19068.163965] audit: type=1701 audit(1507163359.888:1070): auid=4294967295 uid=0 gid=0 ses=4294967295 subj=system_u:system_r:init_t:s0 pid=18420 comm="systemd" exe="/usr/lib/systemd/systemd" sig=11 res=1 
[19068.187530]  in libc-2.26.90.so[7f1bd69f7000+1ae000] 
[19068.222427] audit: type=1701 audit(1507163359.946:1071): auid=4294967295 uid=0 gid=0 ses=4294967295 subj=system_u:system_r:init_t:s0 pid=18428 comm="systemd" exe="/usr/lib/systemd/systemd" sig=11 res=1 
[19068.230738] audit: type=1701 audit(1507163359.954:1072): auid=4294967295 uid=0 gid=0 ses=4294967295 subj=system_u:system_r:init_t:s0 pid=18427 comm="systemd" exe="/usr/lib/systemd/systemd" sig=11 res=1 
[19068.245281]  in libpthread-2.26.90.so[7f1bd6daf000+19000] 
[19068.263182] audit: type=1701 audit(1507163359.986:1073): auid=4294967295 uid=0 gid=0 ses=4294967295 subj=system_u:system_r:init_t:s0 pid=18440 comm="systemd" exe="/usr/lib/systemd/systemd" sig=11 res=1 
[19068.274157] systemd[1]: segfault at 1 ip 00007f1bd6af9200 sp 00007ffdf112bf88 error 6 in libc-2.26.90.so[7f1bd69f7000+1ae000] 
[19073.129453] kauditd_printk_skb: 643 callbacks suppressed 
[19073.130097] audit: type=1701 audit(1507163364.797:1351): auid=4294967295 uid=0 gid=0 ses=4294967295 subj=system_u:system_r:init_t:s0 pid=18989 comm="systemd" exe="/usr/lib/systemd/systemd" sig=11 res=1 
[19073.140874] systemd[19002]: segfault at 7ffdf09f0f88 ip 0000560790523256 sp 00007ffdf09f0f90 error 6 in systemd[560790437000+167000] 
[19073.146986] audit: type=1701 audit(1507163364.870:1353): auid=4294967295 uid=0 gid=0 ses=4294967295 subj=system_u:system_r:init_t:s0 pid=18998 comm="systemd" exe="/usr/lib/systemd/systemd" sig=11 res=1 
[19073.151265] systemd[18999]: segfault at 7ffdf09f0ff8 ip 000056079052324c sp 00007ffdf09f0ed0 error 6 in systemd[560790437000+167000] 
[19073.160638] systemd[19000]: segfault at 7ffdf09f0f08 ip 0000560790523256 sp 00007ffdf09f0f10 error 6 in systemd[560790437000+167000] 
[19073.167475] systemd[19001]: segfault at 7ffdf09f0f48 ip 0000560790523256 sp 00007ffdf09f0f50 error 6 in systemd[560790437000+167000] 
[19073.172559] systemd[19003]: segfault at 7ffdf09f0fc8 ip 0000560790523256 sp 00007ffdf09f0fd0 error 6 in systemd[560790437000+167000] 
[19073.180104] systemd[19004]: segfault at 7ffdf09f0ff8 ip 00007f1bd6dc0f02 sp 00007ffdf09f0ef0 error 6 in libpthread-2.26.90.so[7f1bd6daf000+19000] 
[19073.196284] systemd[1]: segfault at 1 ip 00007f1bd6af9200 sp 00007ffdf109cf88 error 6 in libc-2.26.90.so[7f1bd69f7000+1ae000] 
[19073.214105] systemd[19011]: segfault at 7ffdf09f0f98 ip 00007f1bd6dc1191 sp 00007ffdf09f0f80 error 6 in libpthread-2.26.90.so[7f1bd6daf000+19000] 
[19078.135481] show_signal_msg: 675 callbacks suppressed 
[19078.136132] systemd[19595]: segfault at 7ffdf09f0f48 ip 0000560790523256 sp 00007ffdf09f0f50 error 6 
[19078.138512] systemd[1]: segfault at 1 ip 00007f1bd6af9200 sp 00007ffdf100a788 error 6 
[19078.155260] audit: type=1701 audit(1507163369.875:1652): auid=4294967295 uid=0 gid=0 ses=4294967295 subj=system_u:system_r:init_t:s0 pid=19592 comm="systemd" exe="/usr/lib/systemd/systemd" sig=11 res=1 
[19078.187546]  in systemd[560790437000+167000] 
[19078.207918] systemd[19601]: segfault at 7ffdf09f0fc8 ip 0000560790523256 sp 00007ffdf09f0fd0 error 6 in systemd[560790437000+167000] 
[19078.215945]  in libc-2.26.90.so[7f1bd69f7000+1ae000] 
[19078.217825] systemd[19602]: segfault at 7ffdf09f0ff8 ip 00007f1bd6dc0f02 sp 00007ffdf09f0ef0 error 6 in libpthread-2.26.90.so[7f1bd6daf000+19000] 
[19078.233505] audit: type=1701 audit(1507163369.957:1653): auid=4294967295 uid=0 gid=0 ses=4294967295 subj=system_u:system_r:init_t:s0 pid=19595 comm="systemd" exe="/usr/lib/systemd/systemd" sig=11 res=1 
[19078.260118] systemd[19600]: segfault at 7ffdf09f0f88 ip 0000560790523256 sp 00007ffdf09f0f90 error 6 in systemd[560790437000+167000] 
[19078.287082] systemd[1]: segfault at 1 ip 00007f1bd6af9200 sp 00007ffdf1009f88 error 6 in libc-2.26.90.so[7f1bd69f7000+1ae000] 
[19078.300765] audit: type=1701 audit(1507163370.024:1654): auid=4294967295 uid=0 gid=0 ses=4294967295 subj=system_u:system_r:init_t:s0 pid=19601 comm="systemd" exe="/usr/lib/systemd/systemd" sig=11 res=1 
[19078.310518] systemd[19607]: segfault at 7ffdf09f0fb8 ip 00007f1bd6dc0f19 sp 00007ffdf09f0f30 error 6 in libpthread-2.26.90.so[7f1bd6daf000+19000] 
[19083.141199] kauditd_printk_skb: 667 callbacks suppressed 
[19083.141743] audit: type=1701 audit(1507163374.865:1947): auid=4294967295 uid=0 gid=0 ses=4294967295 subj=system_u:system_r:init_t:s0 pid=20188 comm="systemd" exe="/usr/lib/systemd/systemd" sig=11 res=1 
[19083.150970] systemd[1]: segfault at 1 ip 00007f1bd6af9200 sp 00007ffdf0f77788 error 6 in libc-2.26.90.so[7f1bd69f7000+1ae000] 
[19083.156520] systemd[20190]: segfault at 7ffdf09f0ff8 ip 000056079052324c sp 00007ffdf09f0ed0 error 6 in systemd[560790437000+167000] 
[19083.174568] systemd[20192]: segfault at 7ffdf09f0f08 ip 0000560790523256 sp 00007ffdf09f0f10 error 6 in systemd[560790437000+167000] 
[19083.182500] systemd[20194]: segfault at 7ffdf09f0f48 ip 0000560790523256 sp 00007ffdf09f0f50 error 6 in systemd[560790437000+167000] 
[19083.206494] systemd[20197]: segfault at 7ffdf09f0f88 ip 0000560790523256 sp 00007ffdf09f0f90 error 6 in systemd[560790437000+167000] 
[19083.243673] systemd[20198]: segfault at 7ffdf09f0fc8 ip 0000560790523256 sp 00007ffdf09f0fd0 error 6 
[19083.246090] audit: type=1701 audit(1507163374.970:1948): auid=4294967295 uid=0 gid=0 ses=4294967295 subj=system_u:system_r:init_t:s0 pid=20190 comm="systemd" exe="/usr/lib/systemd/systemd" sig=11 res=1 
[19083.251545]  in systemd[560790437000+167000] 
[19083.258655] systemd[1]: segfault at 1 ip 00007f1bd6af9200 sp 00007ffdf0f76f88 error 6 in libc-2.26.90.so[7f1bd69f7000+1ae000] 
[19083.267801] audit: type=1701 audit(1507163374.991:1949): auid=4294967295 uid=0 gid=0 ses=4294967295 subj=system_u:system_r:init_t:s0 pid=20192 comm="systemd" exe="/usr/lib/systemd/systemd" sig=11 res=1 
[19088.161961] show_signal_msg: 604 callbacks suppressed 
[19088.162538] systemd[1]: segfault at 1 ip 00007f1bd6af9200 sp 00007ffdf0ef4f88 error 6 
[19088.175007] systemd[20723]: segfault at 7ffdf09f0f48 ip 0000560790523256 sp 00007ffdf09f0f50 error 6 
[19088.177063]  in libc-2.26.90.so[7f1bd69f7000+1ae000] 
[19088.185946] systemd[20727]: segfault at 7ffdf09f0f88 ip 0000560790523256 sp 00007ffdf09f0f90 error 6 in systemd[560790437000+167000] 
[19088.192718]  in systemd[560790437000+167000] 
[19088.194489] audit: type=1701 audit(1507163379.918:2216): auid=4294967295 uid=0 gid=0 ses=4294967295 subj=system_u:system_r:init_t:s0 pid=20711 comm="systemd" exe="/usr/lib/systemd/systemd" sig=11 res=1 
[19088.199785] systemd[20731]: segfault at 7ffdf09f0ff8 ip 00007f1bd6dc0f19 sp 00007ffdf09f0f70 error 6 
[19088.200696] systemd[20730]: segfault at 7ffdf09f0fb8 ip 00007f1bd6dc0f19 sp 00007ffdf09f0f30 error 6 
[19088.205413]  in libpthread-2.26.90.so[7f1bd6daf000+19000] 
[19088.215399] audit: type=1701 audit(1507163379.939:2217): auid=4294967295 uid=0 gid=0 ses=4294967295 subj=system_u:system_r:init_t:s0 pid=20723 comm="systemd" exe="/usr/lib/systemd/systemd" sig=11 res=1 
[19088.222267] systemd[20728]: segfault at 7ffdf09f0fc8 ip 0000560790523256 sp 00007ffdf09f0fd0 error 6 in systemd[560790437000+167000] 
[19088.243504] audit: type=1701 audit(1507163379.967:2218): auid=4294967295 uid=0 gid=0 ses=4294967295 subj=system_u:system_r:init_t:s0 pid=20727 comm="systemd" exe="/usr/lib/systemd/systemd" sig=11 res=1 
[19088.246937] systemd[1]: segfault at 1 ip 00007f1bd6af9200 sp 00007ffdf0ef4788 error 6 in libc-2.26.90.so[7f1bd69f7000+1ae000] 
[19088.254341]  in libpthread-2.26.90.so[7f1bd6daf000+19000] 
[19093.173168] show_signal_msg: 979 callbacks suppressed 
[19093.173697] systemd[1]: segfault at 1 ip 00007f1bd6af9200 sp 00007ffdf0e1f788 error 6 
[19093.187760] systemd[21578]: segfault at 7ffdf09f0ff8 ip 000056079052324c sp 00007ffdf09f0ed0 error 6 
[19093.201365]  in libc-2.26.90.so[7f1bd69f7000+1ae000] 
[19093.208753] systemd[21580]: segfault at 7ffdf09f0f08 ip 0000560790523256 sp 00007ffdf09f0f10 error 6 in systemd[560790437000+167000] 
[19093.220943] systemd[21581]: segfault at 7ffdf09f0f48 ip 0000560790523256 sp 00007ffdf09f0f50 error 6 in systemd[560790437000+167000] 
[19093.232909] systemd[21583]: segfault at 7ffdf09f0f88 ip 0000560790523256 sp 00007ffdf09f0f90 error 6 in systemd[560790437000+167000] 
[19093.240385] systemd[21585]: segfault at 7ffdf09f0fc8 ip 0000560790523256 sp 00007ffdf09f0fd0 error 6 in systemd[560790437000+167000] 
[19093.245174] systemd[21587]: segfault at 7ffdf09f0ff8 ip 00007f1bd6dc0f02 sp 00007ffdf09f0ef0 error 6 in libpthread-2.26.90.so[7f1bd6daf000+19000] 
[19093.248666]  in systemd[560790437000+167000] 
[19093.261939] systemd[1]: segfault at 1 ip 00007f1bd6af9200 sp 00007ffdf0e1ef88 error 6 in libc-2.26.90.so[7f1bd69f7000+1ae000] 
[19093.266428] audit: type=1701 audit(1507163384.990:2647): auid=4294967295 uid=0 gid=0 ses=4294967295 subj=system_u:system_r:init_t:s0 pid=21578 comm="systemd" exe="/usr/lib/systemd/systemd" sig=11 res=1 
[19093.271216] audit: type=1701 audit(1507163384.995:2648): auid=4294967295 uid=0 gid=0 ses=4294967295 subj=system_u:system_r:init_t:s0 pid=21580 comm="systemd" exe="/usr/lib/systemd/systemd" sig=11 res=1 
[19098.174040] show_signal_msg: 3649 callbacks suppressed 
[19098.175086] systemd[1]: segfault at 1 ip 00007f1bd6af9200 sp 00007ffdf0b09f88 error 6 
[19098.175900] systemd[24769]: segfault at 7ffdf09f0fb8 ip 00007f1bd6dc0f19 sp 00007ffdf09f0f30 error 6 
[19098.177114] audit: type=1701 audit(1507163389.901:4239): auid=4294967295 uid=0 gid=0 ses=4294967295 subj=system_u:system_r:init_t:s0 pid=24773 comm="systemd" exe="/usr/lib/systemd/systemd" sig=11 res=1 
[19098.179876] systemd[24772]: segfault at 7ffdf09f0fd8 ip 00007f1bd6dc1191 sp 00007ffdf09f0fc0 error 6 in libpthread-2.26.90.so[7f1bd6daf000+19000] 
[19098.181953]  in libc-2.26.90.so[7f1bd69f7000+1ae000] 
[19098.182679] systemd[1]: segfault at 1 ip 00007f1bd6af9200 sp 00007ffdf0b09788 error 6 in libc-2.26.90.so[7f1bd69f7000+1ae000] 
[19098.185530]  in libpthread-2.26.90.so[7f1bd6daf000+19000] 
[19098.186229] audit: type=1701 audit(1507163389.910:4240): auid=4294967295 uid=0 gid=0 ses=4294967295 subj=system_u:system_r:init_t:s0 pid=24781 comm="systemd" exe="/usr/lib/systemd/systemd" sig=11 res=1 
[19098.187862] audit: type=1701 audit(1507163389.912:4241): auid=4294967295 uid=0 gid=0 ses=4294967295 subj=system_u:system_r:init_t:s0 pid=24772 comm="systemd" exe="/usr/lib/systemd/systemd" sig=11 res=1 
[19098.189992] audit: type=1701 audit(1507163389.914:4242): auid=4294967295 uid=0 gid=0 ses=4294967295 subj=system_u:system_r:init_t:s0 pid=24769 comm="systemd" exe="/usr/lib/systemd/systemd" sig=11 res=1 
[19098.192311] systemd[1]: segfault at 1 ip 00007f1bd6af9200 sp 00007ffdf0b08f88 error 6 in libc-2.26.90.so[7f1bd69f7000+1ae000] 
[19098.194048] audit: type=1701 audit(1507163389.916:4243): auid=4294967295 uid=0 gid=0 ses=4294967295 subj=system_u:system_r:init_t:s0 pid=24785 comm="systemd" exe="/usr/lib/systemd/systemd" sig=11 res=1 
====

Here's the actual kernel panic. I don't see any systemtap functions in the stack trace.

====
[19098.826574] Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000008b 
[19098.826574]  
[19098.827510] CPU: 0 PID: 1 Comm: systemd Tainted: G           OE   4.14.0-0.rc3.git0.1.fc28.x86_64 #1 
[19098.828386] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 
[19098.828930] Call Trace: 
[19098.829161]  dump_stack+0x63/0x8b 
[19098.829463]  panic+0xe4/0x237 
[19098.829738]  do_exit+0xbb9/0xbc0 
[19098.830039]  do_group_exit+0x47/0xb0 
[19098.830376]  get_signal+0x27b/0x590 
[19098.830706]  do_signal+0x37/0x680 
[19098.831003]  ? __probe_kernel_read+0x5a/0x90 
[19098.831420]  exit_to_usermode_loop+0x6e/0xb0 
[19098.831825]  prepare_exit_to_usermode+0x74/0x80 
[19098.832261]  retint_user+0x8/0x10 
[19098.832601] RIP: 0033:0x7f1bd6dc1191 
[19098.832956] RSP: 002b:00007ffdf09f0f40 EFLAGS: 00010202 
[19098.833454] RAX: 000056079047fea0 RBX: 000000000000000b RCX: 00007ffdf09f0f40 
[19098.834106] RDX: 0000000000000000 RSI: 00007ffdf09f1120 RDI: 0000000000000011 
[19098.834725] RBP: 0000000000006532 R08: 0000000000000000 R09: 0000000000000000 
[19098.835384] R10: 0000000000000004 R11: 0000000000000246 R12: 00007ffdf09f1120 
[19098.836037] R13: 0000000000000000 R14: 00007ffdf09f18a0 R15: 0000000000000000 
[19098.837392] Kernel Offset: 0x1f000000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff) 
[19098.838401] ---[ end Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000008b
====
Comment 1 David Smith 2017-10-06 20:45:47 UTC 
I tried to simplify this a bit, so here's an update. First, I reconfigured without dyninst, to get rid of all those dyninst segfaults. I also turned off systemd-coredump. With those changes, I got the following console output:

====
[ 2059.437921] stap_39a623e80f942e3f1ad318c0a09bc408_8846: loading out-of-tree module taints kernel. 
[ 2059.451140] stap_39a623e80f942e3f1ad318c0a09bc408_8846: module verification failed: signature and/or required key missing - tainting kernel 
[ 3040.947653] stap_96032c2366fb4c972bf2a4c92e8671d5_7864: `z29' invalid for parameter `var1' 
[ 3041.218072] stap_96032c2366fb4c972bf2a4c92e8671d5_7875: unknown parameter 'var2' ignored 
[ 5490.898599] Systemtap Error at _stp_mempool_init:50  
[ 5490.898601] Memory allocation failed. 
[ 5490.911076] Systemtap Error at _stp_register_ctl_channel:738  
[ 5490.911077] Error creating systemtap control channel. 
[ 5834.338830] stap_f8b3c6b1576e3ecc57deda9d89da28dd_6016: unknown parameter 'pt' ignored 
[ 7471.401936] hrtimer: interrupt took 1278 ns 
[ 7784.646918] Scheduler tracepoints stat_sleep, stat_iowait, stat_blocked and stat_runtime require the kernel parameter schedstats=enable or kernel.sched_schedstats=1 
[ 8740.442413] nr_pdflush_threads exported in /proc is scheduled for removal 
[18595.783147] traps: beah-rhts-task[855] trap invalid opcode ip:7f0608c1d080 sp:7ffc6fde41c8 error:0 in libc-2.26.90.so[7f0608b9b000+1ae000] 
[18595.790367] auditd[522]: segfault at 1 ip 00007fa2242a2260 sp 00007ffe50a84138 error 6 in libc-2.26.90.so[7fa2241a0000+1ae000] 
[18595.791510] systemd-journal[442]: segfault at 7fd25fec4268 ip 00007fd25fec4262 sp 00007fff403e1068 error 7 in libc-2.26.90.so[7fd25fdc2000+1ae000] 
[18595.799273] systemd[1]: segfault at 2 ip 00007effb34fc260 sp 00007ffdbc92d1b8 error 6 in libc-2.26.90.so[7effb33fa000+1ae000] 
[18595.801984] systemd[1]: segfault at 1 ip 00007effb34fc200 sp 00007ffdbc92c988 error 6 in libc-2.26.90.so[7effb33fa000+1ae000] 
[18595.803362] systemd[1]: segfault at 1 ip 00007effb34fc200 sp 00007ffdbc92c188 error 6 in libc-2.26.90.so[7effb33fa000+1ae000] 
[18595.808521] systemd[1]: segfault at 1 ip 00007effb34fc200 sp 00007ffdbc92b988 error 6 in libc-2.26.90.so[7effb33fa000+1ae000] 
[18595.809739] systemd[19961]: segfault at 4df9 ip 00007effb342e800 sp 00007ffdbc92ca88 error 6 in libc-2.26.90.so[7effb33fa000+1ae000] 
[18595.813196] audit: type=1701 audit(1507318677.561:235): auid=4294967295 uid=0 gid=0 ses=4294967295 subj=system_u:system_r:init_t:s0 pid=19961 comm="systemd" exe="/usr/lib/systemd/systemd" sig=11 res=1 
[18595.814920] systemd[19962]: segfault at 4dfa ip 00007effb342e800 sp 00007ffdbc92c288 error 6 in libc-2.26.90.so[7effb33fa000+1ae000] 
[18600.785023] show_signal_msg: 10925 callbacks suppressed 
[18600.785600] systemd[27233]: segfault at 6a61 ip 00007effb342e800 sp 00007ffdbc213288 error 6 
[18600.785665] systemd[27234]: segfault at 6a62 ip 00007effb342e800 sp 00007ffdbc212a88 error 6 
[18600.786582] systemd[27235]: segfault at 6a63 ip 00007effb342e800 sp 00007ffdbc212288 error 6 
[18600.787565] systemd[27237]: segfault at 6a65 ip 00007effb342e800 sp 00007ffdbc211288 error 6 
[18600.790231] systemd[27236]: segfault at 6a64 ip 00007effb342e800 sp 00007ffdbc211a88 error 6 
[18600.791735] systemd[27238]: segfault at 6a66 ip 00007effb342e800 sp 00007ffdbc210a88 error 6 
[18600.792609]  in libc-2.26.90.so[7effb33fa000+1ae000] 
[18600.794108] systemd[1]: segfault at 1 ip 00007effb34fc200 sp 00007ffdbc210188 error 6 in libc-2.26.90.so[7effb33fa000+1ae000] 
[18600.795487]  in libc-2.26.90.so[7effb33fa000+1ae000] 
[18600.797543]  in libc-2.26.90.so[7effb33fa000+1ae000] 
[18600.798533]  in libc-2.26.90.so[7effb33fa000+1ae000] 
[18600.800502]  in libc-2.26.90.so[7effb33fa000+1ae000] 
[18600.801140]  in libc-2.26.90.so[7effb33fa000+1ae000] 
[18600.801780] systemd[27239]: segfault at 6a67 ip 00007effb342e800 sp 00007ffdbc210288 error 6 in libc-2.26.90.so[7effb33fa000+1ae000] 
[18600.803228] audit: type=1701 audit(1507318682.552:3880): auid=4294967295 uid=0 gid=0 ses=4294967295 subj=system_u:system_r:init_t:s0 pid=27234 comm="systemd" exe="/usr/lib/systemd/systemd" sig=11 res=1 
[18600.805702] audit: type=1701 audit(1507318682.554:3881): auid=4294967295 uid=0 gid=0 ses=4294967295 subj=system_u:system_r:init_t:s0 pid=27233 comm="systemd" exe="/usr/lib/systemd/systemd" sig=11 res=1 
[18601.261136] Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000008b 
[18601.261136]  
[18601.262065] CPU: 0 PID: 1 Comm: systemd Tainted: G           OE   4.14.0-0.rc3.git0.1.fc28.x86_64 #1 
[18601.262954] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 
[18601.263517] Call Trace: 
[18601.263795]  dump_stack+0x63/0x8b 
[18601.264146]  panic+0xe4/0x237 
[18601.264447]  do_exit+0xbb9/0xbc0 
[18601.264766]  do_group_exit+0x47/0xb0 
[18601.265128]  get_signal+0x27b/0x590 
[18601.265506]  do_signal+0x37/0x680 
[18601.265845]  ? __probe_kernel_read+0x5a/0x90 
[18601.266273]  exit_to_usermode_loop+0x6e/0xb0 
[18601.266694]  prepare_exit_to_usermode+0x74/0x80 
[18601.267146]  retint_user+0x8/0x10 
[18601.267476] RIP: 0033:0x7effb34fc200 
[18601.267830] RSP: 002b:00007ffdbc12e188 EFLAGS: 00010206 
[18601.268343] RAX: 0000000000000001 RBX: 0000000000000001 RCX: 0000000000000004 
[18601.269038] RDX: 00007ffdbc12e2a0 RSI: 0000000000006df5 RDI: 0000000000000001 
[18601.270369] RBP: 0000000000006df5 R08: 0000000000000000 R09: 0000000000000000 
[18601.271048] R10: 0000000000000004 R11: 0000000000000246 R12: 00007ffdbc12e2a0 
[18601.271740] R13: 0000000000000004 R14: 00007ffdbc12e2a0 R15: 0000000000000000 
[18601.273159] Kernel Offset: 0x8000000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff) 
[18601.274204] ---[ end Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000008b 
====

I was doing a "tail -f" on systemtap.log, and the last test I saw being run was 
 testsuite/semok/twenty.stp. Now, that test does a lot of wildcarding. After a reboot, I ran that test by hand, then did a free:

====
# free -h
              total        used        free      shared  buff/cache   available
Mem:           3.9G         31M        3.8G          0B         60M        3.7G
Swap:          3.0G        156M        2.8G
====

So, that test doesn't appear to be using a ton of memory.
Comment 2 David Smith 2017-10-09 13:39:00 UTC 
(In reply to David Smith from comment #1)
> So, that test doesn't appear to be using a ton of memory.

Well, it looks like I was wrong.

If I remove testsuite/semok/twenty.stp, the entire testsuite will run to completion. So, it is definitely semok/twenty.stp. What's odd of course is that I can run semok/twenty.stp by itself with no problem.

I modified testsuite/systemtap.base/semok.exp to run "top" before each test. Here's what I see right before trying to run twenty.stp:

====
top - 01:19:58 up  5:08,  0 users,  load average: 1.05, 1.25, 1.18
Tasks: 102 total,   1 running,  57 sleeping,   0 stopped,   0 zombie
%Cpu(s): 80.3 us, 10.2 sy,  0.0 ni,  8.9 id,  0.1 wa,  0.5 hi,  0.1 si,  0.0 st
KiB Mem :  4044076 total,  2474496 free,   284012 used,  1285568 buff/cache
KiB Swap:  3145724 total,  3145724 free,        0 used.  3425984 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S %CPU %MEM     TIME+ COMMAND
  852 root      20   0  184284  45968  11452 S  0.0  1.1   0:00.59 beah-beaker+
  864 root      20   0  179072  45144  11188 S  0.0  1.1   0:10.24 beah-rhts-t+
  849 root      20   0  167260  39216  11012 S  0.0  1.0   0:00.46 beah-srv
  850 root      20   0  151916  36756   9448 S  0.0  0.9   0:00.44 beah-fwd-ba+
  686 root      20   0  173460  33448  32332 S  0.0  0.8   0:00.17 sssd_nss
 8428 root      20   0   74640  25952   4544 S  0.0  0.6   0:11.79 expect
  453 root      20   0   97340  16836  16092 S  0.0  0.4   0:00.57 systemd-jou+
  557 root      20   0  599284  15512  12388 S  0.0  0.4   0:00.51 NetworkMana+
Running /root/src.copy/testsuite/semok/twenty.stp
====

That's a lot of memory & swap still available. I wonder if the bpf changes have upped our memory requirements.
Comment 3 David Smith 2017-10-09 19:18:55 UTC 
(In reply to David Smith from comment #2)
> I wonder if the bpf changes
> have upped our memory requirements.

Here's the pre-bpf code on this rawhide system:

====
# ./stap -vwp2 -e 'probe kernel.function("*") {} probe module("*").function("*") {} probe kernel.function("*").call {} probe module("*").function("*").call {} probe kernel.function("*").return {} probe module("*").function("*").return {} probe kernel.function("*").inline {} probe module("*").function("*").inline {}' > /dev/null
Pass 1: parsed user script and 467 library scripts using 143236virt/47356res/8128shr/39336data kb, in 120usr/30sys/147real ms.
Pass 2: analyzed script: 2100235 probes, 0 functions, 0 embeds, 0 globals using 4998228virt/3816824res/1660shr/4894328data kb, in 73530usr/9430sys/122315real ms.
====

Here's the current HEAD code on the same system:

====
# /usr/local/bin/stap -vwp2 -e 'probe kernel.function("*") {} probe module("*").function("*") {} probe kernel.function("*").call {} probe module("*").function("*").call {} probe kernel.function("*").return {} probe module("*").function("*").return {} probe kernel.function("*").inline {} probe module("*").function("*").inline {}' > /dev/null
Pass 1: parsed user script and 469 library scripts using 143748virt/47616res/7852shr/39692data kb, in 130usr/30sys/470real ms.
Pass 2: analyzed script: 2100235 probes, 0 functions, 0 embeds, 0 globals using 5031620virt/3829572res/1592shr/4927564data kb, in 70740usr/9160sys/109159real ms.
====

So, in pass 2 we've gone from:

virt: 4998228 -> 5031620
res:  3816824 -> 3829572
shr:  1660    -> 1592
data: 4894328 -> 4927564

So, the memory use has been upped a bit, but nothing dramatic.
Comment 4 Frank Ch. Eigler 2020-02-19 21:39:03 UTC 
The twenty.stp test is our most memory-hungry one.  On a modern box, seeing such numbers:

stap -vuwp2 -e 'probe kernel.function("*") {} probe module("*").function("*") {} probe kernel.function("*").call {} probe module("*").function("*").call {} probe kernel.function("*").return {} probe module("*").function("*").return {} probe kernel.function("*").inline {} probe module("*").function("*").inline {}' > /dev/null
Pass 1: parsed user script and 505 library scripts using 373036virt/143836res/13180shr/130332data kb, in 250usr/40sys/291real ms.
Pass 2: analyzed script: 2722800 probes, 0 functions, 0 embeds, 0 globals using 6493280virt/6016040res/14220shr/6250576data kb, in 58160usr/3890sys/102443real ms.

i.e., over 6 GB memory is used for probing every single function (+ all their inlined copies) in the entire kernel.  That's very high, but note that it doesn't even run any kernel module code.  So if the kernel has a problem, and it doesn't signal this as an ordinary OOM, it's a kernel bug, not a stap one.