Sultan Alsawaf [Wed, 21 Oct 2020 19:27:24 +0000 (12:27 -0700)]
task_finder_vma: rewrite using RCU to fix performance issues
The use of a single global rwlock to protect this file's hash table
results in significantly degraded performance when there are many
processes using the vma tracker in flight. A lot of time is spent
spinning on the rwlock when this happens. For exmaple, it is using
most of the CPU time in the following kernel-space CPU flame graph:
There are other code paths which would invoke the same spinlock, as in
_stp_umodule_relocate().
To remedy this, make the hash table RCU safe so we'll never block upon
reading a hash list.
We now use the hash_ptr() function to generate the hashes, and the task
pointers themselves are hashed now instead of their PID for reliability,
since PIDs are not a stable anchor point to a task struct.
While we're at it, clean up the rest of this file to bring it up to
current Linux kernel coding standards as well.
This leads to dramatic CPU time reduction when
1. the current system has a lot of running processes, or
2. some processes have a lot of DSO dependencies, and
3. also -x PID is not used for stap or staprun, and
4. there are quite a few CPU cores.
For a typical test run, we have the following CPU utilization changes:
Serhei Makarov [Wed, 21 Oct 2020 20:06:48 +0000 (16:06 -0400)]
PR26755 temporary kprobes_onthefly.exp: also disable m* on ppc
XXX Need to investigate which of the tracepoints under m* is failing. XXX
For now, blacklist these in kprobes_onthefly.exp to allow the buildbot
testsuite to finish running.
An onthefly testcase started freezing the kernel on the 'hardcore'
case which grabs lock:lock_{acquire,acquired,release,contended} tracepoints.
I'm still tracking down which kernel change caused this
since the tracepoints exist on older codebases (probably lockdep kernels),
but for now it seems that we should avoid probing these in onthefly testing.
Sultan Alsawaf [Thu, 1 Oct 2020 22:19:47 +0000 (15:19 -0700)]
PR26697: fix NULL pointer deref in get_utrace_lock()
task_utrace_struct() can return NULL via __task_utrace_struct(). This fixes
the following crash:
BUG: unable to handle kernel NULL pointer dereference at (null)
#9 [ffff8843e56ffd20] get_utrace_lock at ffffffffc08258c6 [stap_X_40544]
The reason why it can return NULL is because engine->ops is protected by
utrace->lock, but we don't have the utrace pointer, and the purpose of
get_utrace_lock() is to get the utrace pointer. Therefore, there's no way
to ensure engine->ops remains unchanged inside get_utrace_lock(), so
get_utrace_lock()'s checks on engine->ops can be incorrect/stale, which
leads to the NULL pointer dereference.
Previous logic for detecting whether module-signing was required
was broken by the presence of non-systemtap MOK keys. This led
the client to find no matching stap-servers, leading to no
compilation attempt or MOK assignment. New code filters local
MOK keys for Systemtap ones only, and even in an absence,
communicates the need for a signature via a "missing" marker.
The server eagerly passes back (new or old) MOK keys to such a
client now.
Tested on a rhel8 uefi/secureboot kvm vm. Transport /sys/debug
dependencies are still blocking full function due to kernel_lockdown,
but that's coming next.
tentative fix: update hand-written Makefiles to use KBUILD_EXTMOD=
kernel 5.3+ deprecated and then removed the SUBDIRS=
flag for building an external module. In conjunction with KDIR=,
this led to unpredictable results (i.e. a build failure which
would delete the system kernel-devel package's config files (!)).
Commit 0126be38d9 counsels to use M= or KBUILD_EXTMOD= instead.
No harm in specifying both since newer kernels should ignore SUBDIRS=
The rhel4 era (<2.6.15) relayfs transport hasn't been tested for stap
releases in many years, and appears to have no compelling reason to
keep the code around. Let's remove the code.
The non-default ring_buffer transport hasn't even compiled in some
time, and appears to have no compelling reason to keep the code
around. Let's remove the code.
Frank Ch. Eigler [Thu, 27 Aug 2020 17:05:43 +0000 (13:05 -0400)]
man stap.1: belatedly mention type auto-casting
Way back in 2014, starting with 0fb0cac98ad48, an auto-casting
facility was added. This allows context-variable derived pointers to
retain their type information as the pointers are moved around inside
systemtap integer variables & expressions. This was only documented
in the 2.6 release NEWS. Now it's in the man page too.
1) In semantic_pass_optimize1, preserve synthetic probes even with
empty bodies, which would normally be elided. This allows
probe-conditions to be more properly initialized, and
initially-false probes to be disarmed early.
2) In c_unparser::emit_probe(), call emit_lock() based on the
needs_global_locks() lockworthiness of the proper probe (the
derived_probe being translated, rather than the other derived_probe
whose conditions are being evaluated)
This lets the new lock-pushdown.* tests work with or without -u (now
tested).
Frank Ch. Eigler [Mon, 17 Aug 2020 19:49:35 +0000 (15:49 -0400)]
PR26296: lock pushdown optimization
Implements an algorithm to push lock/unlock operations downward in the
syntax tree, to just enclose the smallest possible region that deals
with global variables. This means two common patterns run with much more
concurrency than before:
global a
probe foo {
if (condition)
{ a++ }
else
{ something_else() }
}
will only lock globals -if- the condition is true, so something_else()
would run unlocked. Also:
global a
probe foo {
if (a)
{ long_twisty_operation(); }
}
will unlock globals right after the condition is evaluated, so
long_twisty runs unlocked. Previous behaviour is avilable with
--compatible=4.3. New test case lock-pushdown.stp asserts locking
conditions throughout various relevant constructs.
PR26392: removed the "stable" flag from unwunding tapset functions
By design, stable function calls should be lightweight ones like pid()
and target(). unwinding functions like ubacktrace() are kinda expensive.
And marking them as stable calls would make them get always evaluated no
matter what.
William Cohen [Wed, 5 Aug 2020 13:43:17 +0000 (09:43 -0400)]
Make dtrace generated code work with LTO (take 2)
LTO needs to know which variables might be accessed by code. In sdt.h
there is a fair amount of assembly code which LTO cannot analyze. As
a result LTO would assume that the semaphores variables for various
userspace probes were not accessed by the generated code. The linking
would fail because LTO would optimize away the semphore variables
causing undefined references.
The fix adds the semaphore variables as input operands to the assembly
statements. This gives the LTO analyzer information that the
variables are used within the assembly statements and should not be
removed.
PR26307: adapt to kernel module_sect_attr changes in 5.8+
Linux v5.8 rc5ish, commit ed66f991bb19, introduced a change to the
internal module_sect_attr types. It caused a kernel BUG: with
pr14546.exp. This showed it's futile to keep chasing it this
particular way in runtime/transport/symbols.c, so for new enough
kernels (4.7+), a method based on
kernel_read_file_from_path("/sys/module/$MODULE/sections/$SECTION") is
used to extract module section base-addresses.
This is done by a new paper-thin abstraction. Tested on RHEL7, F32,
rawhide.
PR26249: "%p" -> "0x%lx" pointer formatting in *conversions.stp error messages
function::user_string() and function::user_string_warn() in uconversions.stp
were passing an "unsigned long" to the "%p" format-specifier, which upset gcc10.
Switch to 0x%lx. In the process, change other uses of %p to use 0x%lx instead,
changing casts from (void*) to (unsigned long) where necessary.
Fixes the following error on gcc10 (Fedora 32):
In function ‘function___global_user_string_n_warn__overload_1’:
/tmp/stapXXXXXX/stap_[...]_src.c: error: format ‘%lx’ expects
argument of type ‘long unsigned int’, but argument 5 has type
‘void *’ [-Werror=format=]
PR25568 / RHBZ1857749: buildid/uprobes/inode rework, task_finder etc. side
During work on a new stress tests for build-id based probes (coming in
next commit), it was found that the task_finder2 logic for buildid
verification didn't, well, work, because it was never run (due to an
erroneous pathlen conditional), and couldn't be safely run where it
was (because it was under spinlock but would have done
access_process_vm). Reworked the relevant bits of task_finder2 to
perform build-id verification for processes later - during the quiesce
callback periods. (Buildid verification for solibs is already done
in the task_finder2 consumer uprobes-inode.c.)
Testing with sdt_misc indicated a case where a preexisting process
(with solib sdt.h semaphores) was being attached to by a new stap
binary. task_finder2's enumeration of the preexising processes'
memory map segments violated assumptions by recent code related to
tracking in stapiu_process[] lists. (It did not mirror the temporal
ld.so mmap sequence.) Changed this tracking to use the inode* as the
key, and stop trying to track mapping lengths, to make positive
matches and eliminate duplicate stapiu_process[] entries for the same
(process,solib) permutation. Reworked stapiu_process[] accumulation
generally to move to the two immediate task_finder callbacks, out of
stapiu_change_plus().
Added lots of commentary and diagnostics throughout. stap
-DDEBUG_UPROBES give meaningful info about uprobes & sdt semaphores;
with -DDEBUG_TASK_FINDER, more but not overwhelming relevant info
appears.
rhbz1857749: uprobes-inode regression in sdt semaphore setting
Previous code neglected to set sdt.h semaphores for more than the
first process systemtap happened to encounter. This was from a
mistaken understanding of what it meant for stapiu_change_plus() to be
called with the same inode/consumer combination. Even though uprobes
are automatically shared, each new process still needs its perfctr and
sdt-semaphores individually set, so we do that now (as before the
rework of this code). Mechanized testing incoming shortly.
configury: make systemtap buildable with gcc -flto
Two problems corrected:
1 - Some of the stap object files compiled the interned_string type
differently based on whether config.h was #included early enough
or not. Now they all do it at the top.
2 - The staprun/libstrfloctime.a pseudo-archive (added in commit 0a5f4aa83e5b4 to work around an autoconf limitation), needs a
-Wl,--whole-archive wrapper around it for lto linking purposes.
Now also stap builds on f32 both with and without:
William Cohen [Tue, 14 Jul 2020 20:36:24 +0000 (16:36 -0400)]
Make dtrace generated code work with LTO
LTO will attempt to remove variables if they are not clearly as used
in assembly code and the linking will fail with errors about undefined
variables. The semaphores variables in the sdt.h macros need to be
marked as global to ensure that LTO doesn't incorrect remove them.
William Cohen [Mon, 13 Jul 2020 20:10:22 +0000 (16:10 -0400)]
Make dtrace generated code work with LTO
When variables are marked with the unused attribute LTO will attempt
to remove variables when they are only used by assembly code. Marking
the semaphores variables generated by the dtrace with the used
attribute will ensure that LTO doesn't incorrect remove them.
PR26234: stapbpf should warn about unsupported utrace_derived_probes
Probes such as process.begin are handled by task_finder, which is
not supported by BPF. Until some workaround is found, print
a warning instead of silently ignoring the probes.
testsuite: block listing_mode_sanity large tests on small machine
Use a /proc/meminfo based heuristic to block two particularly
cpu/ram-costly tests on small machines, for same reasons as we nuked
semok/twenty.stp some time ago.
RHBZ1847676 cont'd: one more uprobes-inode/onthefly concurrency control
In uprobes-inode.c (stapiu_change_plus), the runtime can react to
arrivals of new mappings of a solib or executable by registering new
uprobes. Due to an assumption that this could not happen at
inconvenient times (such as a stapiu_refresh or near shutdown times),
the actual uprobes registration operation was done outside the
consumer_lock mutex being held. But it appears this can happen at bad
times, so the mutex needs to be held, just like within
stapiu_consumer_refresh().
The onthefly tests now survive iterating testing on rawhide+lockdep
and rhel7+lockdep.
On recent kernels, printk("...%p...") pointers are obfuscated by
default. Tweak the systemtap startup dmesg to present a real pointer
(via "%lx"), which is essential for troubleshooting.
run-stap.in: forget about bundled elfutils build mode
The "run-stap" script in the build tree allows developers to run
a freshly built, un-installed copy of systemtap. It still knew of
our former bundled-elfutils build mode. That has been removed,
so this logic is now gone from this script.
PR25549: statement probe visibility for openmp / lto binaries
gcc -flto and -fopenmp can synthesize function bodies that may miss
DWARF attributes such as AT_decl_file. Previously, systemtap included
a check to match the value of that attribute against a list of file
names extracted from the source-file directory, and would reject any
function DIE that doesn't match. So these -flto/-fopenmp functions
were invisible to systemtap probes. We now override this for the case
for functions that completely lack that attribute, to expose these to
normal dwarfy probing. New test case included (using -fopenmp).
Martin Cermak [Thu, 9 Jul 2020 07:19:01 +0000 (09:19 +0200)]
Tapset and testsuite updates against @cast() change 00ee19ff03
Commit 00ee19ff030f665df7e087a579f39105256a0253 changed how @cast()
operations work and they no longer default to using the kernel
debuginfo for type information. Need to include kernel as location for
this information for the @cast() rather than just assuming a default.
These are respective tapset and testsuite minor updates.
RHBZ1847676 cont'd: more uprobes-inode/onthefly concurrency controls
The systemtap.onthefly/*.exp tests had recently become hang-prone on
some kernels, for reasons still not completely understood. This set
of patches adds:
- irq*-block spinlocks into uprobes-invoked paths, in case there is
peculiar reentrancy (from irq-related tracepoints)
- a mutex lock/unlock into the stapiu_exit() path, in case there is
a concurrent stapiu_refresh() invoked by onthefly machinery around
exit time
- restrictions into the onthefly module_refresh() translator code to
preclude STAP_SESSION_STOPPING as a time to do any sort of refresh
operation. Now probes that were disarmed will stay disarmed during
probe-end/error/etc. processing, which is always valid with the
spec, and avoids a class of late module-refresh ops
Testing on rhel7 and rawhide indicates the reproducible hang is gone.
Our testsuite already tortures this code; invoke by hand via:
% sudo make installcheck RUNTESTFLAGS="-v affection.exp hrtimer_onthefly.exp kprobes_onthefly.exp tracepoint_onthefly.exp uprobes_onthefly.exp"
Some versions of gcc complain about an error-message
formatting mismatch:
'probe begin {println(user_string_n_warn(0, 20)) }'
->
/var/tmp/stapk2aFPs/stap_b9f8a6b29bbfa7f7e051c7587bbf7762_1907_src.c:288:40: error: format ‘%lx’ expects argument of type ‘long unsigned int’, but argument 5 has type ‘void *’ [-Werror=format=]
288 | "user string copy fault 0x%ld at %lx [man error::fault]", rc,
This switches to "0x%lx", (uintptr_t) consistently in related functions.
Frank Ch. Eigler [Tue, 30 Jun 2020 23:04:16 +0000 (19:04 -0400)]
PR26142: adapt to linux mmap_sem api transition
With upstream linux 5.8-bound commit da1c55f1b27, the
mm_struct->mmap_sem field is renamed, and a new <mmap_lock.h> api is
introduced to lock/unlock it. Adapting only the most recent runtime
parts (task_finder2.c and access_process_vm.h) to the new api by
providing autoconf-driven macro wrappers. Other
backward-compatibility parts of the stap runtime still use mmap_sem,
but those are only compiled on old kernels, so don't appear to need
this porting. Tested on 5.8.0-0.rc2.20200626git4a21185cda0f.1.fc33.x86_64
and 5.6.19-300.fc32.x86_64.
Frank Ch. Eigler [Tue, 30 Jun 2020 19:24:18 +0000 (15:24 -0400)]
PR26142: Adapt to linux/vermagic.h file hiding
Linux commit 51161bfc66a68 (2020-04-19) enforces a convention of not
including <linux/vermagic.h> from normal source files. Switch to
<generated/utsrelease.h>, which has UTS_RELEASE, which is all that we
really wanted anyway.
Martin Cermak [Mon, 29 Jun 2020 14:30:34 +0000 (16:30 +0200)]
PR26181: Use explicit @cast() within get_ip_from_client()
Commit 00ee19ff030f665df7e087a579f39105256a0253 changed how @cast()
operations work and they no longer default to using the kernel
debuginfo for type information. Need to include kernel as location for
this information for the @cast() rather than just assuming a default.
Also, fix the type of server_ip, which historically had been a long,
but since systemtap_v >= "4.3", it is a string.
William Cohen [Mon, 22 Jun 2020 15:28:32 +0000 (11:28 -0400)]
Use explicit @cast() operators pointing to kernel for tapsets
Commit 00ee19ff030f665df7e087a579f39105256a0253 changed how @cast()
operations work and they no longer default to using the kernel
debuginfo for type information. Need to include kernel as location for
this information for the @cast() rather than just assuming a default.
PR26131: garbled data might appear in staprun data channel output
The relay v2 data writing lacked inode locking protection against
concurrent readers (on the stapio side).
The garbled data often contains '\0' bytes and might replace good data
or get appended to the good data stream.
Because the context of relay data writers may not allow sleeping, we use
a simple bounded spinlock to acquire the inode lock at our best effort
(up to about 300K cycles). Data loss might occur when we fail to obtain
the inode lock in which case it will be recorded as a "transportation
failure" and will get reported as a warning message. Increasing the
buffer size via 'staprun -b SIZE' would help reduce the risk of lock
contention.
The relay v1 implementation is not covered in this patch since it is old
and not even used on CentOS 6.
Succcessfully stress-tested on CentOS 6, CentOS 7, and Fedora 28. Also
tested with the kernel-debug packages' kernels on CentOS 7 and Fedora 28.
William Cohen [Thu, 18 Jun 2020 21:14:30 +0000 (17:14 -0400)]
Use kernel.trace("sched:sched_process_fork") for kprocess.create when possible
With optimization the copy_process function is often inlined making it
impossible for kprocess.create to probe the return of the copy_process
function. The equivalent tracepoint sched:sched_process_fork should
be used instead to avoid that issue. This change allows the
forktracker.stp and spawn_seekeer.stp examples to run even on kernels
where copy_process has been inlined.
William Cohen [Thu, 18 Jun 2020 17:32:50 +0000 (13:32 -0400)]
Use explicit @cast() operators for periodic.stp
Commit 00ee19ff030f665df7e087a579f39105256a0253 changed how @cast()
operations work and they no longer default to using the kernel
debuginfo for type information. Need to include kernel as location for
this information for the @cast() rather than just assuming a default.
Serhei Makarov [Wed, 17 Jun 2020 19:48:56 +0000 (15:48 -0400)]
PR24758: increase stack size for BPF userspace interpreter
Requires parametrizing MAX_BPF_STACK by the target.
* bpf-internal.h (MAX_BPF_KERNEL_STACK): new constant.
(MAX_BPF_USER_STACK): new constant.
(MAX_BPF_STACK): change to a macro depending on target.
(program::use_tmp_space): pass target to MAX_BPF_STACK.
* bpf-opt.cxx (alloc_literal_str): pass target to MAX_BPF_STACK.
(spill): pass target to MAX_BPF_STACK.
* stapbpf/bpfinterp.cxx (bpf_interpret): increase stack size
to match MAX_BPF_USER_STACK in bpf-internal.h.
William Cohen [Wed, 17 Jun 2020 17:39:20 +0000 (13:39 -0400)]
Use explicit @cast() operators for pfiles.stp and ioctl_handler.stp
Commit 00ee19ff030f665df7e087a579f39105256a0253 changed how @cast()
operations work and they no longer default to using the kernel
debuginfo for type information. Need to include kernel as location for
this information for the @cast() rather than just assuming a default.
William Cohen [Wed, 17 Jun 2020 17:13:43 +0000 (13:13 -0400)]
Remove the unneeded test_support check the lwtools meta info
The test_support is to check to see if some needed feature or probe
point is available. Using the example itself as the test for support
is not useful, so removing them.
William Cohen [Wed, 17 Jun 2020 17:08:30 +0000 (13:08 -0400)]
Use explicit @cast() operators to fslatency-nd.stp and fsslower-nd.stp
Commit 00ee19ff030f665df7e087a579f39105256a0253 changed how @cast()
operations work and they no longer default to using the kernel
debuginfo for type information. Need to include kernel as location for
this information for the @cast() rather than just assuming a default.
William Cohen [Wed, 17 Jun 2020 15:57:18 +0000 (11:57 -0400)]
Fix sizeof.stp to explicitly use kernel debuginfo if one not specified
Commit 00ee19ff030f665df7e087a579f39105256a0253 changed how @cast()
operations work and they no longer default to using the kernel
debuginfo for type information. Need to use the @cast_module_sizeof()
instead of @cast_size() to use the kernel debuginfo.
Frank Ch. Eigler [Wed, 17 Jun 2020 00:35:53 +0000 (20:35 -0400)]
RHBZ1847676: uprobes-inode tweaks redux
Added (back) a spinlock to manage the stapiu_consumer -> process_list
structure, since it is occasionally travered from uprobe pre-handlers,
which are sometimes entered in atomic context (e.g. on rhel7). There,
the normal mutex_t is unsafe. So restoring a spinlock_t just for
those shortlived traversals, rhel7 and rawhide are both happy.
William Cohen [Thu, 11 Jun 2020 20:19:26 +0000 (16:19 -0400)]
Avoid exceeding space constraints for the BPF environment in mmfilepage.stp
The BPF runtime environment is very space constraint. The
mmfilepage.stp example has been reorganized to reduce the stack space
and number of BPF registers needed to run the code to allow the
generated BPF code to fit within the constraints.
William Cohen [Thu, 11 Jun 2020 19:54:32 +0000 (15:54 -0400)]
Avoid exceeding space constraints for BPF environment in mmwriteback.stp
BPF is very contrained on the space allowed when running the code.
Using too complicated printf statements caused too much stack space to
be used for temporaries and the stap compile would fail for the BPF
backend. Replacing the complicated printf with simpler string printf
statements uses less stack space and allows the BPF backend to compile
this example.
Serhei Makarov [Thu, 11 Jun 2020 00:22:06 +0000 (20:22 -0400)]
PR26019 stapbpf: don't continue startup if begin probe already exited
if a bpf begin probe sets exit status, the main thread will skip the
pause() call and immediately deallocate global data structures. Then
the perf_event_loop thread will run on the corrupted data and
occasionally fail. Example result is an assertion failure due
count_active_cpus() running on corrupted data.
Could change the main thread to join() instead of detach(), but then
all threads must be modified to listen for exit status including exit
status from begin thread. TODO Consider doing so later.
I think if the begin probe was running, it wasn't correct to start the
perf_event_loop (or PERF_EVENT_IOC_ENABLE, or any of the procfs
threads) in the first place, as those things logically happen after
the begin probe has already exited. Simply clean up and exit.
Frank Ch. Eigler [Thu, 11 Jun 2020 00:28:01 +0000 (00:28 +0000)]
uprobes-inode: rework for buildid vs. onthefly work
Tweak locking in the cleanup paths. stapiu_consumer_unreg (at
stap exit time) now does all cleanup, including its process_list,
and stapiu_change_minus (at probed process exit) does nothing.
This seems to strike a good balance between cheering up lockdep
and using a little more memory during very long-lived stap jobs.
uprobes-inode: rework for buildid vs. onthefly work
Addition of the buildid-based probes has highlighted some weaknesses
in the uprobes-inode code with respect to the interlinking & sharing
of various tracking data structures. This patch reworks the code to
simplify and document. The uprobes_onthefly.exp test case easily
survives now.
gcc compatibility: use gcc #pragma to suppress -Wtautological-compare
On some compilers / distros, the kbuild-flavour cc-options runtime
test for suppressing -Wtautological-compare does not work. So we
switch to a pragama based suppression that works as far back as rhel7
gcc 4.8.5. We need to suppress that warning because script-originated
generated code can easily include tautologies.
Serhei Makarov [Wed, 10 Jun 2020 17:47:07 +0000 (13:47 -0400)]
PR26074 fixup: disable #else paths on !USE_KALLSYMS_ON_EACH_SYMBOL
These previously #else paths should only be taken if
USE_KALLSYMS_ON_EACH_SYMBOL does not hold, not if
USE_KALLSYMS_ON_EACH_SYMBOL is true but the runtime condition is
false.
git://sourceware.org / systemtap.git / log