Previous code in dump_symbol_tables() couldn't tolerate the case where
new elfutils opened /boot/vmlinuz* as an elf file for the kernel, and
could not extract a symbol table from that (nor an absent -debuginfo).
New code instead emits a warning and moves on. A special error code
triggers return to the dump_kallsyms() path that prior elfutils/stap
versions rely on.
Newer gcc does not recognize the fallthrough comment due to the text
following the fallthrough keyword. The "see above" doesn't seem too
valuable, so this patch drops it to fix the build.
Frank Ch. Eigler [Thu, 23 Jan 2020 18:35:30 +0000 (13:35 -0500)]
RHBZ1788662: check rcu_is_watching() before probe entry
Some tracepoints are problematic because they are called from an idle
context, where RCU/lockdep is not legal to call. On lockdep kernels,
RCU warnings and even possibly-related panics have been reported.
Kernel tracepoint handlers protect themselves by wrapping their
innards in rcu_irq_enter/rcu_irq_exit(), which flips the legality flag
back on (even during idle), but these functions are not
module-exported, and it's not clear they'd be sufficient anyway. So
we call the module-export'd rcu_is_watching() in
_stp_runtime_get_context() to reject any attempt to start a probe in
such an idling-cpu context. This covers the cpu_idle tracepoint as
well as others.
Frank Ch. Eigler [Thu, 23 Jan 2020 18:35:30 +0000 (13:35 -0500)]
RHBZ1788662: blacklist kernel cpu_idle tracepoint
This tracepoint is problematic because it is called from an idle
context, where RCU/lockdep is not legal to call. On lockdep kernels,
RCU warnings and even possibly-related panics have been reported.
Kernel tracepoint handlers protect themselves by wrapping their
innards in rcu_irq_enter/rcu_irq_exit(), which flips the legality flag
back on (even during idle), but these functions are not
module-exported, and it's not clear they'd be sufficient anyway. So
we blacklist in non-guru mode.
RHBZ1788648: parse arm64 sys/sdt.h operand format [x?, x?]
On arm64, the sys/sdt.h operand [x0, x1] was observed, and not
handled. New code in
sdt_uprobe_var_expanding_visitor::try_parse_arg_register_pair
recognizes that and parses it indirectly via
sdt_uprobe_var_expanding_visitor::try_parse_arg_effective_addr .
Sagar Patel [Tue, 12 Nov 2019 17:00:58 +0000 (12:00 -0500)]
PR22315: error handling support for stapbpf
This patch introduces enhancements for the BPF backend: try-catch block support,
error probes and an improved error tapset that works with the try-catch block
implementation.
The error tapset works as follows: if the error is called from a try block, the
program's execution will jump to the corresponding catch block. If the error is
called outside a try block, it gets registered and the program will terminate.
Epilogues are now added to each probe. The epilogue checks if an error occurred,
and calls a perf_event to print the error message. It also performs a check to
see if a MAXERRORS constraint is violated and if so, it signals a hard error to
the runtime via another perf_event.
It's worthwhile noting that error messages are copied into userspace (when the
error is registered) and then printed later (during epilogue). This is because
the BPF stack space is very limited which means the string cannot be efficiently
stored on the stack. If future versions of BPF permit more stack space, the
error message string could be stored directly onto the BPF stack instead of
storing it in userspace.
1) Added try-catch block generation and error probe handling.
2) Updated error tapset to work with try-catch statements.
3) Added new internal global variable, perf_events and BPF assembly statements.
4) Updated prologue to include a check for hard errors.
5) Designed epilogue which prints error messages and signals hard errors.
6) Added mapping for resource constraints set with -D.
7) Added test cases for the error tapset and try-catch blocks.
8) Updated NEWS.
The "Build-id mismatch" error now becomes a warning
It is normal for different binaries with the same file path coexist in
the same system via chroot or namespaces, therefore we make sure
build-id mismatch is really a warning instead of an error which leads
to non-zero exit codes of staprun.
For safety, we still make sure we do not register uprobes and etc to
wrong binaries when the build-id check fails.
Bug: MAXACTION errors + try/catch would lead to wrong src loc.
The MAXACTION error is not catchable but the try/catch code always
cleared its error message. Thus the source location reported was
inaccurate and always ended being the top-level try/catch statement's.
PR25290: process(EXE).begin probes might not get fired.
Occasionally the utrace callbacks might get called in the session state
of STAP_SESSION_STARTING so we have to be prepared to that otherwise
the user probe handler would never be called.
Also added a lot of debugging log output controlled by -DDEBUG_PROBES
and -DDEBUG_TASK_FINDER=LEVEL.
Sagar Patel [Thu, 19 Dec 2019 17:33:10 +0000 (12:33 -0500)]
PR25298: stapbpf unused blocks may cause segfault
The BPF optimizer removes unused blocks, however, it does not delete
all of the references to the deleted blocks. In the new implementation,
edges leading to the unused blocks are set as nullptr.
This increases the kernel module compilation parallelism level.
The Pass-4 is now about 700ms faster when --ldd is used for a
simple .stp script file using print_ubacktrace() on an Intel
Core i9-9900K machine.
For the dyninst runtime, the new CU is still compiled into the
resulting .so file but this backend is not using the symbols yet
and the compilation is not parallelized.
buildrun.cxx: make the stapconf_xxx.h file generation process parallellable.
This makes a typical .stp script's Pass-4 time reduces from 12.4s to
4.3s on a 8-core/16-thread system (Core i9-9900K) when stapconf_xxx.h
misses the cache.
Frank Ch. Eigler [Wed, 11 Dec 2019 20:50:10 +0000 (15:50 -0500)]
tapsets: multiple syscalls: __args type warnings
The __args symbol is used in several socket-related syscalls as a
place to put a type-enhanced pointer - or rather as two different
pointers (ulong vs uint pointers). This causes a stap warning.
Renaming this internal symbol avoids the warning. More syscalls
undoubtedly affected!
Frank Ch. Eigler [Wed, 27 Nov 2019 21:01:05 +0000 (16:01 -0500)]
stap-prep: speculative ubuntu/buildbot fixes
Attempt to eradicate recurrence of an intermittent buildbot error seen
internally, wherein debconf was complaining about "TERM unset ..." and
"PATH unset" etc.
Frank Ch. Eigler [Tue, 26 Nov 2019 23:25:20 +0000 (18:25 -0500)]
configury: drop bundled-elfutils mode
Bundling elfutils has only been necessary on systems that
lacked elfutils, and for some reason the normal autoconf
% LDFLAGS=-L/path/ CPPFLAGS=-I/path/ configure
trick didn't work. This kludge is now obstructing us as
some distro build (koji rawhide!) systems presume that even
conditional sources are real sources, and lead to failure.
So long, bundled-elfutils build mode, and thanks for all the fish.
Stan Cox [Fri, 15 Nov 2019 19:41:47 +0000 (14:41 -0500)]
Save entire register set for dyninst runtime on aarch64 and ppclle
Dyninst generated calls only allow register parms, so save register set in
chunks via successive calls to enter_dyninst_uprobe_regs:
arg1 = probe index arg2 = first regno in registers arg3... registers
* runtime/dyninst/uprobes-regs.c (enter_dyninst_uprobe_regs): New
* stapdyn/mutatee.cxx (get_dwarf_registers): Add aarch sp. Don't prune
ppcle and aarch64 register set.
(mutatee::instrument_dynprobe_target): Add dyninst snippet to generate
calls to enter_dyninst_uprobe_regs
Frank Ch. Eigler [Tue, 12 Nov 2019 16:30:09 +0000 (16:30 +0000)]
testsuite/busybox.exp: tweak
Run inferior make only partially, to build the target binary
and not any aux doc stuff (which buildrequires pod2html e.g.).
Tweak separate pass/fail messages to disambiguate $test.
Frank Ch. Eigler [Mon, 11 Nov 2019 23:11:52 +0000 (23:11 +0000)]
configury: default to --enable-pie
On fedora rawhide, -fpic/-fpie flags are sneaking in all over
the place, so plain configure/make stap builds can break if
we don't build everything that way too. So flip around --enable-pie
to default-on. Sorry, PR9922 (2009).
Serhei Makarov [Fri, 8 Nov 2019 16:46:41 +0000 (11:46 -0500)]
relevant to PR25168: add user_string to bpf tapsets, x86_64 only
previously noted https://lwn.net/Articles/803587/
After some discussion with fche, I added user_string() but for x86_64 only.
After bpf_probe_read_user() helpers are added, can extend to other architectures.
PR25174: support string auto-concat more thoroughly
The @const, @var, and @cast operators took strings that were not
subject to the usual auto-concatenation of adjacent string literals.
Fixed by calling consume_string_literals() in parser::expect_unknown().
(Sibling functions nearby are not called with tok_string so are
currently unaffected.)
Sagar Patel [Fri, 25 Oct 2019 18:18:24 +0000 (14:18 -0400)]
PR24812: support order-parametrized begin/end probes
It was found that begin probes were already order parameterized but,
end probes were not. The end probes were being added using a reverse
iterator which made them run in the wrong order. This patch fixes
this by replacing the reverse iterator with a normal iterator.
1) Replaced reverse iterator with normal iterator.
2) Added testcase which checks for correct running order.
3) Updated NEWS.
Sagar Patel [Fri, 25 Oct 2019 17:08:12 +0000 (13:08 -0400)]
PR25133: fix stabpf foreach loop segfault
There was a foreach loop example that was observed to cause a
segmentation fault. After some investigation, it was found that
one of the registers was clobbered in the optimization process.
This patch modifies the register interference implementation.
Previously, variables that were defined at an instruction were
not considered in the register interference graph. However, by
defintion, a variable is live where it is defined and wherever
it holds a value that is used in the future. With this patch,
the register interference graph also takes into account where
the variables are defined, and considers them to be live.
1) Included variables that are being defined in live set.
2) Split foreach_string.stp test probes to accomodate for stack size.
Sagar Patel [Thu, 24 Oct 2019 21:05:08 +0000 (17:05 -0400)]
PR24953 Fix: check for existence of value symbol
In commit 724b7014a, the code assumed that the foreach loop
structure always had a valid value symbol. However, this is
not true. Consider the following examples:
foreach loop (k in arr) ...
foreach loop (v = k in arr) ...
In the former case, the loop is only iterating over the keys
whereas in the latter case, both the key and value are being
iterated over. Consequently, the former loop will have a
nullptr value symbol.
This patch provides a fix by checking for the existence of a
value symbol and generating the BPF instructions accordingly.
Some of the bpf-translation.cxx code was deleted accidentally. This
caused the raw_tracepoint1.stp test to fail as the context variables
could not be resolved.
With this option combination, duplicate probe points are now included
along with their PC addresses which act as discriminators. For example,
a probe that refers to a function that is inlined in multiple places
will appear multiple times in the output with different PC addresses.
1) Modified functionality of -v -L option combination.
2) Nested printing can now be disabled in printsig(...).
Sagar Patel [Wed, 9 Oct 2019 21:12:55 +0000 (17:12 -0400)]
add test to examine impact of LTO on debuginfo
The test uses BusyBox binaries which are compiled with (-g) and
(-g -flto -ffat-lto-objects). It can be observed that when LTO
is enabled, SystemTap is unable to probe some functions and that
some probe points have source file name mismatches.
rhbz1741831: support arm64 with linux/sched.h include order
On some arm64 kernels, <linux/sched.h> needs to be #include'd
ahead of <asm/*> bits in our autoconf-* tests, or else things
break. New variant tested on these kernels plus x86-64 f30.
William Cohen [Tue, 20 Aug 2019 20:18:41 +0000 (16:18 -0400)]
stapbpf PR23478:: Initial support to allow use of multi-dimensional arrays.
The patch allows multi-dimensional arrays to be used lhs and rhs
expression. It also allows deletion and check of existence of
particular elements with multi-dimensional key values. However, this
patch does not support foreach statements with multi-dimensional
arrays. Now something like the following trivial example works with
the bpf backend:
sudo ../install/bin/stap --bpf -v -e 'global x; probe begin{ x[4,2] = 2; x[2,4]= -2; exit()} probe end{ printf("%d ", x[4,2]); printf("%d ", x[3,3]); printf("%d ", x[2,4])}'
Pass 1: parsed user script and 51 library scripts using 250084virt/23932res/9000shr/14756data kb, in 10usr/0sys/15real ms.
Pass 2: analyzed script: 2 probes, 3 functions, 0 embeds, 1 global using 250084virt/23932res/9000shr/14756data kb, in 0usr/0sys/0real ms.
Pass 3: pass skipped for stapbpf runtime in 0usr/0sys/0real ms.
Pass 4: compiled BPF into "stap_29813.bo" in 0usr/0sys/0real ms.
Pass 5: starting run.
2 0 -2 Pass 5: run completed in 0usr/0sys/3real ms.
PR24953: fix foreach (v = v1 in arr) loop behaviour
Previously, keys were correctly propagated to foreach loops, however,
values were defaulted to 0. The new implementation allows values to
be propagated along with the keys.
1) Added instructions to load corresponding values.
2) Added block to check for invalid key/value pairs.
3) Added test which checks for correct key/value pairs.
William Cohen [Wed, 25 Sep 2019 14:58:11 +0000 (10:58 -0400)]
Revise inodewatch2.stp to work on a wider range of kernels and filesystems
The inodewatch2.stp instrumented specific functions that actually
changed the the mode of the inode. Depending on the file system used
the orginally instrumented functions would miss chmod operations on
some file systems. Moved the instrumentation to the common
notify_change function that all the operations go through.
buildrun: adapt to loss of "flags" filter in linux scripts/Kbuild.include
This $(call filter,...) was used since 2008 to pass KBUILD_CFLAGS
through a mechanism that preserved O= data somehow. But this
filter is gone from the kernel as of commit cdd750bf, so add a
fallback for the filtered $KBUILD_CFLAGS directly.
Sagar Patel [Thu, 29 Aug 2019 15:40:21 +0000 (11:40 -0400)]
PR23285 (2): enable prometheus-exporter type scripts on stapbpf
The eBPF backend now supports prometheus-exporter scripts. This implementation
introduces character escaping macros enabling the array dump macros on stapbpf.
However, there are some issues with foreach loops which are used in the macros
(PR24953).
1) Developed character escaping macros helper functions.
2) Added prometheus probe tapset for stapbpf.
3) Added stapbpf procfs file path to stap-exporter.
4) Introduced a sample stapbpf prometheus-exporter script in EXAMPLES.
5) Updated NEWS.
Sagar Patel [Thu, 29 Aug 2019 15:32:17 +0000 (11:32 -0400)]
PR24926: correct printing of utf-8 characters on stapbpf
There were two bugs corrupting the string bytes and instructions. The
first bug involved the implicit sign extension of negative char values.
The second bug involved a faulty optimization (fixup_operands) which
used the incorrect instruction opcode.
1) Cast char to unsigned char before casting to uint32_t.
2) Changed opcode of optimized instruction to (BPF_STX | BPF_MEM | BPF_W).
Frank Ch. Eigler [Fri, 23 Aug 2019 23:23:13 +0000 (19:23 -0400)]
stapbpf: correct 32-bit handling of EXIT bpf-map value type
The bpf bytecode declares a 4byte->8byte map for the shared globals;
the userspace must use a 64-bit type to receive those values, lest
it suffer a most unfortunate stack smash.
Frank Ch. Eigler [Fri, 23 Aug 2019 15:26:06 +0000 (11:26 -0400)]
testsuite procfs_bpf.exp: tolerate stapbpf early exit
It was observed on some platforms that the stap/bpf test case could
abort so quickly that the dejagnu (tclsh) code to open the bpf/procfs
pipes could itself hang - and indefinitely! Replaced these tclsh
level ops with a /bin/timeout-wrapped echo or cat operation.
Frank Ch. Eigler [Wed, 21 Aug 2019 23:29:45 +0000 (19:29 -0400)]
PR23879, PR24875: fix task-finder-vma on f29+
It was reported & rediscovered that some vma-dependent runtime
facilities have been broken: @vma() and *ubacktrace(). It turns out
that modern gcc/ld.so links/loads binaries in slightly different ways
than older toolchains. Specifically, the first page of ELF files is
now loaded only r--p instead of r-xp protection flags. The
_stp_vma_mmap_cb() routine now accepts the r--p case too. It now
ignores the flags entirely.
Frank Ch. Eigler [Wed, 21 Aug 2019 01:20:40 +0000 (21:20 -0400)]
PR24904: support linux 5.2's stacktrace.c changes
The following kernel commit disabled the older struct stack_trace APIs
on architectures that support the newer stackwalk APIs. Provide an
adaptation layer to stack_trace_save_regs().
Serhei Makarov [Thu, 15 Aug 2019 15:09:06 +0000 (11:09 -0400)]
stapbpf PR23858 :: support sorting by value in foreach loop
Previously stapbpf's behaviour on this was wrong.
This slightly nightmarish bit of code handles more cases.
XXX Also need to take into account s->sort_aggr for statistics aggregates
-- currently unsupported.
To support nested foreach loops, map_get_next_key was changed to take & return
addresses-of string addresses rather than string addresses. Previously
we'd need to allocate and copy multiple string keys onto one tiny stack,
which is not feasible.
* bpf-internal.h (SORT_FLAGS): New define packs sort_column, sort_direction in one word.
(GET_SORT_COLUMN): New define unpacks sort_column from SORT_FLAGS.
(GET_SORT_DIRECTION): New define unpacks sort_direction from SORT_FLAGS.
* bpf-translate.cxx (bpf_unparser::visit_foreach_loop): String key/value iteration,
hardcode keysize on stack to 8bytes (either an int or an address of a string),
check for sort_aggr (UNSUPPORTED),
pack sort_column and sort_direction into sort_flags,
properly handle returned address of a string.
* stapbpf/bpfinterp.cxx (map_int_keys, map_str_keys): Remove typedefs.
(struct map_keys): Nightmare begins. Support 4 cases of int/str keys/values.
(convert_int_key, convert_str_key, convert_int_kp, convert_str_kp): More nightmare,
handle different cases of packing/unpacking BPF key/val into C++ key/val.
(convert_key, convert_kp): More nightmare. 'Overload' pack/unpack for ANY key/val.
(compute_key_size): More nightmare. Calculate memory required for a key.
(map_sort): New function handles initial sorting of map.
(map_next): New function handles retrieval from sorted map.
(map_get_next_key): Handle sorted int/str keys/values on different columns,
now takes strings to give a place to allocate returned str key/vals
that is not on the BPF stack.
(bpf_intepret): Pass strings to map_get_next_key.
* testsuite/systemtap.bpf/bpf_tests/foreach_string.stp: Enable full testcase.
* testsuite/systemtap.bpf/bpf_tests/foreach_pr23858.stp: New testcase.
Frank Ch. Eigler [Tue, 13 Aug 2019 19:08:41 +0000 (15:08 -0400)]
PR23285: stapbpf/procfs: use effective-userid consistently
The euid is easy and consistent in the construction of the
stapbpf/procfs path name. Use the umask() field at least for
the fifo nodes (should also on the parent directory probably).
For the stapbpf -v case, print some fifo-related diagnostics.
Split out the .stp test case into a separate file, for easier
hand-execution.
Sagar Patel [Thu, 8 Aug 2019 19:40:10 +0000 (15:40 -0400)]
PR23285 (1): enable procfs probes for stapbpf
The eBPF backend now supports procfs probes. This implementation
uses FIFO special files instead of proc filesystem files. The file
path format used is /var/tmp/systemtap-USER/MODNAME. One limitation
is that both read and write probes cannot exist for the same file.
1) Added procfs probe data structures to hold probe information.
2) Created an interface between target variables and eBPF interpreter.
3) Dedicated a single thread for each file which monitors for I/O.
4) Developed a cleaning routine and error handling mechanisms.
5) Updated NEWS and man pages.
stapbpf pr23875 bugfix :: allocate actual keysize in foreach to avoid stack clobber
* bpf-translate.cxx (bpf_unparser::visit_foreach_loop): allocate actual keysize.
* testsuite/systemtap.bpf/bpf_tests/foreach_string.stp: new partial PR23858 testcase,
added only the parts necessary to trigger a segfault when bugfix not applied.