Stan Cox [Tue, 29 Mar 2022 01:08:34 +0000 (21:08 -0400)]
Add --sign-module to enable users to mok sign their own modules
Add sign-module option. Move MOK_CONFIG_TEXT, mok_dir_valid_p, mok_sign_file,
generate_mok from stap-serverd.cxx to cscommon.cxx. Add sign_module function
to cscommon.cxx. Move MOK_PRIVATE_CERT_NAME, MOK_PRIVATE_CERT_FILE,
MOK_CONFIG_FILE to cscommon.h. Add report_error parameter to generate_mok,
sign_module, mok_dir_valid_p so they can be called from server or client. If
sign-module is requested then call sign_module from passes_0_4. stap-server
continues to mok sign using the same code path.
Sultan Alsawaf [Tue, 22 Mar 2022 23:00:31 +0000 (16:00 -0700)]
PR28974: initialize the VMA tracker before all probes
Now that task finder targets are added to __stp_task_finder_list in the
correct order, a new problem is present since the vma tracker isn't always
the first task finder target that gets registered. In these cases, the
aforementioned ordering fix actually breaks what was once working fine. One
such example is a stap script which contains only a probe.begin and uses
@var on a PIE binary; in this case, the VMA tracker's mmap callback won't
be conveniently chained onto a probe that runs earlier (since there isn't
one), and will instead run after the probe.begin's callback.
To fix this, simply initialize the VMA tracker before all probes by
decoupling it from the task finder and putting it into its own derived
probe group, which is then placed before all other derived probe groups in
all_session_groups().
William Cohen [Tue, 22 Mar 2022 17:39:45 +0000 (13:39 -0400)]
PR28958: Fix tapset macros to allow nfsd-trace.stp and task_paths.stp to work
The task_dentry_path function in the tapset was getting read faults
instead of returning valid strings describing a filesystem path. This
caused the nfsd-trace.stp example and task_path.stp in the testsuite
to not function properly.
The root cause of the problem was some cast operators in macros were
not accessing the proper struct debuginfo due to a missing "kernel" in
the cast operator. The cast operators were corrected and scripts
using the task_dentry_path function now function properly.
William Cohen [Thu, 17 Mar 2022 20:01:36 +0000 (16:01 -0400)]
Fix deviceseeks.stp example to explicitly cast queue variable
The deviceseeks.stp example was failing to build because several
uses of the queue variable were not explicitly casted and result
in the following error message:
semantic error: autocast variable '' may not be used as a structure: operator '->' at testsuite/systemtap.examples/io/deviceseeks.stp:26:8
source: queue->limits->logical_block_size :
^
Used @q_cast(queue) in place of the plain queue to correctly
cast the variable and eliminate the error.
William Cohen [Thu, 17 Mar 2022 14:58:15 +0000 (10:58 -0400)]
Remove unneeded include of <linux/nfsd/nfsfh.h> from nfsd.stp tapset
The nfsfh.h header file was removed from the kernel in 2014. The
nfsd.stp tapset attempting to include the header will cause the stap
module to fail to build. The nfsd.stp tapset does not require
anything from the the nfsfh.h header file and it can be safely
removed. This will eliminate the following systemtap examples
failures on newer kernels:
Sultan Alsawaf [Thu, 17 Mar 2022 02:13:18 +0000 (19:13 -0700)]
PR28974: runtime: add task finder targets to __stp_task_finder_list in order
There's an issue with probes which require the VMA tracker where a
probe.begin can run before the VMA tracker does, leading to unexpected
results. With a reproducer using @var to print the value of a variable in a
PIE binary, this leads to the @var failing with a NULL pointer dereference,
since the VMA tracker's mmap callback runs right after the probe.begin
rather than prior.
The VMA tracker's mmap callback is chained onto the "inode-uprobes"
stapiu_consumer, and the probe.begin's callback is chained onto the
"lifecycle tracking" stap_utrace_probe. While the order in which these are
initialized is correct (stapiu_consumer init comes before stap_utrace_probe
init), the corresponding quiescent state workers are executed by utrace in
reverse order. This happens because stapiu_consumers and stap_utrace_probes
are attached to utrace in reverse order, which is done by the task finder.
Although the task finder iterates forward through __stp_task_finder_list to
create utrace attachments, all of its targets are added to
__stp_task_finder_list in reverse order via list_add() in the first place.
To fix this, simply use list_add_tail() instead of list_add() when adding
task finder targets to __stp_task_finder_list, so that they are processed
in the order with which they are initialized.
Frank Ch. Eigler [Fri, 11 Mar 2022 15:29:55 +0000 (10:29 -0500)]
sys/sdt.h: set x86-64 STAP_SDT_ASM_CONSTRAINT back to "nor"
It turns out the kernel and some other sdt consumers haven't learned
how to use %xmm registers in sdt operands. So under this duress, stap
will go back to the old school integer register set "nor" as a
default. We'll revisit this in the future, though this egg might not
turn into a chicken.
PR28923: dtrace.in: add atexit removal & timeout to .dtrace-temp file
On erroneous inputs or other error cases, it was possible to leave
behind a .dtrace-temp*.c file at exit. That would indefinitely block
a subsequent dtrace job, due to excessive optimism in commit cfabd38cfdd75e. Now we time out, and we try harder to remove the
temp file.
William Cohen [Tue, 1 Mar 2022 21:03:54 +0000 (16:03 -0500)]
Avoid triggering error with -Werror=unused-value
Fedora RPMs are compiled using -Wall which includes
-Werror=unused-value. On architecures that do not have Dyninst
support the flush_analysis_caches define in analysis.h would trigger
the following error:
In file included from elaborate.cxx:20:
elaborate.cxx: In function ‘void build_no_more(systemtap_session&)’:
analysis.h:30:34: error: statement has no effect [-Werror=unused-value]
30 | #define flush_analysis_caches() (0)
| ~^~
elaborate.cxx:1871:3: note: in expansion of macro ‘flush_analysis_caches’
1871 | flush_analysis_caches();
| ^~~~~~~~~~~~~~~~~~~~~
cc1plus: all warnings being treated as errors
Tweaked the flush_analysis_caches define to avoid creating an unused
value.
William Cohen [Fri, 4 Feb 2022 19:18:46 +0000 (14:18 -0500)]
Clear out the Dyninst-related data structures after analysis finishes
The liveness analysis for SystemTap uses Dyninst to examine the
binaries. For large binaries such as the Linux kernel this can
consume quite a bit of memory. Once the analysis is done, the code
needs to clean up as much of that as possible.
dann frazier [Tue, 1 Mar 2022 16:02:27 +0000 (11:02 -0500)]
PR28923: dtrace: Use hash-based scheme for predictable file generation
commit c245153 ("dtrace: Allow for reproducible .o file builds.")
introduced a condition where 2 dtrace processes can race when
generating the same file. Since both processes now use the same
temporary file name, one may delete the temporary .c file the other
is still processing:
--------------------------------------------------------------------
user@host:~/foo$ make -j2
dtrace -o foo.out -G -s /dev/null
dtrace -o foo.out -G -s /dev/null
Traceback (most recent call last):
File "/usr/bin/dtrace", line 455, in <module>
sys.exit(main())
File "/usr/bin/dtrace", line 440, in main
os.remove(fname)
FileNotFoundError: [Errno 2] No such file or directory: 'foo.out.dtrace-temp.c'
make: *** [Makefile:4: ../foo/foo.out] Error 1
--------------------------------------------------------------------
This can happen when a Makefile processes a pattern rule for two different
targets that happen to map to the same file, but addressed by different
relative paths. I discovered this in a real world case involving libvirt,
but here's a contrived reproducer:
It would be ideal if we could inject a null .file directive, then we could
just use a mkstemp() file and keep the build reproducible by avoiding a
record of the source file path in the binary at all, but I can't find a
straightforward way of passing a .file through to the assembler. So,
instead, let's create a reproducible filename by building a hash of the
input and output paths. Note: this still leaves open a race in the case of
2 dtrace processes with identical input/output paths. But, at least in my
testing, GNU Make is smart enough to detect this case and not create
duplicate jobs.
Fixes: Commit c245153 ("dtrace: Allow for reproducible .o file builds.") Signed-off-by: dann frazier <dann.frazier@canonical.com>
Two variables in bpf-translate.cxx can trigger -Werror=maybe-uninitialized.
The code is designed so that uninitialized uses are not actually possible,
but to convince gcc of this we move a throw statement and initialize one
of the variables with a value.
Frank Ch. Eigler [Fri, 25 Feb 2022 01:05:41 +0000 (20:05 -0500)]
gcc12 warning suppression
The translator emits a pair of type declarations that alternate
between a char[] and a char*, depending on the size of strings
involved. The polymorphic client code includes pointer null-checking,
which -Waddress code rejects for the char[] case. The simplest
workaround is just to disable that particular diagnostic.
Stan Cox [Fri, 28 Jan 2022 20:28:27 +0000 (15:28 -0500)]
Attempt to access string in userspace if kernel access fails
Add kernel_or_user_string_quoted(_utf16 _utf32) tapsets to handle
situations where a kernelspace access was assumed but string is in
userspace. Add new kernel_user_var test. Page in the utf strings in the
utf_pretty test.
Frank Ch. Eigler [Wed, 26 Jan 2022 19:10:38 +0000 (14:10 -0500)]
PR28804: tune default stap -s ## buffer size on small RAM machines
Insert a forgotten division by num_online_cpu() to adjust downward the
calculated bufsize. Tweak normal defaults back to 128 * 2 * 64K
(16MB) per CPU, as the stap man page indicates. This may need further
tweaking when balancing against staprun consumption performance, but
at least we have the docs lined up with the code at the moment.
Serhei Makarov [Fri, 21 Jan 2022 23:21:46 +0000 (18:21 -0500)]
gcc12 c++ compatibility re-tweak for rhel6: use function pointer instead of lambdas instead of ptr_fun<>
Saving 2 lines in ltrim/rtrim is probably not a good reason to drop
compatibility with the RHEL6 system compiler. Actually declaring a
named function and passing the function pointer is compatible with
everything.
William Cohen [Tue, 18 Jan 2022 03:02:44 +0000 (22:02 -0500)]
Graceful continuation when not enough memory available for liveness analysis
The dyninst parsing of binaries can take a significant amount of
memory. On machines without enough memory to parse a large binary we
want the analysis to fail gracefully with a warning that the liveness
analysis was unable to run and continue on rather than immediately
exiting with a std::bad_alloc exception.
William Cohen [Fri, 14 Jan 2022 19:00:02 +0000 (14:00 -0500)]
configure finds appropriate default 32-bit or 64-bit Dyninst libraries
Earlier versions of the systemtap configuration would just include
two -L paths to both 32-bit and 64-bit versions of the Dyninst
libraries. However, attempting to link a 32-bit library with a 64-bit
build (and vice versa) may cause the build to fail. This revision of
the configure tests determines which default Dyninst library works
with the compiler being used and selects it.
The configure can't bindly use ${libdir}/dyninst to select the path to
the default Dyninst libary as this only selects the ${prefix}/lib64 on
appropriate machines if the prefix is /usr. If the prefix is set to
something else, ${libdir} is always ${prefix}/lib. The would cause
the build to attempt to link with nonexistent Dyninst libaries in the
${prefix} directory. If systemtap needs to be use a version of
Dysninst in a non-standard place, the --with-dyninst=<path_to_dyninst>
should be used.
Stan Cox [Fri, 7 Jan 2022 17:01:49 +0000 (12:01 -0500)]
Standardize dyninst include file use.
Change stapdyn to use the more standard '#include <dyninst/*.h>' form
and the standard include path. Adjust configure so that
--with-dyninst=PATH works, assuming PATH is a /usr style path laid out
in standard linux form.
It appears that various versions of gcc continue to show signs of
confusion at our newly offered asm-operand alternatives for floating
point sdt.h marker parameters.
Stan Cox [Wed, 1 Dec 2021 21:19:22 +0000 (16:19 -0500)]
Handle user supplied sdt probe argument template
User supplied templates were erroneously removed by commit eaa15b047,
which complicated the template expansion. To do the above the
expansion of STAP_PROBE_ASM(provider, fooprobe,
STAP_PROBE_ASM_TEMPLATE(3)) adds an unused argument:
STAP_PROBE_ASM(provider, fooprobe, /*template expansion*/ "%[SDT..]..",
"use _SDT_ASM_TEMPLATE") A supplied template
STAP_PROBE_ASM(provider, fooprobe, "4@%rdx 8@%rax") is left alone. If
the varargs has 2 args (the fake "use ..") then macro expansion
inserts the expanded string, otherwise "4@.." becomes an ascii op.
Frank Ch. Eigler [Sat, 20 Nov 2021 03:22:45 +0000 (22:22 -0500)]
configury: let python3 be python3
Our baroque heuristics for identifying python2/3 under their various
historical aliases is showing its age. On some modern distros,
/usr/bin/python is to be positively NOT used. Fixing configure.ac
$PYTHON3 search to only look for python3, and not even consider
$PYTHON_UNKNOWN. At some point we'll want to simplify further, and
get rid of python2 remnants.
Frank Ch. Eigler [Mon, 15 Nov 2021 02:59:05 +0000 (21:59 -0500)]
PR28449: runtime/transport: bump up _stp_subbuf_size to 64K
After commit cd48874296, it was reported that the kernel-side I/O
buffers could be exhausted more easily by the some tests in the
testsuite. This resulted in "bufhdr corrupted ..." type messages
coming from the syscall.exp test case for example.
We bump up the $subject variable, so without "stap -s XYZ", the
default kernel->user transport will consist of 256 subbufs of 64K
each, per cpu. (When a user does supply -s XYZ, the number gets
overridden, and the subbuf size may be quite a bit larger.) With the
new default, the syscall.exp suite runs bufhdr-clean on an 8cpu box.
Frank Ch. Eigler [Sun, 14 Nov 2021 21:28:12 +0000 (16:28 -0500)]
PR28557: module probe insertion on modern kernels
For reasons not completely understood, kernels in the 5.* range have
stopped taking module kprobes registrations of the form the runtime
has offered them. New code in runtime/linux/kprobes.c
(stapkp_prepare_kprobe) now tried to fully relocate addresses itself,
rather than letting the kernel try it with symbols / kallsyms.
Added more diagnostic prints in nearby symbol/kprobe code paths and a
few more error checks in transport.
Serguei Makarov [Thu, 4 Nov 2021 15:33:36 +0000 (11:33 -0400)]
procfs_bpf.exp PR28544: fix string handling error
This was causing a procfs_bpf.exp failure on recent RHELs
as a non-zero-padded string passed from the procfs pipe
caused buffer garbage to be passed to the stap script.
Serguei Makarov [Thu, 4 Nov 2021 12:53:11 +0000 (08:53 -0400)]
man/stap.1.in stapregex: clarify POSIX ERE features
Although stapregex was based on re2c, its current feature set implements
POSIX Extended Regular Expressions. However, a lot of POSIX ERE docs
that come up in a web search include extra stuff (e.g. the Boost doc).
Refer the user more clearly to egrep(1) for the exact list of features.
pass-2 elaborate: autocast: suppress "Potential type mismatch in reassignment"
Analysis indicates this warning is not that helpful, and extra
diagnostics just added for autocasting should help work around
polymorphism issues that this warning would have identified earlier.
William Cohen [Tue, 2 Nov 2021 15:20:27 +0000 (11:20 -0400)]
Add -faligned-new to CXXFLAGS when available to compile dyninst 10.1 code
Fedora 30 and 31 have Dyninst 10.1. The code added to dyninst
by commit a00056bec6a0265afb592944dde2ff461b525e8d makes some data
structures cache aligned (128 bytes). The c++ compiler ends up flagging that
as an error:
CXX stap-analysis.o
/notnfs/smakarov/stap-checkout/analysis.cxx: In constructor ‘analysis::analysis(std::string)’:
/notnfs/smakarov/stap-checkout/analysis.cxx:59:37: error: ‘new’ of type ‘Dyninst::ParseAPI::SymtabCodeSource’ with extended alignment 128 [-Werror=aligned-new=]
59 | sts = new SymtabCodeSource(name_str);
| ^
/notnfs/smakarov/stap-checkout/analysis.cxx:59:37: note: uses ‘void* operator new(std::size_t)’, which does not have an alignment parameter
/notnfs/smakarov/stap-checkout/analysis.cxx:59:37: note: use ‘-faligned-new’ to enable C++17 over-aligned new support
cc1plus: all warnings being treated as errors
make[2]: *** [Makefile:1724: stap-analysis.o] Error 1
To make the compiler happy adding a -faligned-new option to the
CXXFLAGS when it is available. This issue does not affect later
releases of dyninst as the problem code was removed from dyninst in
git commit d233ae7596cd74201d634c2f0f7d7a0e3d628e79.
Our type-detail inference (tagging some staptree nodes with DWARF DIE
pointers) in pass 2 was not very verbose about its work, making
type-mismatch warnings harder to diagnose. Add a ::print() virtual
function to exp_type_details() and derived classes. Call it from
resolve_details(), renamed from resolved_details().
Frank Ch. Eigler [Sat, 30 Oct 2021 23:50:21 +0000 (19:50 -0400)]
staprun/relay.c: bypass compiler warning
Newly added code did a read(2), deliberately ignoring how many bytes
were read. Some gcc versions complain. Phrase the same operation
differently to make gcc happier.
Frank Ch. Eigler [Sat, 30 Oct 2021 19:21:31 +0000 (15:21 -0400)]
PR28449: fix cross-cpu message ordering
Correct unintended loss of cross-cpu message total-ordering via commit 8819e2a04596d by packetizing normal printf() traffic. This also
restores bulk mode / stap-merge operation to same as before. staprun
now has a builtin merge equivalent built on threads, with some
message-loss fault-tolerance. A new "dumpalot" test produces ample
output with varying "-s BUFFER" runs. While in the vicinity, the old
"relay_old.c" (<=rhel5) file is nuked; it was not even compiled.
William Cohen [Thu, 28 Oct 2021 20:13:56 +0000 (16:13 -0400)]
Adjust analysis code to compile with older Dyninst 9
The Block getInsn method and InsnLoc constructor have different
signature in older Dyninst 9. To hide those differences between
version of Dyninst just put the method invocation in the constructor
and let the compiler figure out which type the parameter is.
William Cohen [Thu, 28 Oct 2021 17:38:02 +0000 (13:38 -0400)]
Tolerate dyninst10's need for -lboost_system in analysis.cxx
The analysis code uses dyninst. As a result the new analysis code in
the translator will need to link in the boost_system library like
commit 891810c246d6de05a2df80c5b3e9f9aaa13231f7 does for stapdyn.
William Cohen [Wed, 27 Oct 2021 17:49:12 +0000 (13:49 -0400)]
Make the liveness warning 32-bit agnostic
On 32-bit platforms Dwarf_Addr is long long unsigned int which doesn't
match the %lx (long unsigned int) print format to print the warning.
Cast the variable as (void *) so %p format can be used to print its
value on both 32-bit and 64-bit platforms.
William Cohen [Wed, 13 Oct 2021 19:56:21 +0000 (15:56 -0400)]
Assert that anything other than 32-bit or 64-bit processors will not be seen
One last diagnostic print to remove. In this case the mapping between
dwarf register and Dyninst register name needs to take into account
whether this is 32-bit or 64-bit code. However, there is a default in
the switch case to catch the anything other than 4 or 8 bytes. If the
code see something other than one of those two values, something is
very wrong. Figured best to just have an assert stop things, so the
problem is examined.
William Cohen [Wed, 13 Oct 2021 19:27:30 +0000 (15:27 -0400)]
Add caching for the liveness data structure
Dyninst already caches information to make future queries at the same
location less expensive for an executable. Thus, it is enough for the
code to cache the LivenessAnalyzer data on a per executable basis.
This information is stored in the cachedLivenessInfo member of
LivenessAnalyzer class.