Frank Ch. Eigler [Tue, 10 Jan 2017 20:16:58 +0000 (15:16 -0500)]
testsuite: suppress warnings from empty strings passed to stap
A test case passed a ""-valued tcl variable to a stap command line,
which due to tcl quoting rules means passing an empty string (rather
than nothing). Use some harmless real option for the empty case,
since we don't want the "" passed through to stap.
PR21020: reorganize data passing abi for java method parameters
Under a new ABI, all java object types are supported for passing to
java-method probes, by converting them all within the JVM to strings.
The systemtap script side now gets $argX being all utf-char-pointers,
which are loaded into script-side string variables with synthetic
user_string_warn() function calls.
A backward compatibility ABI is preserved in the java helper class
as well as the translator, under --compatible=3.0.
The java testsuite is drastically simplified (and execution time
shortened) to one class. These now test string and object passing
too, with it and stapbm extended to be able to locate the HelperSDT
jar / .so files within the systemtap installation $prefix. This way,
the system /usr/lib/jvm/.... path does not need to be populated just
for stap tests to run. (We should sometime extend the tests to cover
all the argument arities, but eyeballing looks OK for now.)
elaborate.cxx: optimizations should elide elision warnings for synthetic probes
We need to check not only whether a token is associated with a
user-vs-tapset file, but also whether the file is entirely synthetic.
We don't want to warn users about the latter. Relevant to the
new java method probes' argXXX synthetic script variables.
The implementation of the system() tapset function involves sending a
ctl message via the runtime transport system, which can impose smaller
string length limits (STP_CTL_BUFFER_SIZE) than the usual systemtap
strings (MAXSTRINGLEN). Conditions like that resulted in a kernel
message (printk) being printed, which is a quiet failure.
New code increases the ctl-buffer-size from 256 to 384, and tweaks
_stp_ctl_send() to send an _stp_warn() in case of an overlength input.
The system() tapset function docs point out this possibility.
Frank Ch. Eigler [Mon, 26 Dec 2016 16:50:20 +0000 (11:50 -0500)]
staprun: stop forcing off kprobes-optimization for newer kernels
Turn off this protective measure put in place back in the 3.4 era.
We hope modern kernels are robust enough; if not, NEWS contains
workaround instructions, and we can snap this back on in staprun.
Lukas Berk [Fri, 23 Dec 2016 23:12:01 +0000 (18:12 -0500)]
java: add basic support for java objects
Previously stap was only able to probe primitive java types.
Instead of falling back on string cases if a primitive isn't found
(thus erroring out if the type doesn't match, such as an object), we
now explicitly check for the a string type, and default to converting
the object name to a string if no primitive is found and reporting that.
Originally reported: https://sourceware.org/ml/systemtap/2016-q4/msg00125.html
David Smith [Thu, 15 Dec 2016 19:09:49 +0000 (13:09 -0600)]
Add a small runtime fix to __stp_time_local_update().
* runtime/time.c (__stp_time_local_update): Call
write_seqlock_irqsave()/write_sequnlock_irqrestore() instead of
write_seqlock()/write_sequnlock() to correctly wait while interrupts are
disabled.
Josh Stone [Wed, 14 Dec 2016 19:55:07 +0000 (11:55 -0800)]
Simplify embedded-tags visitors
Both embeddedcode and embedded_expr now have tagged_p members, and a few
of the cases checking for embedded tags can now do so more directly,
rather than trying a deep traversal.
David Smith [Tue, 13 Dec 2016 22:55:36 +0000 (16:55 -0600)]
Fix get_user_pages_remote() autoconf test for 4.8 kernels.
* runtime/linux/autoconf-get_user_pages_remote-flags.c: Since later
autoconf-style test don't get the results of previous tests, remove
STAPCONF_GET_USER_PAGES_REMOTE check.
Cody Santing [Fri, 9 Dec 2016 21:40:10 +0000 (16:40 -0500)]
PR20889: metadatabase.db location
* Makefile.am: Remove DOCDIR. Copy examples to $pkgdatadir rather than $docdir.
* Makefile.in: Updated using autoreconf.
* NEWS: Mention new examples location.
* doc/Makefile.in: Updated using autoreconf.
* doc/beginners/Makefile.in: same
* interactive.cxx: Look for metadatabase.db under PKGDATADIR rather than DOCDIR.
* java/Makefile.in: Updated using autoreconf.
* systemtap.spec: Update location of examples directory.
Cody Santing [Thu, 1 Dec 2016 19:46:14 +0000 (14:46 -0500)]
socket-events.stp example revamp WIP
* testsuite/systemtap.examples/general/socket-events.stp: Made the output much more compact. Still has a few things needing to be updated, mainly occasional timing inaccuracies.
Martin Cermak [Tue, 29 Nov 2016 13:33:05 +0000 (14:33 +0100)]
Fix PR20820 by making optim_stats.exp smaller.
Avoid "soft lockups" generated by optim_stats.exp by lowering
its cpu load down. Avoid division by zero caused by rounding
errors by numerical scaling. Introduce per-arch tresholds.
Move the testcase description from optim_stats1.stp to
optim_stats.exp.
Martin Cermak [Wed, 23 Nov 2016 18:45:10 +0000 (19:45 +0100)]
Fix the boot time probing feature for RHEL6 and Fedora19+ systems.
Before this fix, the boot time probing feature was proved working
on RHEL7. This patch adds a few little fixes making the feature
work on RHEL6 and Fedora19+ systems. Changes:
configure.ac: Make the dracut bin dir configurable at the configure
time. This is because of Fedora systems that have the dracut binary
located in /usr/bin rather than /sbin which is the typical location
for RHEL. Also mark initscript/99stap/{install,check} as AC config
files.
initscript/99stap/{install,check}.in: These files are bodies of the
install() and check() functions defined in module-setup.sh extracted
into standalone executable shell scripts. The older dracut shipped
with RHEL6 expects these bits right there.
initscript/99stap/start-staprun.sh: Don't create the pid file at this
stage. This script runs very early at the "cmdline" dracut hoo time.
At this time, RHEL7 systems offer writable /run directory, so that
the pid file can be stored there, but RHEL6 systems do not have a
writable place like that. Rhel6 only has /dev writable at that
stage, but misusing /dev for the pid file purposes wouldn't be clean.
So we use a mechanism based on /proc cescribed below instead.
initscript/systemtap.in: Don't move over any pid files from
/var/run/systemtap, because we stopped creating pid files at the
boot time per above. Instead we use stap -E to add a probe
advertising the running service in /proc/systemtap.
systemtap.spec: A few simple changes making the boot time probing
feature work on RHEL6 and Fedora19 systems.
Frank Ch. Eigler [Thu, 17 Nov 2016 16:52:33 +0000 (11:52 -0500)]
PR20821/PR14924: further gradual deprecation of .return { $var } use
* tapsets.cxx (var_expanding_visitor::visit_defined_op): Don't crash
if something weird is nested beneath @defined(...) that throws a
semantic error.
(visit_target_symbol_saved_return): Use save_and_restore<> object
instead of manual save/restore steps.
(dwarf_derived_probe ctor): Ditto.
(visit_entry_op): Ditto. Also, detect nestedness in @defined()
to trigger operand expansion attempt. This is tricky.
* main.cxx (passes_0_4): If -vvv, then pretty-print the script even
if erroneous, to see aftereffects of optimization, $var expansion
etc.
* staptree.cxx (vardecl::printsig): Some synthetic globals don't have
unmangled names, so print their normal name as a fallback.
* elaborate.cxx (visit_perf_op): note that @defined(@perf("junk"))
returns the literal 1. (Recently, it merely crashed the translator
in a .return probe.)
* testsuite/buildok/fortyfour.stp: New test case.
Martin Cermak [Fri, 18 Nov 2016 15:12:36 +0000 (16:12 +0100)]
Fix PR14924 by warning when not using @entry($var) in a return probe.
tapsets.cxx: Show warning when a $var gets used in a return probe
instead of @return($var). Allow the @defined(@entry()) construct.
tapset/linux/*: Use @entry() in return probes.
testsuite/systemtap.base/at_var.exp: Use -w to suppress the newly added
warning in tests that basically check that $var == @entry($var) in a
return probe.
Frank Ch. Eigler [Thu, 17 Nov 2016 11:12:59 +0000 (06:12 -0500)]
translator pass-3: close auxiliary_outputs[] after we're done with them
Otherwise we hang onto potentially dozens of unnecessary fds into
pass4. It is messy that a translator_output doesn't necessarily take
an fstream, and can write to a generic ostream, so we can only close
in the former case.
Cody Santing [Wed, 16 Nov 2016 18:40:16 +0000 (13:40 -0500)]
Update sample command to take multiple keywords
* NEWS: Added a small mention of the new sample command.
* interactive.cxx: Sample now accepts FTS full-text index query statements rather than just one keyword.
* testsuite/systemtap.examples/examples-index-gen.pl: Update to now use the new virtual table.
* testsuite/systemtap.examples/metadatabase.db: New virtual table created that uses FTS3.
Cody Santing [Fri, 11 Nov 2016 16:47:49 +0000 (11:47 -0500)]
New example script socket-events.stp
* general/socket-events.meta: Associated meta file
* general/socket-events.stp: Tracks life cycle and syscall events of all sockets of a specified PID.
* index.html: Automatically updated using perl script to include information on new example.
* index.txt: Same
* keyword-index.html: Same
* keyword-index.txt: Same
* metadatabase.db: Same
David Smith [Thu, 10 Nov 2016 15:59:16 +0000 (09:59 -0600)]
Fix PR20735 by updating kprobes.c to avoid a soft lockup.
* runtime/linux/kprobes.c: Instead of grabbing the module mutex around
calling kallsyms_on_each_symbol(), just disable/enable preemption
instead to avoid a soft lockup.
Regressions in the testsuite indicated the first approach was not
sufficient. user_scripts include testsuite-supplied -E fragments,
which of course don't look at $*, and thus aren't appropriate for
end-of-file-parse judgement.
Switch instead to a session-stored used_args[] vector, and include a
special hack to account for tapset/argv.stp, which is the only
tapset-resident user of command line arguments. (An end-user script
that happens to require the argv.stp tapset should get credit for
using all $* parameters, even if it doesn't mention them literally.)
mips: tapsets.cxx: add dwarf register definitions for SDT uprobes
Depending on the optimisations used, the compiler may generate either
numeric register definitions or typical aliases, so define both.
Since MIPS register names start with a dollar '$', add an escape
sequence to the register names when creating regname regular
expressions.
Signed-off-by: Marcin Nowakowski <marcin.nowakowski@imgtec.com>
Martin Cermak [Wed, 5 Oct 2016 06:59:42 +0000 (08:59 +0200)]
Introduce runtime optimizations for statistical computations per PR10234.
This update improves the performance of systemtap runtime statistical
computations by optimizing out unneeded parts of the __stp_stat_add()
function. It is based on parametrizing and inlining it. The stap
translator now generates _stp_stat_add(), or _stp_pmap_add_*() calls
that have additional "optimization" parameters respective to stats
in use for given global. GCC uses this for optimizing the inlined
__stp_stat_add() calls.
The optimization effect significantly depends on compiler version,
platform architecture, and the stat operators being used for given
global. At the moment, the available stat operators are @count,
@sum, @min, @max, @avg, and @variance. The most computionally
expensive is @variance. The effect of optimizing @variance is
significant. Other stat operators are computionally chap and so
the effect of their optimizations is relatively low.
Using gcc-6.2.1-1.fc26.x86_64, the @count, @sum, @min, and @max
optimizations brings approximately 8% run time shrinkage. The
@variance optimization shrinkage is up to 70% using this compiler.
For other architectures, namely for power, the optimization is less
effective.
runtime/map-gen.c: Pass the additional optimization parameters through
the map API generator macros.
runtime/map.c: Modify _new_map_set_stat() to accept optimization
additional parameters.
runtime/map.h: Ditto.
runtime/pmap-gen.c: Pass the additional optimization parameters to
__stp_map_set*() and to _stp_pmap_add*().
runtime/stat-common.c: Add optimization params to __stp_stat_add().
runtime/stat.c: Add optimization params to _stp_stat_add().
tapsets.cxx: Generate parametrized calls to the runtime.
translate.cxx: Ditto.
testsuite/systemtap.base/optim_stats*: New testcase.
* Makefile.am: Updated using autoreconf in order to make DOCDIR available for interactive.cxx.
* Makefile.in: same
* doc/beginners/Makefile.in: same
* interactive.cxx: Added sample command, uses sqlite3.h.
* java/Makefile.in:: Updated using autoreconf in order to make DOCDIR available.
* testsuite/systemtap.examples/README: Added a small note about needing perl DBI to rerun perl script.
* testsuite/systemtap.examples/examples-index-gen.pl: As documentation files are generated, SQLite table is now populated with meta data.
* testsuite/systemtap.examples/index.html: Automatically updated documentation for new example after running perl script.
* testsuite/systemtap.examples/index.txt: same
* testsuite/systemtap.examples/keywords-index.html: same
* testsuite/systemtap.examples/keywords-index.txt: same
* testsuite/systemtap.examples/metadatabase.db: New database file that stores meta table.
David Smith [Mon, 19 Sep 2016 19:58:46 +0000 (14:58 -0500)]
Another PR20589 fix by updating user_buffer_quoted[_error]().
* tapset/uconversions.stp (user_buffer_quoted): Validate the
'outlen' parameter, ensuring it is between 0 and MAXSTRINGLEN.
(user_buffer_quoted_error): Ditto.
* tapset/linux/conversions.stp (kernel_buffer_quoted): Fix use of outlen.
* testsuite/buildok/conversions.stp: Move uconversion tests to
uconversions.stp and update list of functions tested.
* testsuite/buildok/uconversions.stp: New file.
It was reported that one version of oracle db has unwind tables in
excess of 24MB in size, and the prior stap limit of 16MB precludes
backtracing through it.
Martin Cermak [Tue, 13 Sep 2016 08:56:14 +0000 (10:56 +0200)]
Fix PR20599 by only updating needed members of statistic_decl.
After introducing @variance in commit 63ead7fa, stat_ops and
bit_shift need to be tracked within the statistic_decl. Before
this update, new histogram always rewrote these new members,
which was causing the problem described in PR20599.
elaborate.cxx: Only rewrite type, linear_low, linear_high, and
linear_step, when binding new histogram with a global.
Martin Cermak [Tue, 13 Sep 2016 08:33:09 +0000 (10:33 +0200)]
Fix PR20597 and move @avg() calculations out of runtime.
Revert those parts of commit 63ead7f, that introduce new "avg"
member of the struct stat_data and move computations of @avg()
from the code directly emitted by the translator to the runtime.
The goal of this update is both to fix PR20597, and to speed up
the runtime. Note that @variance() calculations are based on the
avg too, but the variance vehicle carries its own avg_s, which is
an average scaled by certain bit_shift for numerical precision
improvement purpose. Variance calculations are optimized out if
unneeded.
So after this change, @avg computations should be as fast as they
were before @variance has been introduced in commit 63ead7f.
elaborate.cxx: Calculate @avg based on sum and count.
translate.cxx: Ditto.
runtime/map.c: Ditto, plus remove the online @avg calculations.
runtime/stat.c: Remove online @avg calculations.
runtime/stat.h: Drop the 'avg' member from the struct stat_data.
David Smith [Mon, 12 Sep 2016 20:53:46 +0000 (15:53 -0500)]
Revert last change to flightrec*.exp testcases and add comments.
* testsuite/systemtap.base/flightrec1.exp: Revert last commit and add
comments explaining what is going on. We expect the original 'stap'
command to finish and leave stapio running in the background.
* testsuite/systemtap.base/flightrec2.exp: Ditto.
* testsuite/systemtap.base/flightrec3.exp: Ditto.
* testsuite/systemtap.base/flightrec4.exp: Ditto.
* testsuite/systemtap.base/flightrec5.exp: Ditto.
David Smith [Mon, 12 Sep 2016 18:05:44 +0000 (13:05 -0500)]
Fix PR200601 by updating __get_skb_iphdr() for newer 32-bit kernels.
* tapset/linux/ip.stp (__get_skb_iphdr): Updated for newer 32-bit
kernels. Validates the skb structure, then calls the real
skb_network_header() function.
David Smith [Mon, 12 Sep 2016 16:47:51 +0000 (11:47 -0500)]
Fix PR20589 by updating kernel_buffer_quoted().
* tapset/linux/conversions.stp (kernel_buffer_quoted): Validate the
'outlen' parameter, ensuring it is between 0 and MAXSTRINGLEN.
(kernel_buffer_quoted_error): Ditto.
gcc's indentation warnings pointed a finger of suspicion at the
new code that merges variance-related partial statistics across
stats structs. Two blocks are now conditioned on STAT_OP_VARIANCE.
PR20423: tweak error message generation for case without e->components[]
... e.g. a case where a C function returns a struct, $return would
trigger the "... is being accessed instead of member" message, but the
operative tok* is the e node, not a member of the empty
e->components[].