David Smith [Tue, 13 Dec 2016 22:55:36 +0000 (16:55 -0600)]
Fix get_user_pages_remote() autoconf test for 4.8 kernels.
* runtime/linux/autoconf-get_user_pages_remote-flags.c: Since later
autoconf-style test don't get the results of previous tests, remove
STAPCONF_GET_USER_PAGES_REMOTE check.
Cody Santing [Fri, 9 Dec 2016 21:40:10 +0000 (16:40 -0500)]
PR20889: metadatabase.db location
* Makefile.am: Remove DOCDIR. Copy examples to $pkgdatadir rather than $docdir.
* Makefile.in: Updated using autoreconf.
* NEWS: Mention new examples location.
* doc/Makefile.in: Updated using autoreconf.
* doc/beginners/Makefile.in: same
* interactive.cxx: Look for metadatabase.db under PKGDATADIR rather than DOCDIR.
* java/Makefile.in: Updated using autoreconf.
* systemtap.spec: Update location of examples directory.
Cody Santing [Thu, 1 Dec 2016 19:46:14 +0000 (14:46 -0500)]
socket-events.stp example revamp WIP
* testsuite/systemtap.examples/general/socket-events.stp: Made the output much more compact. Still has a few things needing to be updated, mainly occasional timing inaccuracies.
Martin Cermak [Tue, 29 Nov 2016 13:33:05 +0000 (14:33 +0100)]
Fix PR20820 by making optim_stats.exp smaller.
Avoid "soft lockups" generated by optim_stats.exp by lowering
its cpu load down. Avoid division by zero caused by rounding
errors by numerical scaling. Introduce per-arch tresholds.
Move the testcase description from optim_stats1.stp to
optim_stats.exp.
Martin Cermak [Wed, 23 Nov 2016 18:45:10 +0000 (19:45 +0100)]
Fix the boot time probing feature for RHEL6 and Fedora19+ systems.
Before this fix, the boot time probing feature was proved working
on RHEL7. This patch adds a few little fixes making the feature
work on RHEL6 and Fedora19+ systems. Changes:
configure.ac: Make the dracut bin dir configurable at the configure
time. This is because of Fedora systems that have the dracut binary
located in /usr/bin rather than /sbin which is the typical location
for RHEL. Also mark initscript/99stap/{install,check} as AC config
files.
initscript/99stap/{install,check}.in: These files are bodies of the
install() and check() functions defined in module-setup.sh extracted
into standalone executable shell scripts. The older dracut shipped
with RHEL6 expects these bits right there.
initscript/99stap/start-staprun.sh: Don't create the pid file at this
stage. This script runs very early at the "cmdline" dracut hoo time.
At this time, RHEL7 systems offer writable /run directory, so that
the pid file can be stored there, but RHEL6 systems do not have a
writable place like that. Rhel6 only has /dev writable at that
stage, but misusing /dev for the pid file purposes wouldn't be clean.
So we use a mechanism based on /proc cescribed below instead.
initscript/systemtap.in: Don't move over any pid files from
/var/run/systemtap, because we stopped creating pid files at the
boot time per above. Instead we use stap -E to add a probe
advertising the running service in /proc/systemtap.
systemtap.spec: A few simple changes making the boot time probing
feature work on RHEL6 and Fedora19 systems.
Frank Ch. Eigler [Thu, 17 Nov 2016 16:52:33 +0000 (11:52 -0500)]
PR20821/PR14924: further gradual deprecation of .return { $var } use
* tapsets.cxx (var_expanding_visitor::visit_defined_op): Don't crash
if something weird is nested beneath @defined(...) that throws a
semantic error.
(visit_target_symbol_saved_return): Use save_and_restore<> object
instead of manual save/restore steps.
(dwarf_derived_probe ctor): Ditto.
(visit_entry_op): Ditto. Also, detect nestedness in @defined()
to trigger operand expansion attempt. This is tricky.
* main.cxx (passes_0_4): If -vvv, then pretty-print the script even
if erroneous, to see aftereffects of optimization, $var expansion
etc.
* staptree.cxx (vardecl::printsig): Some synthetic globals don't have
unmangled names, so print their normal name as a fallback.
* elaborate.cxx (visit_perf_op): note that @defined(@perf("junk"))
returns the literal 1. (Recently, it merely crashed the translator
in a .return probe.)
* testsuite/buildok/fortyfour.stp: New test case.
Martin Cermak [Fri, 18 Nov 2016 15:12:36 +0000 (16:12 +0100)]
Fix PR14924 by warning when not using @entry($var) in a return probe.
tapsets.cxx: Show warning when a $var gets used in a return probe
instead of @return($var). Allow the @defined(@entry()) construct.
tapset/linux/*: Use @entry() in return probes.
testsuite/systemtap.base/at_var.exp: Use -w to suppress the newly added
warning in tests that basically check that $var == @entry($var) in a
return probe.
Frank Ch. Eigler [Thu, 17 Nov 2016 11:12:59 +0000 (06:12 -0500)]
translator pass-3: close auxiliary_outputs[] after we're done with them
Otherwise we hang onto potentially dozens of unnecessary fds into
pass4. It is messy that a translator_output doesn't necessarily take
an fstream, and can write to a generic ostream, so we can only close
in the former case.
Cody Santing [Wed, 16 Nov 2016 18:40:16 +0000 (13:40 -0500)]
Update sample command to take multiple keywords
* NEWS: Added a small mention of the new sample command.
* interactive.cxx: Sample now accepts FTS full-text index query statements rather than just one keyword.
* testsuite/systemtap.examples/examples-index-gen.pl: Update to now use the new virtual table.
* testsuite/systemtap.examples/metadatabase.db: New virtual table created that uses FTS3.
Cody Santing [Fri, 11 Nov 2016 16:47:49 +0000 (11:47 -0500)]
New example script socket-events.stp
* general/socket-events.meta: Associated meta file
* general/socket-events.stp: Tracks life cycle and syscall events of all sockets of a specified PID.
* index.html: Automatically updated using perl script to include information on new example.
* index.txt: Same
* keyword-index.html: Same
* keyword-index.txt: Same
* metadatabase.db: Same
David Smith [Thu, 10 Nov 2016 15:59:16 +0000 (09:59 -0600)]
Fix PR20735 by updating kprobes.c to avoid a soft lockup.
* runtime/linux/kprobes.c: Instead of grabbing the module mutex around
calling kallsyms_on_each_symbol(), just disable/enable preemption
instead to avoid a soft lockup.
Regressions in the testsuite indicated the first approach was not
sufficient. user_scripts include testsuite-supplied -E fragments,
which of course don't look at $*, and thus aren't appropriate for
end-of-file-parse judgement.
Switch instead to a session-stored used_args[] vector, and include a
special hack to account for tapset/argv.stp, which is the only
tapset-resident user of command line arguments. (An end-user script
that happens to require the argv.stp tapset should get credit for
using all $* parameters, even if it doesn't mention them literally.)
mips: tapsets.cxx: add dwarf register definitions for SDT uprobes
Depending on the optimisations used, the compiler may generate either
numeric register definitions or typical aliases, so define both.
Since MIPS register names start with a dollar '$', add an escape
sequence to the register names when creating regname regular
expressions.
Signed-off-by: Marcin Nowakowski <marcin.nowakowski@imgtec.com>
Martin Cermak [Wed, 5 Oct 2016 06:59:42 +0000 (08:59 +0200)]
Introduce runtime optimizations for statistical computations per PR10234.
This update improves the performance of systemtap runtime statistical
computations by optimizing out unneeded parts of the __stp_stat_add()
function. It is based on parametrizing and inlining it. The stap
translator now generates _stp_stat_add(), or _stp_pmap_add_*() calls
that have additional "optimization" parameters respective to stats
in use for given global. GCC uses this for optimizing the inlined
__stp_stat_add() calls.
The optimization effect significantly depends on compiler version,
platform architecture, and the stat operators being used for given
global. At the moment, the available stat operators are @count,
@sum, @min, @max, @avg, and @variance. The most computionally
expensive is @variance. The effect of optimizing @variance is
significant. Other stat operators are computionally chap and so
the effect of their optimizations is relatively low.
Using gcc-6.2.1-1.fc26.x86_64, the @count, @sum, @min, and @max
optimizations brings approximately 8% run time shrinkage. The
@variance optimization shrinkage is up to 70% using this compiler.
For other architectures, namely for power, the optimization is less
effective.
runtime/map-gen.c: Pass the additional optimization parameters through
the map API generator macros.
runtime/map.c: Modify _new_map_set_stat() to accept optimization
additional parameters.
runtime/map.h: Ditto.
runtime/pmap-gen.c: Pass the additional optimization parameters to
__stp_map_set*() and to _stp_pmap_add*().
runtime/stat-common.c: Add optimization params to __stp_stat_add().
runtime/stat.c: Add optimization params to _stp_stat_add().
tapsets.cxx: Generate parametrized calls to the runtime.
translate.cxx: Ditto.
testsuite/systemtap.base/optim_stats*: New testcase.
* Makefile.am: Updated using autoreconf in order to make DOCDIR available for interactive.cxx.
* Makefile.in: same
* doc/beginners/Makefile.in: same
* interactive.cxx: Added sample command, uses sqlite3.h.
* java/Makefile.in:: Updated using autoreconf in order to make DOCDIR available.
* testsuite/systemtap.examples/README: Added a small note about needing perl DBI to rerun perl script.
* testsuite/systemtap.examples/examples-index-gen.pl: As documentation files are generated, SQLite table is now populated with meta data.
* testsuite/systemtap.examples/index.html: Automatically updated documentation for new example after running perl script.
* testsuite/systemtap.examples/index.txt: same
* testsuite/systemtap.examples/keywords-index.html: same
* testsuite/systemtap.examples/keywords-index.txt: same
* testsuite/systemtap.examples/metadatabase.db: New database file that stores meta table.
David Smith [Mon, 19 Sep 2016 19:58:46 +0000 (14:58 -0500)]
Another PR20589 fix by updating user_buffer_quoted[_error]().
* tapset/uconversions.stp (user_buffer_quoted): Validate the
'outlen' parameter, ensuring it is between 0 and MAXSTRINGLEN.
(user_buffer_quoted_error): Ditto.
* tapset/linux/conversions.stp (kernel_buffer_quoted): Fix use of outlen.
* testsuite/buildok/conversions.stp: Move uconversion tests to
uconversions.stp and update list of functions tested.
* testsuite/buildok/uconversions.stp: New file.
It was reported that one version of oracle db has unwind tables in
excess of 24MB in size, and the prior stap limit of 16MB precludes
backtracing through it.
Martin Cermak [Tue, 13 Sep 2016 08:56:14 +0000 (10:56 +0200)]
Fix PR20599 by only updating needed members of statistic_decl.
After introducing @variance in commit 63ead7fa, stat_ops and
bit_shift need to be tracked within the statistic_decl. Before
this update, new histogram always rewrote these new members,
which was causing the problem described in PR20599.
elaborate.cxx: Only rewrite type, linear_low, linear_high, and
linear_step, when binding new histogram with a global.
Martin Cermak [Tue, 13 Sep 2016 08:33:09 +0000 (10:33 +0200)]
Fix PR20597 and move @avg() calculations out of runtime.
Revert those parts of commit 63ead7f, that introduce new "avg"
member of the struct stat_data and move computations of @avg()
from the code directly emitted by the translator to the runtime.
The goal of this update is both to fix PR20597, and to speed up
the runtime. Note that @variance() calculations are based on the
avg too, but the variance vehicle carries its own avg_s, which is
an average scaled by certain bit_shift for numerical precision
improvement purpose. Variance calculations are optimized out if
unneeded.
So after this change, @avg computations should be as fast as they
were before @variance has been introduced in commit 63ead7f.
elaborate.cxx: Calculate @avg based on sum and count.
translate.cxx: Ditto.
runtime/map.c: Ditto, plus remove the online @avg calculations.
runtime/stat.c: Remove online @avg calculations.
runtime/stat.h: Drop the 'avg' member from the struct stat_data.
David Smith [Mon, 12 Sep 2016 20:53:46 +0000 (15:53 -0500)]
Revert last change to flightrec*.exp testcases and add comments.
* testsuite/systemtap.base/flightrec1.exp: Revert last commit and add
comments explaining what is going on. We expect the original 'stap'
command to finish and leave stapio running in the background.
* testsuite/systemtap.base/flightrec2.exp: Ditto.
* testsuite/systemtap.base/flightrec3.exp: Ditto.
* testsuite/systemtap.base/flightrec4.exp: Ditto.
* testsuite/systemtap.base/flightrec5.exp: Ditto.
David Smith [Mon, 12 Sep 2016 18:05:44 +0000 (13:05 -0500)]
Fix PR200601 by updating __get_skb_iphdr() for newer 32-bit kernels.
* tapset/linux/ip.stp (__get_skb_iphdr): Updated for newer 32-bit
kernels. Validates the skb structure, then calls the real
skb_network_header() function.
David Smith [Mon, 12 Sep 2016 16:47:51 +0000 (11:47 -0500)]
Fix PR20589 by updating kernel_buffer_quoted().
* tapset/linux/conversions.stp (kernel_buffer_quoted): Validate the
'outlen' parameter, ensuring it is between 0 and MAXSTRINGLEN.
(kernel_buffer_quoted_error): Ditto.
gcc's indentation warnings pointed a finger of suspicion at the
new code that merges variance-related partial statistics across
stats structs. Two blocks are now conditioned on STAT_OP_VARIANCE.
PR20423: tweak error message generation for case without e->components[]
... e.g. a case where a C function returns a struct, $return would
trigger the "... is being accessed instead of member" message, but the
operative tok* is the e node, not a member of the empty
e->components[].
Martin Cermak [Thu, 8 Sep 2016 12:03:57 +0000 (14:03 +0200)]
Introduce new statistics operator @variance().
New statistics @variance() operator using the Welford's online algorithm
for per-cpu computation, and the Total Variance formula authored by
Niranjan Kamat and Arnab Nandi from the Ohio State University for the
cross-cpu aggregation.
Commit c4a048331603a3fe19c886498e957f5f2a577ed9 blacklists hcall_*
kernel tracepoints for powerpc to avoid recursive page faults and
kernel panic due to that. powerpc:hash_fault should be blacklisted
too, since, its placed in the same path of kernel code which is
invoked on a page fault.
tweak error message for exhausted-overloaded-functions
Correct the error message generated for "all functions exhausted" to
focus on the token of the function-call site, rather than whatever
inner statement was last executed before a "next".
Josh Stone [Fri, 26 Aug 2016 20:57:08 +0000 (13:57 -0700)]
dwflpp: cache CU entry PCs for pr15123_retry_addr
Previously it was calling getscopes(pc) to see if we're at the start of
the outermost function, which may be a preamble for -mfentry. But the
scopes are inside-out -- [0] is the innermost -- so inlines were
comparing against their own entry pc, then inappropriately applying the
mfentry workaround. It also assumes we're already focussed on the CU,
which may not be the case for .callee probes that expand across CUs.
And besides all that, dwarf_getscopes is an expensive traversal.
Instead, we can just keep a cached set of known entry PCs, built from
our existing cache of concrete functions, and check if possible mfentry
PCs are in that set.
Ravi Bangoria [Tue, 23 Aug 2016 11:24:28 +0000 (06:24 -0500)]
ppc64le: Fix LEP usage for probing
PPC64 ELF ABI v2 has a Global Entry Point and a Local Entry Point for
the functions. Debuginfo of ELF contains GEP which is same as entrypc
while symbol table contains GEP and offset, from which we can calculate
LEP. LEP is used to call function within single CU, when TOC pointer
update is not required. Placing a probe on LEP catches call from both
the GEP and the LEP but, by default, systemtap probes on GEP.
Commit b4c6a4b1cd00 ("Prioritize symbol table lookup for ppc64le") solve
this issue by storing LEP in symbol table and prioritizing symbol table
over debuginfo for ppc64le.
But there are few regression effect of this patch. Couple of examples
are given below.
1. If target program is compiled without optimization and user is
interested in function parameter, systemtap should probe after function
prologue. But above patch forces probe on LEP and which result in garbage
value of function parameter will get recorded.
$ make verbose=1 installcheck RUNTESTFLAGS='at_var.exp -v --debug'
...
# of expected passes 1
# of unexpected failures 1
2. Probe on shared library function with parameter is failing at Pass 2.
$ make verbose=1 installcheck RUNTESTFLAGS='exelib.exp -v --debug'
...
# of expected passes 10
# of unexpected failures 64
3. When symbol_name with offset is used to register kprobe, kernel itself
will find LEP and adds offset to it. Systemtap using LEP to find offset
is resulting in offset being added two times.
GEP + lep_offset (by systemtap) + lep_offset (by kernel)
This can be solved by calculating LEP only at a time of adding a probe.
That will make effect of LEP local to that area and won't have any
regression effect.
After applying patch:
$ make verbose=1 installcheck RUNTESTFLAGS='at_var.exp -v --debug'
...
# of expected passes 2
$ make verbose=1 installcheck RUNTESTFLAGS='exelib.exp -v --debug'
...
# of expected passes 74
Fixes: Commit b4c6a4b1cd00 ("Prioritize symbol table lookup for ppc64le") Reported-by: Athira Rajeev <atrajeev@linux.vnet.ibm.com>
[ Reported about issue with shared library ] Signed-off-by: Ravi Bangoria <ravi.bangoria@linux.vnet.ibm.com>
Ravi Bangoria [Tue, 23 Aug 2016 11:24:27 +0000 (06:24 -0500)]
ppc64le: Store correct function entry address in symbol_table
PPC64 ELF ABI v2 has a Global Entry Point and a Local Entry Point for
the functions. Debuginfo of ELF contains GEP which is same as entrypc
while symbol table contains GEP and offset, from which we can calculate
LEP. LEP is used to call function within single CU, when TOC pointer
update is not required. Placing a probe on LEP catches call from both
the GEP and the LEP but, by default, systemtap probes on GEP.
For ppc64le, Systemtap stores LEP in symbol table and prioritize symbol
table over debuginfo. But, storing LEP in symbol table has couple of
regression effect. As LEP is only required at a time of adding a probe,
don't store it in symbol table.
No need to prioritize symbol table as well because debuginfo and symbol
table both will contain Global Entry Point.
Revert commit b4c6a4b1cd00 ("Prioritize symbol table lookup for ppc64le")
partially.
Signed-off-by: Ravi Bangoria <ravi.bangoria@linux.vnet.ibm.com>
Felix Lu [Tue, 23 Aug 2016 13:49:57 +0000 (09:49 -0400)]
modify interactive mode load behavior
The "load" command no longer appends to the existing script. The user
is prompted if they want to overwrite the existing file if the script is
not empty.
git://sourceware.org / systemtap.git / log