David Smith [Tue, 25 Oct 2016 17:32:36 +0000 (12:32 -0500)]
Handle multiple python probes at the same location.
* python/HelperSDT/__init__.py (Dispatcher.pytrace_dispatch): Handle
multiple breakpoints at the same breakpoint location.
(Dispatcher.do_b): Make sure the line number field is numeric. Clear the
line number field if we've got a return break.
* tapset-python.cxx (python_builder::build): Add newlines to debug prints.
David Smith [Mon, 24 Oct 2016 20:59:00 +0000 (15:59 -0500)]
Read the list of python breakpoints from the systemtap module's procfs file.
* python/HelperSDT/__init__.py (Dispatcher.__init__): Read the list of
breakpoint commands from the procfs file exported by systemtap.
(do_b): New function
* staprun/staprun_funcs.c (insert_module): Export an environment variable
called 'SYSTEMTAP_MODULE' with the name of the inserted module.
* tapset-python.cxx (python_builder::resolve): Fix syntax.
David Smith [Fri, 21 Oct 2016 17:51:01 +0000 (12:51 -0500)]
Differentiate between python line number probes and function call probes.
* python/stap-resolve-module-function.py (resolve_patterns): Differentiate
between line number probes and function call probes.
* tapset-python.cxx (python_builder::resolve): Handle the 'call' flag
coming back from stap-resolve-module-function.py.
(python_builder::build): If needed, synthesize a new 'call' component.
David Smith [Fri, 21 Oct 2016 15:50:34 +0000 (10:50 -0500)]
Fix stap-resolve-module-function.py to handle relative paths with wildcards.
* python/stap-resolve-module-function.py (resolve_patterns): Fix searching
for wildcarded paths by appending the result of glob() onto the list
(instead of replacing the list with the result of glob()).
David Smith [Thu, 20 Oct 2016 18:44:56 +0000 (13:44 -0500)]
Fix stap-resolve-module-function.py to handle relative paths.
* python/stap-resolve-module-function.py (_get_default_search_path): New
function.
(_find_wildcarded_modules): Use _get_default_search_path() instead of
sys.path, which doesn't include the current working directory.
(_find_wildcarded_modules): Ditto.
David Smith [Wed, 19 Oct 2016 18:44:17 +0000 (13:44 -0500)]
Use the python executable names found by configure when running a script.
* tapset-python.cxx (python_builder::resolve): Use the python executable
names found by configure when running the stap-resolve-module-function.py
script.
Martin Cermak [Wed, 5 Oct 2016 06:59:42 +0000 (08:59 +0200)]
Introduce runtime optimizations for statistical computations per PR10234.
This update improves the performance of systemtap runtime statistical
computations by optimizing out unneeded parts of the __stp_stat_add()
function. It is based on parametrizing and inlining it. The stap
translator now generates _stp_stat_add(), or _stp_pmap_add_*() calls
that have additional "optimization" parameters respective to stats
in use for given global. GCC uses this for optimizing the inlined
__stp_stat_add() calls.
The optimization effect significantly depends on compiler version,
platform architecture, and the stat operators being used for given
global. At the moment, the available stat operators are @count,
@sum, @min, @max, @avg, and @variance. The most computionally
expensive is @variance. The effect of optimizing @variance is
significant. Other stat operators are computionally chap and so
the effect of their optimizations is relatively low.
Using gcc-6.2.1-1.fc26.x86_64, the @count, @sum, @min, and @max
optimizations brings approximately 8% run time shrinkage. The
@variance optimization shrinkage is up to 70% using this compiler.
For other architectures, namely for power, the optimization is less
effective.
runtime/map-gen.c: Pass the additional optimization parameters through
the map API generator macros.
runtime/map.c: Modify _new_map_set_stat() to accept optimization
additional parameters.
runtime/map.h: Ditto.
runtime/pmap-gen.c: Pass the additional optimization parameters to
__stp_map_set*() and to _stp_pmap_add*().
runtime/stat-common.c: Add optimization params to __stp_stat_add().
runtime/stat.c: Add optimization params to _stp_stat_add().
tapsets.cxx: Generate parametrized calls to the runtime.
translate.cxx: Ditto.
testsuite/systemtap.base/optim_stats*: New testcase.
* Makefile.am: Updated using autoreconf in order to make DOCDIR available for interactive.cxx.
* Makefile.in: same
* doc/beginners/Makefile.in: same
* interactive.cxx: Added sample command, uses sqlite3.h.
* java/Makefile.in:: Updated using autoreconf in order to make DOCDIR available.
* testsuite/systemtap.examples/README: Added a small note about needing perl DBI to rerun perl script.
* testsuite/systemtap.examples/examples-index-gen.pl: As documentation files are generated, SQLite table is now populated with meta data.
* testsuite/systemtap.examples/index.html: Automatically updated documentation for new example after running perl script.
* testsuite/systemtap.examples/index.txt: same
* testsuite/systemtap.examples/keywords-index.html: same
* testsuite/systemtap.examples/keywords-index.txt: same
* testsuite/systemtap.examples/metadatabase.db: New database file that stores meta table.
David Smith [Mon, 19 Sep 2016 19:58:46 +0000 (14:58 -0500)]
Another PR20589 fix by updating user_buffer_quoted[_error]().
* tapset/uconversions.stp (user_buffer_quoted): Validate the
'outlen' parameter, ensuring it is between 0 and MAXSTRINGLEN.
(user_buffer_quoted_error): Ditto.
* tapset/linux/conversions.stp (kernel_buffer_quoted): Fix use of outlen.
* testsuite/buildok/conversions.stp: Move uconversion tests to
uconversions.stp and update list of functions tested.
* testsuite/buildok/uconversions.stp: New file.
David Smith [Thu, 15 Sep 2016 21:28:47 +0000 (16:28 -0500)]
Added autoconf logic to search for python version 2 and 3.
* configure.ac: Add code to search for python version 2 and 3. Change the
'--with-python3' option to check and make sure python version 3 is
installed on the system.
* dtrace.in: Updated method to select preferred version of python.
* Makefile.in: Regenerated.
* configure: Ditto
* doc/Makefile.in: Ditto
* doc/SystemTap_Tapset_Reference/Makefile.in: Ditto
* doc/beginners/Makefile.in: Ditto
* java/Makefile.in: Ditto
* man/Makefile.in: Ditto
* man/cs/Makefile.in: Ditto
* python/Makefile.in: Ditto
* stapdyn/Makefile.in: Ditto
* staprun/Makefile.in: Ditto
It was reported that one version of oracle db has unwind tables in
excess of 24MB in size, and the prior stap limit of 16MB precludes
backtracing through it.
David Smith [Tue, 13 Sep 2016 19:30:00 +0000 (14:30 -0500)]
Rename the '--with-python' configure option to '--with-python2'.
* configure.ac: Rename the '--with-python' configure option to
'--with-python2' (and all the associated variables).
* tapset-python.cxx (register_tapset_python): Handle renamed 'HAVE_PYTHON'
to 'HAVE_PYTHON2' defined. Split python probe support into
'HAVE_PYTHON2' and 'HAVE_PYTHON3' sections.
* config.in: Ditto.
* configure: Ditto.
Martin Cermak [Tue, 13 Sep 2016 08:56:14 +0000 (10:56 +0200)]
Fix PR20599 by only updating needed members of statistic_decl.
After introducing @variance in commit 63ead7fa, stat_ops and
bit_shift need to be tracked within the statistic_decl. Before
this update, new histogram always rewrote these new members,
which was causing the problem described in PR20599.
elaborate.cxx: Only rewrite type, linear_low, linear_high, and
linear_step, when binding new histogram with a global.
Martin Cermak [Tue, 13 Sep 2016 08:33:09 +0000 (10:33 +0200)]
Fix PR20597 and move @avg() calculations out of runtime.
Revert those parts of commit 63ead7f, that introduce new "avg"
member of the struct stat_data and move computations of @avg()
from the code directly emitted by the translator to the runtime.
The goal of this update is both to fix PR20597, and to speed up
the runtime. Note that @variance() calculations are based on the
avg too, but the variance vehicle carries its own avg_s, which is
an average scaled by certain bit_shift for numerical precision
improvement purpose. Variance calculations are optimized out if
unneeded.
So after this change, @avg computations should be as fast as they
were before @variance has been introduced in commit 63ead7f.
elaborate.cxx: Calculate @avg based on sum and count.
translate.cxx: Ditto.
runtime/map.c: Ditto, plus remove the online @avg calculations.
runtime/stat.c: Remove online @avg calculations.
runtime/stat.h: Drop the 'avg' member from the struct stat_data.
David Smith [Mon, 12 Sep 2016 20:53:46 +0000 (15:53 -0500)]
Revert last change to flightrec*.exp testcases and add comments.
* testsuite/systemtap.base/flightrec1.exp: Revert last commit and add
comments explaining what is going on. We expect the original 'stap'
command to finish and leave stapio running in the background.
* testsuite/systemtap.base/flightrec2.exp: Ditto.
* testsuite/systemtap.base/flightrec3.exp: Ditto.
* testsuite/systemtap.base/flightrec4.exp: Ditto.
* testsuite/systemtap.base/flightrec5.exp: Ditto.
David Smith [Mon, 12 Sep 2016 18:05:44 +0000 (13:05 -0500)]
Fix PR200601 by updating __get_skb_iphdr() for newer 32-bit kernels.
* tapset/linux/ip.stp (__get_skb_iphdr): Updated for newer 32-bit
kernels. Validates the skb structure, then calls the real
skb_network_header() function.
David Smith [Mon, 12 Sep 2016 16:47:51 +0000 (11:47 -0500)]
Fix PR20589 by updating kernel_buffer_quoted().
* tapset/linux/conversions.stp (kernel_buffer_quoted): Validate the
'outlen' parameter, ensuring it is between 0 and MAXSTRINGLEN.
(kernel_buffer_quoted_error): Ditto.
gcc's indentation warnings pointed a finger of suspicion at the
new code that merges variance-related partial statistics across
stats structs. Two blocks are now conditioned on STAT_OP_VARIANCE.
PR20423: tweak error message generation for case without e->components[]
... e.g. a case where a C function returns a struct, $return would
trigger the "... is being accessed instead of member" message, but the
operative tok* is the e node, not a member of the empty
e->components[].
Martin Cermak [Thu, 8 Sep 2016 12:03:57 +0000 (14:03 +0200)]
Introduce new statistics operator @variance().
New statistics @variance() operator using the Welford's online algorithm
for per-cpu computation, and the Total Variance formula authored by
Niranjan Kamat and Arnab Nandi from the Ohio State University for the
cross-cpu aggregation.
Commit c4a048331603a3fe19c886498e957f5f2a577ed9 blacklists hcall_*
kernel tracepoints for powerpc to avoid recursive page faults and
kernel panic due to that. powerpc:hash_fault should be blacklisted
too, since, its placed in the same path of kernel code which is
invoked on a page fault.
tweak error message for exhausted-overloaded-functions
Correct the error message generated for "all functions exhausted" to
focus on the token of the function-call site, rather than whatever
inner statement was last executed before a "next".
Josh Stone [Fri, 26 Aug 2016 20:57:08 +0000 (13:57 -0700)]
dwflpp: cache CU entry PCs for pr15123_retry_addr
Previously it was calling getscopes(pc) to see if we're at the start of
the outermost function, which may be a preamble for -mfentry. But the
scopes are inside-out -- [0] is the innermost -- so inlines were
comparing against their own entry pc, then inappropriately applying the
mfentry workaround. It also assumes we're already focussed on the CU,
which may not be the case for .callee probes that expand across CUs.
And besides all that, dwarf_getscopes is an expensive traversal.
Instead, we can just keep a cached set of known entry PCs, built from
our existing cache of concrete functions, and check if possible mfentry
PCs are in that set.
Ravi Bangoria [Tue, 23 Aug 2016 11:24:28 +0000 (06:24 -0500)]
ppc64le: Fix LEP usage for probing
PPC64 ELF ABI v2 has a Global Entry Point and a Local Entry Point for
the functions. Debuginfo of ELF contains GEP which is same as entrypc
while symbol table contains GEP and offset, from which we can calculate
LEP. LEP is used to call function within single CU, when TOC pointer
update is not required. Placing a probe on LEP catches call from both
the GEP and the LEP but, by default, systemtap probes on GEP.
Commit b4c6a4b1cd00 ("Prioritize symbol table lookup for ppc64le") solve
this issue by storing LEP in symbol table and prioritizing symbol table
over debuginfo for ppc64le.
But there are few regression effect of this patch. Couple of examples
are given below.
1. If target program is compiled without optimization and user is
interested in function parameter, systemtap should probe after function
prologue. But above patch forces probe on LEP and which result in garbage
value of function parameter will get recorded.
$ make verbose=1 installcheck RUNTESTFLAGS='at_var.exp -v --debug'
...
# of expected passes 1
# of unexpected failures 1
2. Probe on shared library function with parameter is failing at Pass 2.
$ make verbose=1 installcheck RUNTESTFLAGS='exelib.exp -v --debug'
...
# of expected passes 10
# of unexpected failures 64
3. When symbol_name with offset is used to register kprobe, kernel itself
will find LEP and adds offset to it. Systemtap using LEP to find offset
is resulting in offset being added two times.
GEP + lep_offset (by systemtap) + lep_offset (by kernel)
This can be solved by calculating LEP only at a time of adding a probe.
That will make effect of LEP local to that area and won't have any
regression effect.
After applying patch:
$ make verbose=1 installcheck RUNTESTFLAGS='at_var.exp -v --debug'
...
# of expected passes 2
$ make verbose=1 installcheck RUNTESTFLAGS='exelib.exp -v --debug'
...
# of expected passes 74
Fixes: Commit b4c6a4b1cd00 ("Prioritize symbol table lookup for ppc64le") Reported-by: Athira Rajeev <atrajeev@linux.vnet.ibm.com>
[ Reported about issue with shared library ] Signed-off-by: Ravi Bangoria <ravi.bangoria@linux.vnet.ibm.com>
Ravi Bangoria [Tue, 23 Aug 2016 11:24:27 +0000 (06:24 -0500)]
ppc64le: Store correct function entry address in symbol_table
PPC64 ELF ABI v2 has a Global Entry Point and a Local Entry Point for
the functions. Debuginfo of ELF contains GEP which is same as entrypc
while symbol table contains GEP and offset, from which we can calculate
LEP. LEP is used to call function within single CU, when TOC pointer
update is not required. Placing a probe on LEP catches call from both
the GEP and the LEP but, by default, systemtap probes on GEP.
For ppc64le, Systemtap stores LEP in symbol table and prioritize symbol
table over debuginfo. But, storing LEP in symbol table has couple of
regression effect. As LEP is only required at a time of adding a probe,
don't store it in symbol table.
No need to prioritize symbol table as well because debuginfo and symbol
table both will contain Global Entry Point.
Revert commit b4c6a4b1cd00 ("Prioritize symbol table lookup for ppc64le")
partially.
Signed-off-by: Ravi Bangoria <ravi.bangoria@linux.vnet.ibm.com>
Felix Lu [Tue, 23 Aug 2016 13:49:57 +0000 (09:49 -0400)]
modify interactive mode load behavior
The "load" command no longer appends to the existing script. The user
is prompted if they want to overwrite the existing file if the script is
not empty.
David Smith [Thu, 18 Aug 2016 20:30:28 +0000 (15:30 -0500)]
Fix a '--rlimit-*' option problem identified by BZ1368188.
* session.cxx (systemtap_session::parse_cmdline): Improve the '--rlimit-*'
options to report an error and return if the option value isn't
specified or doesn't convert properly. If the limit can't be set,
return if the error isn't EPERM.
* man/stap.1.in: Remove confusing language in the '--rlimit-*' option
descriptions.
* testsuite/parseko/rlimit-as01.stp: New test case.
* testsuite/parseko/rlimit-as02.stp: Ditto.
* testsuite/parseko/rlimit-cpu01.stp: Ditto.
* testsuite/parseko/rlimit-cpu02.stp: Ditto.
* testsuite/parseko/rlimit-fsize01.stp: Ditto.
* testsuite/parseko/rlimit-fsize02.stp: Ditto.
* testsuite/parseko/rlimit-nproc01.stp: Ditto.
* testsuite/parseko/rlimit-nproc02.stp: Ditto.
* testsuite/parseko/rlimit-stack01.stp: Ditto.
* testsuite/parseko/rlimit-stack02.stp: Ditto.
David Smith [Thu, 18 Aug 2016 15:37:12 +0000 (10:37 -0500)]
Update procmod_watcher.stp example for more modern kernels.
* testsuite/systemtap.examples/process/procmod_watcher.stp: Update example
to report 'clone' syscalls as 'fork' syscalls (since on some
architectures like aarch64 fork() is implemented using clone()). Also
report 'exit_group' syscalls as 'exit' syscalls.
David Smith [Wed, 17 Aug 2016 17:59:07 +0000 (12:59 -0500)]
Make [nd_]syscall.sched_[gs]etaffinity probes fully optional.
* tapset/linux/syscalls2.stp (syscall.sched_getaffinity): Make fully
optional for kernels with no 'affinity' support.
(syscall.sched_setaffinity): Ditto.
* tapset/linux/nd_syscalls2.stp (nd_syscall.sched_getaffinity): Ditto.
(nd_syscall.sched_setaffinity): Ditto.
Cody Santing [Fri, 12 Aug 2016 19:03:13 +0000 (15:03 -0400)]
PR15932: printf %m/%M user variant
* man/stap.1.in: update documentation regarding %m/%M
* runtime/vsprintf.c: m/M cases in _stp_vsprint_memory now check for '#'/STP_SPECIAL flag
* testsuite/systemtap.printf/print_user_buffer.*: test case for new variant
Felix Lu [Fri, 12 Aug 2016 14:44:58 +0000 (10:44 -0400)]
some auto_path test case
* main.cxx: Check for PATH directory in all include paths.
* testsuite/systemtap.base/auto_path.exp: New testcase.
* testsuite/systemtap.base/auto_path.c: Test program.
* testsuite/systemtap.base/tapset/PATH/*: Auto path tapsets.
Martin Cermak [Fri, 12 Aug 2016 05:21:03 +0000 (07:21 +0200)]
Fix testsuite's list-unfinished makefile target
Do not only check systemtap.log in the main testsuite directory,
but also check other locations, namely the artifacts directory
used by the installcheck-parallel target.
Cody Santing [Thu, 11 Aug 2016 19:29:29 +0000 (15:29 -0400)]
Update interactive mode
* cmdline.cxx: introduce new '--interactive' switch
* cmdline.h: introduce new '--interactive' switch
* interactive.cxx: Print a few sample commands upon entering interactive mode. Add new 'help' alias, '?'.
* session.cxx: Map new '--ineractive' switch to interactive mode. When a script is not specified, new error message suggests interactive mode.
David Smith [Thu, 11 Aug 2016 16:31:28 +0000 (11:31 -0500)]
Fix(ish) for PR20433 by avoiding a fedora kernel crash.
* runtime/vma.c (_stp_vma_mmap_cb): Use _stp_strcmp() to avoid a kernel
crash.
* runtime/stp_string.c (_stp_strcmp): New function, a variant of strcmp()
that treats NULL pointers as empty strings.
* runtime/stp_string.h: Add _stp_strcmp() declaration.
Felix Lu [Tue, 9 Aug 2016 19:36:14 +0000 (15:36 -0400)]
filename based auto path tapset support
Sometimes the process name may change with new versions and the
tapset must be modified. Process probe points in the PATH directory
may now have their arguments replaced by the full file path starting
from /PATH.
* parse.cxx: Pass alias_seen flag to parse_probe_points(). Add filepath
argument to process component.
Felix Lu [Mon, 8 Aug 2016 18:48:37 +0000 (14:48 -0400)]
some function overloading optimizations
This removes some unnecessary code in the generated module.
* elaborate.cxx: Store next flag in functiondecl instead of in a set.
* staptree.h: struct functiondecl:: new has_next flag.
* translate.cxx: c_unparser::visit_functioncall - Don't generate
redundant functioncall code for overloaded functions.
David Smith [Thu, 4 Aug 2016 20:39:19 +0000 (15:39 -0500)]
Fix a couple of conversions.stp tapset problems and improve its testing.
* tapset/linux/conversions.stp (kernel_buffer_quoted): Properly print the
address as an unsigned long (instead of an unsigned int).
(kernel_buffer_quoted_error): Add the faulting address in the error
message.
* testsuite/buildok/conversions-embedded.stp: Add compile tests for
several missing functions.
Felix Lu [Thu, 28 Jul 2016 14:39:58 +0000 (10:39 -0400)]
PR10485: auto-path tapset support
This prefixes the process probes placed in the tapset/PATH/ directory
with the location of the tapset.
* main.cxx: New collect_{stp,stpm} functions for passing to nftw(). Parse all
.stp files placed in include_path[0]/PATH/. nftw() is now used instead of
globbing for tapset searching to allow recursive traversal.
* parse.h: New pf_auto_path flag.
* parse.cxx: Prefix process argument with location of tapset.
PR20423: improve 'struct {...} being accessed instead of member' message
Esp. for anonymous structures, this message was not very helpful.
The new message is much more complete:
semantic error: 'struct {...}' (include/linux/types.h:175) is being accessed instead of a member such as '->counter': operator '->' at <input>:1:81
source: probe kernel.function("dput") { println( @cast($dentry->d_inode, "struct inode")->i_count) }
^
Igor Zhbanov [Fri, 29 Jul 2016 17:51:21 +0000 (13:51 -0400)]
systemtap.spec: add systemd as BuildRequires & Requires if %{with_systemd}
It should have both "Requires: systemd" and "BuildRequires: systemd".
BuildRequires because it uses %{_unitdir} and %{_tmpfilesdir}, and
Requires because of systemctl in %post.
The dwarf_var_expanding_visitor::visit_entry_op function inherited
some old logic from expanding $var's in .function().return probes.
This logic is not approproate for the more formal way by which current
code synthesizes a .call probe to collect @entry(EXPR) values. Now we
pass the EXPR tree to the .call probe unmodified, for resolution
there, rather than in a fake non-.return context here. This makes
@entry(@perf("...")) expressions work.
David Smith [Tue, 26 Jul 2016 17:57:59 +0000 (12:57 -0500)]
Fix a locking problem in the task_finder.
* runtime/linux/task_finder.c (stap_start_task_finder): Be sure to ulock
the task if we exit early.
* runtime/linux/task_finder2.c (stap_start_task_finder): Ditto.
git://sourceware.org / systemtap.git / log