Frank Ch. Eigler [Fri, 13 Oct 2017 23:00:37 +0000 (19:00 -0400)]
bpf translator: catch runtime_errors
Some simple test cases can run up against bpf-translate limitations.
Those that are signalled with a runtime_error exception need to be
caught at least in translate_bpf_pass() and a message improvised, so
that there is something more to go on than an anonymous pass-4
failure.
David Smith [Fri, 13 Oct 2017 20:52:44 +0000 (15:52 -0500)]
Make small improvements to the systemtap server testsuite.
* testsuite/systemtap.server/server.exp: When restarting the avahi daemon,
try to fix a restart failure.
* testsuite/systemtap.server/client.exp (list_servers): Add debug output.
* testsuite/lib/systemtap.exp (shutdown_server): Try to ensure that the
systemtap server is really gone before returning.
David Smith [Wed, 11 Oct 2017 15:53:19 +0000 (10:53 -0500)]
Update the server_locale.exp test case to avoid spurious failures.
* testsuite/systemtap.server/server_locale.exp: Handle the case where the
French langpack isn't installed on the system. Do a French sniff test
before trying to use the French locale.
* client-nss.h (client_backend): Removed 'server_tmpdir' declaration that
shadowed the declaration in the base class. This caused the code in
csclient.cxx to get the wrong value for 'server_tmpdir'.
Aaron Merey [Fri, 6 Oct 2017 23:34:24 +0000 (19:34 -0400)]
stapbpf: fix global variable initialization bug.
* bpf-translate.cxx: The translator now adds assignment statements to
the beginning of the "stap_begin" bpf program in order to initialize
global variables with any user-supplied default values. Currently there
is no support for string literals, the translator will still ignore them.
(struct init_block): manages the resources associated with these
assignments.
* tapsets.cxx(sort_for_bpf), tapset-been.cxx (sort_for_bpf): immediately
returns if the probe group pointer is NULL.
* bpf_tests: added new tests and fixed a syntax error in globals1.stp and
globals2.stp.
David Smith [Fri, 6 Oct 2017 15:32:53 +0000 (10:32 -0500)]
Add a PR20516 workaround by using kallsyms_lookup_name to find udelay_simple.
* runtime/linux/common_session_state.h (stp_session_init): Use
kallsyms_lookup_name() to find udelay_simple() if it exists but isn't
exported.
* runtime/linux/runtime.h: Ditto.
* buildrun.cxx (compile_pass): Add an new autoconf test for
udelay_simple().
* runtime/linux/autoconf-udelay_simple.c: New file.
Aaron Merey [Thu, 5 Oct 2017 16:35:44 +0000 (12:35 -0400)]
stapbpf: fix temporary registers appearing in translator output and null ptr deref.
* bpf-opt.cxx (reg_alloc): When replacing temp registers with hard registers in
bpf instructions, allow for a maximum of 10 dereferences of vector "partition",
up from 2.
(post_alloc_cleanup): Avoid removing redundant instructions if they
are their block's only instruction since removing them will cause a null pointer
deref in bpf-translate.cxx (output_probe).
David Smith [Thu, 5 Oct 2017 17:31:26 +0000 (12:31 -0500)]
Add the server version number to the http protocol.
* httpd/api.cxx (result_info::generate_response): Send the server version
number to the client.
(build_info::generate_response): Ditto.
* client-http.cxx (unpack_response): Get the server version number.
Sandipan Das [Thu, 5 Oct 2017 09:09:41 +0000 (14:39 +0530)]
powerpc: Change SDT argument constraint
With the 'o' memory constraint, any memory operand which
has an offsettable address is allowed. However, for some
architectures such as powerpc, this allows operands like
the ones shown below in the readelf output from Fedora 26
to be generated.
The second and third argument shown above are both having
operands which are pointers to static data anchors. Since
these static anchors are not included in the symbol table,
they cannot be resolved from the binary itself. So, such
arguments cannot be read via their corresponding markers.
Using the 'Z' memory constraint instead solves this issue
as it will only allow a memory operand that is an indexed
or indirect from a register.
So, for powerpc, we set STAP_SDT_ARG_CONSTRAINT to 'nZr'
but keep it as 'nor' for all other architectures.
Signed-off-by: Sandipan Das <sandipan@linux.vnet.ibm.com>
David Smith [Wed, 4 Oct 2017 20:48:19 +0000 (15:48 -0500)]
Add more http client updates and rearrange some code.
* client-nss.cxx: Moved MOK_PUBLIC_CERT_NAME to cscommon.h.
(nss_client_backend:nss_client_backend): Moved here from client-nss.h.
(process_response): Moved to csclient.cxx.
* client-nss.h (nss_client_backend): Moved class constructor to
client-nss.cxx.
* client-http.h (http_client_backend): Moved class constructor to
client-http.cxx.
* client-http.cxx (http_client_backend::http_client_backend): Moved here
from header file and initialize 'server_tmpdir'.
(http_client_backend::unpack_response): Move processing from
process_response() to here and delete process_response().
* csclient.cxx (process_response): Moved from client-nss.cxx.
* csclient.h: Move 'process_response' from the backend to the main
code. Add 'server_tmpdir' and 'server_version' to the backend.
* cscommon.h: Move MOK_PUBLIC_CERT_NAME and MOK_PUBLIC_CERT_FILE
definitions here.
* stap-serverd.cxx: Remove MOK_PUBLIC_CERT_NAME and MOK_PUBLIC_CERT_FILE
definitions.
* util.cxx: Add another write_to_file() template instantiation.
David Smith [Mon, 2 Oct 2017 21:29:08 +0000 (16:29 -0500)]
Add more http client updates.
* client-http.cxx (http_client::download): Turn off following HTML
redirects, so we can handle things ourselves.
(http_client::get_response_code): New function.
(http_client_backend::find_and_connect_to_server): Don't try to look up
details for the "kernel" module. Reverse return value logic - only
return success if everything worked.
(http_client_backend::process_response): Rework logic to keep looping
until we get a 303 response code. The next GET should be our result
info. Make the "files" array optional, since not all stap requests
return a module.
David Smith [Wed, 27 Sep 2017 21:43:33 +0000 (16:43 -0500)]
Fix PR22158 by fixing the shutdown sequence of events.
* runtime/stp_utrace.c (utrace_exit): Free everything.
(utrace_cleanup): No longer cancel pending task_work requests.
(utrace_cancel_all_task_work): New function.
(untrace_shutdown): Cancel all pending task_work requests, wait on all
running items to finish, then free everything.
* runtime/linux/task_finder2.c (__stp_tf_cancel_all_task_work): Renamed
from __stp_tf_cancel_task_work.
(__stp_tf_free_all_task_work) New function.
(stap_stop_task_finder): Just cancel all pending task_work requests
before waiting for them to finish. After they have all finished, free
everything.
David Smith [Thu, 21 Sep 2017 20:41:21 +0000 (15:41 -0500)]
Add some http client updates.
* client-http.cxx (http_client::post): Reindent a bit and add FIXME
comment.
(http_client::add_script_file): Don't assume a placeholder for the
script file.
(http_client_backend::process_response): Instead of calling
json_object_to_json_string(), call json_object_get_string(). The former
returns a quoted string (and turns '/' into '\/'). The latter returns a
"raw" string, which is what we want.
* runtime/linux/task_finder.c (__stp_call_mmap_callbacks_for_task):
Instead of a 'struct path' pointer, store the 'struct path' structure
itself in vma_cache_t.
* runtime/linux/task_finder2.c: Ditto.
* bpf-translate.cxx:
All bpf programs produced by the translator now include a bpf map
containing internal globals intended for use by the runtime. Currently
this map contains a single element which indicates whether exit()
has been called.
Add support for BPF_LD_MAP pseudo-instruction for tapsets with
embedded bytecode.
Add tags to printf() format strings to facilitate parsing from trace
buffer at runtime.
* bpf-internal.h (struct globals): Add ABI for internal globals.
* configure: Add "-pthread" to stapbpf_LIBS
* main.cxx (passes_0_4): Include "/bpf" as a possible runtime_prefix
* parse.cxx (eval_pp_conditional): Include "bpf" as option for
runtime-based preprocessor conditionals.
* stapbpf/bpfinterp.cxx: Add function remove_tags().
* stapbpf/libbpf.c: Fix bug where some bpf_attrs weren't memset to 0
upon initialization.
David Smith [Tue, 19 Sep 2017 15:51:54 +0000 (10:51 -0500)]
Fix PR22158 by updating the runtime to handle the removal of spin_unlock_wait.
* runtime/stp_helper_lock.h: Removed stp_spin_unlock_wait(), since the
underlying kernel functionality was removed in kernel commit d3a024abbc43827785.
* runtime/stp_utrace.c (utrace_finish_stop): Instead of calling
stp_spin_unlock_wait(), just try to lock and unlock the spin lock.
William Cohen [Fri, 15 Sep 2017 15:37:36 +0000 (11:37 -0400)]
Avoid pulling in old compatibility RPM 4.4 API
RPM 4.14.0 rc1 removes a number of RPM 4.4 compatibility APIs enabled by
_RPM_4_4_COMPAT define. SystemTap has been updated to the newer API
and keeping this define causes the build to fail on Fedora rawhide.
David Smith [Thu, 14 Sep 2017 21:42:55 +0000 (16:42 -0500)]
Update systemtap.stress/conversions.exp.
* testsuite/systemtap.stress/conversions.exp (run_conv_test): Turn off
overload protection so that we get the right errors ("copy fault") and
not the wrong errors ("probe took too long").
The bpf infrastructure has a complex graph data types, which are
created, mutated, partly deleted, added etc., representing a
worst-case scenario for a statically typed non-GC system like c++.
The following little test case triggers memory corruption
(access-after-free).
./stap --runtime=bpf -p4 -e 'probe kernel.function("sys_open")
{x = 1; if (x) x = 0; else x = 2 } '
This patch disables some of the responsible code - by turning off
program::program destructors' detailed cleanup of the embedded graph
structure. So we leak memory instead. :-(
Another little hunk deals with an un-C++-ish typecasting error that
valgrind also caught (operator new[] vs. free() on same pointer).
David Smith [Tue, 12 Sep 2017 21:15:47 +0000 (16:15 -0500)]
Improve fix for PR22117 by updating the translator.
* dwflpp.cxx (dwflpp::translate_base_ref): Add an even better fix to the
number of bytes comparison to avoid "noncontiguous location for base
fetch" semantic error on 32-bit systems (and not break 64-bit systems).
David Smith [Tue, 12 Sep 2017 18:51:07 +0000 (13:51 -0500)]
Fix PR22124 by properly handling perf counter reads in uretprobes.
* tapsets.cxx (uprobe_derived_probe_group::emit_module_utrace_decls): Be
sure to properly handle perf counter reads in uretprobes.
* testsuite/systemtap.base/perf.exp: If this system doesn't support
hardware perf probes, try with software perf probes.
* testsuite/systemtap.base/perf03sw.stp: New file.
* testsuite/systemtap.base/perf04sw.stp: Ditto.
David Smith [Fri, 8 Sep 2017 19:08:18 +0000 (14:08 -0500)]
Fix PR22110 by updating the translator.
* loc2stap.cxx (location_context::new_symref): Set sym->referent to avoid
"internal error: unresolved symbol 'pointer'" when processing
semok/autocast07.stp.
David Smith [Fri, 8 Sep 2017 19:05:37 +0000 (14:05 -0500)]
Fix PR22117 by updating the translator.
* dwflpp.cxx (dwflpp::translate_base_ref): Fix number of bytes comparison
to avoid "noncontiguous location for base fetch" semantic error on
32-bit systems.
stapregex PR15065 (8/8) :: add back re2c's mini-optimizer
* stapregex-tree.cxx (ins_optimize): New function.
(regexp::compile): Initialize i.marked in newly allocated ins to
stop a harmless but very noisy Valgrind complaint.
* stapregex-tree.h (ins_optimize): New function.
* stapregex-dfa.cxx (stapregex_compile): Invoke ins_optimize.
This is the only part of the TNFA functionality that touches
the kernel, so it is crucially important to review for safety.
* stapregex.cxx (stapdfa::emit_declaration): Set up and clean
up tag state, define additional YY* macros used by TNFA actions.
* stapregex-dfa.cxx (span::emit_jump): Add diagnostic printing,
continue matching even after entering an accepting state.
(span::emit_final): Emit finalizer TNFA action, save final
tags if the finalizer produced a suitable longest-match.
(state::emit): Add diagnostic printing, handle unknown chars.
(dfa::emit): Set up tags, emit initializer TNFA action, emit
finalizer when the regex accepts at the first char.
(dfa::emit_action): New function.
(dfa::emit_tagsave): Vestigial functionality -- just sets ntags.
* stapregex-dfa.h (dfa::emit_action): New function.
* stapregex-dfa.cxx: Implement Laurikari's TNFA algorithm.
(arc_priority::refine_higher, ::refine_lower): Detect overflow.
(arc_compare): Previous implementation was incorrect, fix it.
(dfa::add_map_item): New function.
(te_closure): Properly consider arc_priorities.
(same_ins): New function.
(dfa::find_equivalent): Algorithm to compute reordering actions.
(dfa::compute_action): Algorithm to compute TNFA actions.
(dfa::compute_finalizer): Algorithm to compute finalizers.
(dfa::dfa): Consider arc_priorities and reordering actions.
(operator << for map_item): New function.
(operator << for tdfa_action): Use << for map_item.
(kernel_point::print): New function.
(state::print): Include diagnostic info for TNFA algorithm.
(operator << for state): New function.
(operator << for dfa): New function.
* stapregex-dfa.h (operator << for map_item): New function.
(struct tdfa_insn): Add save_tag option for finalizers.
(kernel_point::print): New function.
(struct span): Add jump_pairs field.
(struct state): Add dfa field to access containing DFA. Also
add accept_kp field to track TNFA actions for accepting states.
(state::state): Add dfa field to constructor.
(state::print): Use ordinary ostream, not translator_output.
(operator << for state): New function.
(struct dfa): Add success_outcome and fail_outcome field to
explicitly track the indices of 'accept' and 'fail' outcomes.
(dfa::dfa): Add an option to set index of 'accept' outcome.
(dfa::add_map_item): New function.
(dfa::compute_action): New function.
(dfa::compute_finalizer): New function.
* stapregex.cxx (regex_to_stapdfa): Add comment -- here is how
you can force TNFA tag operations to be generated even when the
stap script doesn't request any groups.
stapregex PR15065 (4/8) :: switch DFA engine to an unsigned char type
* stapregex-defines.h: New file. The rchar type is an unsigned
char. Within an NFA or DFA, rchar 128 is used to represent
unknown characters (i.e., anything that does not fit within
the ASCII standard encoding). This is placeholder functionality
meant to sanely handle other encodings such as UTF prior to
implementing full support for them.
* stapregex-dfa.h, stapregex-dfa.cxx: Use rchar throughout.
* stapregex-parse.h, stapregex-parse.cxx: Use rchar throughout.
* stapregex-tree.h, stapregex-tree.cxx: Use rchar throughout.
* stapregex.h: Use rchar throughout.
* session.h (systemtap_session):
Replace dfa_maxstate with dfa_maxmap.
* session.cxx (systemtap_session::systemtap_session):
Replace dfa_maxstate with dfa_maxmap.
* translate.cxx (systemtap_session):
Replace dfa_maxstate with dfa_maxmap.
* stapregex-dfa.h (struct dfa): New variable nmapitems.
* stapregex.h (stapdfa::num_map_items): New function.
* stapregex.cxx (regex_to_stapdfa): Set value of dfa_maxmap.
(stapdfa::num_map_items): New function.
* regex.stp: New tapset providing access to TNFA grouping data.
(matched_str): Return last matched str.
(matched): Return n'th subexpression from last match.
(ngroups): Return the number of subexpressions in last match.
David Smith [Tue, 5 Sep 2017 18:30:54 +0000 (13:30 -0500)]
Fix PR22054 by making sure we know whether a pointer is an lvalue or not.
* dwflpp.cxx: Pass around 'lvalue' in several places where needed.
(dwflpp::translate_pointer): Instead of assuming a pointer isn't an
lvalue, use the new parameter.
* dwflpp.h: Update prototypes.
* tapsets.cxx: Pass around 'lvalue' in several places where needed.
David Smith [Thu, 31 Aug 2017 14:38:18 +0000 (09:38 -0500)]
Add a minor update to the strace example and meta file.
* testsuite/systemtap.examples/process/strace.stp (filter_p): Remove
comment about PR6762, which has been fixed.
* testsuite/systemtap.examples/process/strace.meta: Run the script on a
specific process, instead of on the entire system (where we would hit
"probe overhead exceeded threshold").
David Smith [Thu, 31 Aug 2017 14:35:41 +0000 (09:35 -0500)]
Update tapset/python2.stp for RHEL6-era systems.
* tapset/python2.stp (Py2Long_AsLongLongAndOverflow_Small): Don't bother
trying to see if the Py2LongObject is defined, go ahead and use it. The
check to see if it is defined seems to fail.
(Py2Long_AsLongLongAndOverflow_Big): Ditto.
David Smith [Tue, 29 Aug 2017 21:27:23 +0000 (16:27 -0500)]
Fix PR22036 by handling "struct bio" kernel changes.
* buildrun.cxx (compile_pass): Add "autoconf"-style test for bi_opf field
of the "bio" structure.
* runtime/linux/autoconf-bio-bi_opf.c: New file.
* tapset/linux/ioblock.stp (bio_op): Use new STAPCONF_BIO_BI_OPF define.
David Smith [Tue, 29 Aug 2017 16:03:06 +0000 (11:03 -0500)]
Fix PR22012 by updating the way we read user strings.
* runtime/linux/loc2c-runtime.h (_stp_deref_string_nofault): Renamed from
kderef_string_() and modified to be able to read either a kernel or user
string.
(kderef_string): Modified to call the new _stp_deref_string_nofault().
* runtime/linux/copy.c (_stp_strncpy_from_user): Replace all code with a
call to _stp_deref_string_nofault(). This avoids a "scheduling while
atomic" kernel BUG seen when reading strings.
David Smith [Thu, 24 Aug 2017 21:42:51 +0000 (16:42 -0500)]
Tweak more systemtap.examples meta files to improve test passes.
* testsuite/systemtap.examples/general/key.meta: Check that the probed
kernel function exists before trying to compile/run the example.
* testsuite/systemtap.examples/general/keyhack.meta: Ditto.
David Smith [Thu, 24 Aug 2017 16:00:59 +0000 (11:00 -0500)]
Perform process cleanup in testsuite/systemtap.interactive/*.exp.
* testsuite/systemtap.interactive/char_and_line.exp: Be sure to kill
systemtap at the end of the test and wait on it to finish.
* testsuite/systemtap.interactive/input.char.exp: Ditto.
* testsuite/systemtap.interactive/input.line.exp: Ditto.
David Smith [Thu, 24 Aug 2017 15:42:58 +0000 (10:42 -0500)]
Add a fix to the 'container_check' example.
* testsuite/systemtap.examples/profiling/container_check.stp: Handle the
fact that the kernel functions ns_capable() and capable() have different
function signatures. On RHEL6 systems, we were getting bogus output.
David Smith [Thu, 24 Aug 2017 15:34:22 +0000 (10:34 -0500)]
Tweak some systemtap.examples meta files to improve test passes.
* testsuite/systemtap.examples/memory/hugepage_collapse.meta: Check that
the probed kernel function exists before trying to compile/run the
example.
* testsuite/systemtap.examples/memory/hugepage_split.meta: Ditto.
* testsuite/systemtap.examples/general/func_time_stats.meta: Probe
syscall.read instead of syscall.nanosleep, since we're much more likely
to hit that syscall.
Aaron Merey [Wed, 23 Aug 2017 22:54:49 +0000 (18:54 -0400)]
PR21891: perform _stext relocation on kprobes, fix kretprobe registration.
* stapbpf/stapbpf.cxx (maybe_collect_kprobe): read value of _stext from
/proc/kallsyms and add to kprobe symbol offsets.
* tapsets.cxx (dwarf_derived_probe::dwarf_derived_probe): set symbol_name
to funcname for kretprobes.
David Smith [Wed, 23 Aug 2017 21:00:24 +0000 (16:00 -0500)]
Fix PR21998 by updating tapset/linux/ip.stp.
* tapset/linux/ip.stp: Use the new 'iphdr_cast' macro to properly cast a
value to a iphdr struture pointer. For RHEL6, this fixes some "kbuild
exited with status: 2" warnings.
David Smith [Wed, 23 Aug 2017 19:05:21 +0000 (14:05 -0500)]
Add small fixes in tapset/linux/memory.stp.
* tapset/linux/memory.stp: Handle the case where __GFP_MEMALLOC doesn't
exist (like on RHEL6). In __vm.kmalloc_node.kp, add @entry() around
variables in a return probe.
David Smith [Wed, 23 Aug 2017 16:34:56 +0000 (11:34 -0500)]
Fix PR21996 by updating DEREF_FAULT for dyninst.
* runtime/loc2c-runtime.h (DEREF_FAULT): Fix dyninst compile errors on
32-bit systems where 'intptr_t' is an 'int'. Be sure to cast to 'long'
in DEREF_FAULT and STORE_DEREF_FAULT before printing the value with
"%lx".
David Smith [Tue, 22 Aug 2017 19:44:03 +0000 (14:44 -0500)]
Add and use a new macro @inet_sock_cast().
* tapset/linux/inet.stpm: Added new macro file.
* tapset/linux/inet_sock.stp: Use new @inet_sock_cast() macro to
standardize casting a pointer to a "struct inet_sock".
* tapset/linux/ip.stp: Ditto.
* tapset/linux/tcp.stp: Ditto.
David Smith [Tue, 22 Aug 2017 15:15:29 +0000 (10:15 -0500)]
PR21984: Fix "cast to pointer from integer of different size" errors.
* runtime/linux/loc2c-runtime.h: Modify the __Xread() and __Xwrite()
macros for 32-bit systems so that 64-bit values are handled properly.
* tapset/linux/atomic.stp: Make sure all pointer values are cast
properly. Their values must be cast to a 'uintptr_t' value then the
final pointer type value. This is necessary for 32-bit systems (since
systemtap variables are 64-bits). Some pointer casts weren't cast to a
numeric value, some were cast to a signed value, etc. Standardize all of
them.
* tapset/linux/aux_syscalls.stp: Ditto.
* tapset/linux/conversions-guru.stp: Ditto.
* tapset/linux/conversions.stp: Ditto.
* tapset/linux/i386/aux_syscalls.stp: Ditto.
* tapset/linux/ioblock.stp: Ditto.
* tapset/linux/ioscheduler.stp: Ditto.
* tapset/linux/ip.stp: Ditto.
* tapset/linux/ipmib.stp: Ditto.
* tapset/linux/kprocess.stp: Ditto.
* tapset/linux/netfilter.stp: Ditto.
* tapset/linux/nfs.stp: Ditto.
* tapset/linux/nfsd.stp: Ditto.
* tapset/linux/proc_mem.stp: Ditto.
* tapset/linux/rcu.stp: Ditto.
* tapset/linux/s390/aux_syscalls.stp: Ditto.
* tapset/linux/signal.stp: Ditto.
* tapset/linux/socket.stp: Ditto.
* tapset/linux/task.stp: Ditto.
* tapset/linux/tcp.stp: Ditto.
* tapset/linux/tcpmib.stp: Ditto.
* tapset/linux/x86_64/aux_syscalls.stp: Ditto.
* tapset/string.stp: Ditto.
* tapset/uconversions-guru.stp: Ditto.
* tapset/uconversions.stp: Ditto.