Mark Wielaard [Thu, 22 Feb 2018 09:54:36 +0000 (10:54 +0100)]
dwflpp::function_entrypc if entry or low pc isn't given use base or start.
On Fedora rawhide a kernel compiled with GCC8 sometimes has subprograms
without an entry or low pc attribute, but with a ranges attribute. In that
case assume the base address or first start address of the range is the
entry pc to set a probe on.
David Smith [Wed, 21 Feb 2018 22:09:18 +0000 (16:09 -0600)]
Make a few small changes to handle Fedora's switch to python 3 as default.
* configure.ac: Prefer 'python2' over 'python' for the python2
executable.
* configure: Regenerated.
* httpd/backends.cxx (generate_module): Be sure which version of python
we're calling.
* httpd/docker/stap_build_docker_image.py: Remove python shebang line
since httpd/backends.cxx knows the right version of python to call.
* httpd/docker/fedora_install_package.py: Add comment about why using
'/usr/bin/python' is OK.
David Smith [Wed, 14 Feb 2018 17:11:55 +0000 (11:11 -0600)]
Added strncpy()/strncat() code correctness fixes.
* doc/SystemTap_Tapset_Reference/docproc.c: gcc version 8 (on rawhide)
rightly complains that the following code is wrong:
char buf[PATH_MAX];
strncpy(buf, foo, PATH_MAX);
The code should be:
strncpy(buf, foo, PATH_MAX - 1);
Note that a similar problem occurs with strncat(). Fix these errors by
either subtracing one from the buffer length or switching to strlcpy()
instead of strncpy().
* runtime/task_finder_vma.c: Ditto.
* staplog.c: Ditto.
* staprun/ctl.c: Ditto.
* staprun/staprun.c: Ditto.
* stapvirt.c: Ditto.
* tapset/linux/ip.stp: Ditto.
* tapset/tokenize.stp: Ditto.
* tapset/uconversions.stp: Ditto.
* testsuite/buildok/cmdline01.stp: Ditto.
* testsuite/buildok/two.stp: Ditto.
* testsuite/systemtap.syscall/bind.c: Ditto.
* testsuite/systemtap.syscall/nfsservctl.c: Ditto.
David Smith [Tue, 13 Feb 2018 19:24:19 +0000 (13:24 -0600)]
Move the code getting a certificate from a server into each backend.
* nss-server-info.cxx (add_server_trust): Instead of calling the NSS
client-specific client_connect() function, call
backend->trust_server_info(). This allows different clients to implement
getting the server certificate differently.
* csclient.h (client_backend): Add client_backend::trust_server_info()
virtual function declaration.
* client-nss.cxx (client_connect): Make static.
(nss_client_backend::trust_server_info): New function that just calls
client_connect().
* client-nss.h: Add nss_client_backend::trust_server_info() declaration
and remove client_connect() declaration.
* client-http.h: Add http_client_backend::client_backend() declaration.
* client-http.cxx (trust_server_info): New stub function.
Aaron Merey [Thu, 8 Feb 2018 19:23:56 +0000 (14:23 -0500)]
bpf translator: improve interference check during reg alloc
* bpf-opt.cxx (reg_alloc): when merging temporary registers a and b,
add checks for interference between temporaries that have already
merged with either a or b.
David Smith [Thu, 8 Feb 2018 18:45:13 +0000 (12:45 -0600)]
Add the capability to list http servers.
* client-http.cxx (fill_in_server_info): New function.
* client-http.h (client_backend): Added 'fill_in_server_info' prototype.
* client-nss.h (client_backend): Added 'fill_in_server_info' stub.
* csclient.cxx (nss_get_client_backend): New function.
(compile_server_client::passes_0_4): Calls new 'nss_get_client_backend'
function.
* csclient.h: Added client_backend::fill_in_server_info() virtual
function and 'nss_get_client_backend' prototype.
* httpd/Makefile.am (stap_httpd_SOURCES): Now includes ../cscommon.cxx.
* httpd/Makefile.in: Regenerated.
* httpd/nss_funcs.cxx (nss_get_server_cert_info): New function.
* httpd/nss_funcs.h: Added 'nss_get_server_cert_info' prototype.
* httpd/server.cxx (base_dir_rh::GET): When a GET is done on the server
root directory, the server architecture and certificate info is
returned.
* nss-server-info.cxx (compile_server_info::host_specification): New
function.
(query_server_status): Add support for http servers.
(nss_get_specified_server_info): Ditto.
(nss_get_or_keep_compatible_server_info): A server can be compatible if
the architectures match.
(nss_add_online_server_info): New function.
* nss-server-info.h: Added compile_server_info::host_specification() and
nss_add_online_server_info() prototypes.
David Smith [Fri, 2 Feb 2018 16:26:42 +0000 (10:26 -0600)]
Move NSS server info to its own file for reuse by the http server code.
* client-nss.cxx: Moved server info code to nss-server-info.cxx.
* client-nss.h: Ditto.
* nss-server-info.cxx: New file.
* nss-server-info.h: New file.
* Makefile.am (stap_SOURCES): Added new nss-server-info.cxx file.
* Makefile.in: Regenerated.
Stefan Hajnoczi [Thu, 1 Feb 2018 16:24:04 +0000 (11:24 -0500)]
dtrace: accept "char const" via pyparsing
The widely used "const char" declaration specifier can also be written
as "char const". While this form is rarely used, currently produces the
following warning from dtrace(1):
Warning: Proceeding as if --no-pyparsing was given.
This patch extends the pyparsing grammar to accept "char const" as
defined by ISO/IEC 9899:1999 ("C99"). Now "const char", "char const",
and even "const char const" are accepted without a noisy warning.
Note that "const char const" is also valid C and described by the
standard:
6.7.3 Type qualifiers
4 If the same qualifier appears more than once in the same
specifier-qualifier-list, either directly or via one or more
typedefs, the behavior is the same as if it appeared only once.
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
David Smith [Fri, 26 Jan 2018 15:24:51 +0000 (09:24 -0600)]
BZ1527809 - Fix detaching from modules using SIGQUIT.
* staprun/mainloop.c: Put a "/* NOTREACHED */" comment after all calls to
cleanup_and_exit() to remind the reader that cleanup_and_exit() doesn't
return.
(stp_main_loop): If we've got a pending interrupt,
but 'load_only' is set, just detach instead of sending STP_EXIT to
the module. Otherwise using SIGQUIT to detach fails and unloads the
module.
* staprun/monitor.c: Put a "/* NOTREACHED */" comment after all calls to
cleanup_and_exit() to remind the reader that cleanup_and_exit() doesn't
return.
* testsuite/systemtap.base/attach_detach.exp: New test.
Aaron Merey [Fri, 26 Jan 2018 00:48:35 +0000 (19:48 -0500)]
stapbpf: Add tracepoint probe support.
* bpf-translate.cxx (bpf_unparser::emit_context_var): Similar to
bpf_unparser::visit_target_deref, but the size and offset info needed
to read from the context is present in this function's argument instead
of being computed during probe execution.
* dwflpp.h (iterate_over_globals): access modifier changed from private
to public. Needed for reading debuginfo from tracequery modules.
* staptree.h (struct bpf_context_vardecl): specialized vardecl
that includes size, offset and is_signed members needed for
bpf_unparser::emit_context_var.
* loc2stap.cxx (location_context::adapt_pointer_to_bpf): replaces the
location_context's pointer with a bpf_context_vardecl *.
* stapbpf.cxx: Add support for reading tracepoint probes from the
BPF-ELF file and registering/unregistering tracepoint probes.
* tapsets.cxx: Adapt the tracepoint probe derivation process
to --runtime=bpf.
(get_tracequery_modules): Instantiate structs in the tracequery
modules whose members are the tracepoint's $context variables.
(tracepoint_query_type, handle_query_type): Iterate over the
module's debuginfo looking for the Dwarf_Dies of the tracepoint
arg structs.
(build_args_for_bpf): Create tracepoint_arg's from the struct dies
(is_signed_type, get_byte_size): helpers for build_args_for_bpf.
David Smith [Tue, 23 Jan 2018 22:38:05 +0000 (16:38 -0600)]
Added a logfile option to the web server.
* httpd/main.cxx (parse_cmdline): Added a '--log' option to specify a
logfile.
(signal_thread): Replaced all logging via clog/cerr with a call to
server_error().
(setup_main_signals): Ditto.
(main): Stop the logging when finished.
* httpd/utils.cxx (server_error): New function.
(execute_and_capture): Replaced all logging via clog/cerr with a call to
server_error().
* httpd/utils.h: Added server_error() prototype.
* httpd/api.cxx: Replaced all logging via clog/cerr with a call to
server_error().
* httpd/backends.cxx: Ditto.
* httpd/nss_funcs.cxx: Ditto.
* httpd/server.cxx: Ditto.
David Smith [Mon, 22 Jan 2018 17:38:12 +0000 (11:38 -0600)]
Make the http server send the module signature back to the client.
* httpd/api.cxx (result_info): Switch from expecting one file to return to
handling multiple files.
(result_info::add_file): New function.
(result_info::generate_response): Add all output files to the "files"
section.
(result_info::generate_file_response): Handle the list of output files.
(build_info::module_build): If the module signing worked, add the
signature file to the results.
David Smith [Fri, 19 Jan 2018 14:24:44 +0000 (08:24 -0600)]
Update the systemtap.server/client.exp test to use a mount namespace.
* testsuite/systemtap.server/client.exp: Use a custom mount namespace to
test signed modules.
* testsuite/systemtap.server/server_privilege.exp: Remove unneeded call to
'systemtap_check_users'.
David Smith [Thu, 18 Jan 2018 20:12:22 +0000 (14:12 -0600)]
Add a new server test that runs modules.
* testsuite/systemtap.server/server_privilege.exp: New test that tests
modules built by the compile server.
* testsuite/lib/server_ns.exp: New file.
* testsuite/lib/systemtap.exp (systemtap_check_users): New function.
(setup_systemtap_environment): Call systemtap_check_users.
(shutdown_server): Call server_ns_cleanup.
* testsuite/config/unix.exp: Add server_ns.exp.
* testsuite/Makefile.am: Pass the SYSCONFDIR down to the tests.
* testsuite/Makefile.in: Regenerated.
* systemtap.spec: Create the 'stapusr', 'stapsys', and 'stapdev' users for
the testsuite.
David Smith [Mon, 15 Jan 2018 17:08:39 +0000 (11:08 -0600)]
Enable python3 support in RHEL > 7.
* systemtap.spec: Expect python3 to exist in RHEL > 7 (according to a
message on fedora-devel entitled "Python3 will be in next major RHEL
release, please adjust %if statements accordingly" on Jan. 11, 2018).
David Smith [Mon, 15 Jan 2018 16:23:45 +0000 (10:23 -0600)]
Fix BZ1525651 by making the task_finder shutdown more robust.
* runtime/linux/task_finder.c (__stap_utrace_detach): New function, based
on stap_utrace_detach(), but lets the caller know if any the task had
the utrace engine attached to it.
(stap_utrace_detach): Just call __stap_utrace_detach().
(stap_utrace_detach_ops): Perform the detaching in a loop, so if the
process list snapshot we operate on becomes out of date, we'll still
detach everything.
(__stp_utrace_task_finder_report_clone): Move the
__stp_tf_handler_start() call to the very start of the function.
(__stp_utrace_task_finder_report_exec): Ditto.
(__stp_utrace_task_finder_target_exec): Ditto.
(__stp_utrace_task_finder_target_exit): Ditto.
(__stp_utrace_task_finder_target_quiesce): Ditto.
(__stp_utrace_task_finder_target_syscall_entry): Ditto.
(__stp_utrace_task_finder_target_syscall_exit): Ditto.
David Smith [Mon, 8 Jan 2018 17:28:25 +0000 (11:28 -0600)]
Add the beginnings of module signing to the web server.
* httpd/main.cxx (parse_cmdline): Add the new '--ssl' argument.
(main): Initialize NSS before starting the server and shut it down
afterwards.
* httpd/api.cxx (parse_cmd_args): Look for the '--privilege' and
'--unprivileged' arguments.
(module_build): Sign the module if needed.
* httpd/api.h (client_request_data): Add the 'privilege' member.
* httpd/server.h (server): Add new member variable 'cert_db_path' and
function get_cert_db_path().
* httpd/server.cxx (server::server): Take the 'cert_db_path' in the
initializer.
* httpd/nss_funcs.cxx: New file.
* httpd/nss_funcs.h: Ditto.
* httpd/Makefile.am: Add the NSS library flags and libraries. Add
nss_funcs.cxx, ../nsscommon.cxx, and ../privilege.cxx to the list of
sources.
* httpd/Makefile.in: Regenerated.
* configure.ac: Make the NSS libraries a requirement for the web server.
* configure: Regenerated.
Frank Ch. Eigler [Mon, 18 Dec 2017 01:20:43 +0000 (20:20 -0500)]
stap-prep: check_error tweaks
Use printf(1), so that \-codes like \n are expanded properly. Add
advice to "# yum erase kernel-debuginfo" in case a debuginfo-install
results in conflicting /usr/lib/debug/.build-id/XX/YYYY files.
David Smith [Fri, 15 Dec 2017 19:38:36 +0000 (13:38 -0600)]
Move repeated web server code to a new function, execute_and_capture().
* httpd/backends.cxx (local_backend::generate_module): Call
execute_and_capture to do all the work.
(docker_backend::generate_module): Ditto.
* httpd/utils.cxx (execute_and_capture): New function.
* httpd/utils.h: Added execute_and_capture() declaration.
David Smith [Fri, 15 Dec 2017 16:37:55 +0000 (10:37 -0600)]
Rename and rework get_uuid_representation().
* httpd/api.cxx (get_uuid): Renamed and reworked from
get_uuid_representation().
* httpd/api.h: Renamed get_uuid() representation.
* httpd/backends.cxx (generate_module): Call the new get_uuid().
David Smith [Wed, 13 Dec 2017 19:07:34 +0000 (13:07 -0600)]
Update the web service fedora package installer to use packages from koji.
* httpd/docker/fedora_install_package.py: Add support from downloading
packages from koji, the fedora build system.
* httpd/docker/fedora.json: Add 'wget' and 'createrepo_c' to the base list
of packages to be installed.
Aaron Merey [Tue, 12 Dec 2017 23:56:55 +0000 (18:56 -0500)]
BPF translator: change map type of internal globals
* bpf-translate.cxx (build_internal_globals): use BPF_MAP_TYPE_HASH
instead of BPF_MAP_TYPE_ARRAY. The former supports atomic updates.
* stapbpf.cxx (init_internal_globals): replace BPF_EXIST with BPF_ANY
since BPF hash maps elements have to be created before attempting
to update them.
David Smith [Tue, 12 Dec 2017 19:42:59 +0000 (13:42 -0600)]
Grab the stap output from the docker container.
* httpd/backends.cxx (docker_backend::generate_module): Cleanup the
stdout/stderr capturing logic. Use "docker cp" to copy any results from
the container image to the server.
* httpd/api.cxx (get_uuid_representation): Make non-static so other code
can use.
* httpd/api.h: Add get_uuid_representation() declaration.
David Smith [Mon, 11 Dec 2017 16:05:50 +0000 (10:05 -0600)]
Add another PR22551 fix by updating the use of timers for the 4.13 kernel.
* runtime/linux/timer_compatibility.h: Update for the
4.13.16-202.fc26.x86_64 kernel, which has the TIMER_TRACE_FLAGMASK
define (which we were using to determine the old vs. new timer
interface).
* tapset-timers.cxx: Ditto.
David Smith [Thu, 7 Dec 2017 22:07:39 +0000 (16:07 -0600)]
Updated several tapsets for the 4.15 kernel.
* tapset/linux/linuxmib.stp: Update the 'DelayedACKs' probes to handle
a missing '$data' parameter.
* tapset/linux/memory.stp: Handle missing '__GFP_COLD' flag.
* tapset/linux/nfsd.stp: Update the 'nfsd.proc4.rename' probe to handle
upstream changes.
* tapset/linux/signal.stp: Update the 'signal.pending' probe to handle a
missing '$sigsestsize' parameter.
Martin Cermak [Fri, 1 Dec 2017 12:43:43 +0000 (13:43 +0100)]
stap: Add shorthand option --bpf for --runtime=bpf
* cmdline.h: Introduce LONG_OPT_RUNTIME_BPF.
* cmdline.cxx: Define --bpf as LONG_OPT_RUNTIME_BPF.
* session.cxx: Set the runtime option, show the feature
in the --version output, document --bpf.
* man/stap.1.in: Document --bpf.
Aaron Merey [Wed, 22 Nov 2017 20:38:40 +0000 (15:38 -0500)]
Fix stapbpf compile error on older kernels.
* configure.ac: Check for the necessary declarations in linux/bpf.h instead of
simply checking that linux/bpf.h exists. This avoids a compile error if
linux/bpf.h exists but does not contain these declarations (as is the case
with some older kernels). Replaced HAVE_LINUX_BPF_H with HAVE_BPF_DECLS.
* Makefile.am, stapbpf/Makefile.am, main.cxx: Replaced HAVE_LINUX_BPF_H with
HAVE_BPF_DECLS.
* rest: changes caused by autoconf, automake and autoheader.
David Smith [Mon, 20 Nov 2017 21:36:03 +0000 (15:36 -0600)]
Actually run the stap command in the docker container.
* httpd/backends.cxx (docker_backend::generate_module): Actually run the
stap command in the docker container.
* httpd/docker/fedora_install_package.py: Renamed from
fedora_install_packages.py. When installing a kernel, automatically
install the kernel-devel package. Install each package's debuginfo.
* httpd/docker/Makefile.am: Handle fedora_install_package.py rename.
* httpd/docker/fedora.json: Ditto.
* httpd/docker/Makefile.in: Regenerated.
Frank Ch. Eigler [Thu, 16 Nov 2017 22:18:57 +0000 (17:18 -0500)]
stap -L: produce yum/dnf disagnostics if kernel-devel missing
stap listing modes previously suppressed the rpm-finder's suggestions,
if not even kernel-devel was installed, leading to an opaque error
message about .../build/.config not being found. Reenable warnings in
this case, for if kernel-devel was sought and missing, stap as a whole
will shut down soon anyway.
David Smith [Thu, 16 Nov 2017 20:18:39 +0000 (14:18 -0600)]
Update the docker container builder for the web server.
* httpd/docker/fedora_install_packages.py: New file.
* httpd/docker/fedora.json: Change from stages to a scheme where we've got
a docker 'header' and 'footer' items. Each package to be installed runs
the 'install' item.
* httpd/docker/stap_build_docker_container.py: Handle changes to the
distro JSON file.
* httpd/backends.cxx (docker_backend::docker_backend): Fix the docker
build script path.
(docker_backend::generate_module): Update the docker build script
arguments.
* httpd/api.cxx (build_info::module_build): Pass the UUID down to the
backends.
* httpd/backends.h (backend_base): Add a 'uuid' parameter to the
generate_module() function declaration.
* httpd/docker/Makefile.am: Installs the fedora install package script.
* httpd/docker/Makefile.in: Regenerated.
David Smith [Mon, 13 Nov 2017 22:34:36 +0000 (16:34 -0600)]
The web server now writes the client request data to a JSON file.
* httpd/api.h ("client_request_data"): Made 'struct client_request_data'
into a class. Added a get_json_object() function declaration.
* httpd/api.cxx: Change the type on all 'client_request_data' variables.
(response build_collection_rh::POST): Lowercase the distro name to make
things easier when searching for a distro.
(client_request_data::get_json_object): New function.
* httpd/backends.h: Change the type on all 'client_request_data'
variables.
* httpd/backends.cxx: Change the type on all 'client_request_data'
variables.
(docker_backend::docker_backend): Lowercase the distro name to make
things easier when searching for a distro.
(docker_backend::generate_module): Grab a JSON representation of the
client_request_data and write it to a file.
David Smith [Fri, 10 Nov 2017 19:46:53 +0000 (13:46 -0600)]
Fix a couple of subtle bugs in the web server POST parameter handling.
* httpd/server.cxx (get_key_values): Expect a 'post_params_t' instead of
a 'struct request' pointer.
(connection_info::postdataiterator): Be sure to only use 'size' bytes of
'data', otherwise you can get junk values.
(server::access_handler): Pass get_key_values a 'post_params_t' instead
of a 'struct request' pointer.
* httpd/server.h: Define and use the 'post_params_t' type.
David Smith [Wed, 8 Nov 2017 22:12:09 +0000 (16:12 -0600)]
Save the package information on the web server side.
* httpd/api.cxx (build_collection_rh::POST): Package up the file
information (name, package name, build id) from the client.
(build_info::parse_cmd_args): Fix the argument handling. We need to
add an argument 0 to make getopt_long() happy. Grab the right perpass
verbosity level as our level.
* httpd/api.h: Add the 'file_info' structure.
David Smith [Wed, 8 Nov 2017 16:41:52 +0000 (10:41 -0600)]
Rework the web client package information to be valid JSON.
* client-http.cxx (http_client::post): Rework the package information to
be valid JSON. Turn the package information inside out so that instead
of one array of multiple fields, we'll have multiplie single-dimension
arrays.
David Smith [Mon, 6 Nov 2017 15:50:50 +0000 (09:50 -0600)]
Move the web server unzip logic to its proper place.
* httpd/api.cxx (build_collection_rh::POST): Move the unzip logic to
build_info::module_build().
(build_info::module_build): Do the unzip here, so that we don't slow
down the thread handling http requests.
David Smith [Fri, 3 Nov 2017 20:06:13 +0000 (15:06 -0500)]
The web server now tries to build a docker container.
* httpd/backends.cxx (local_backend::generate_module): No longer fail if
the stdout/stderr files already exist.
(docker_backend::docker_backend): Clear out the docker path if we
couldn't find the docker executable. Fix choping off the json file
extension.
(docker_backend::generate_module): Run the script that builds the
docker container.
* httpd/api.cxx (parse_cmd_args): New function.
(module_build): Call parse_cmd_args(). Always create the stdout/stderr
files, so the client will always have something to grab.
* httpd/api.h (client_request_data): Add 'verbose' field.
* httpd/Makefile.am: Add ../cmdline.cxx to the list of files to be compiled.
* httpd/Makefile.in: Regenerated.
David Smith [Mon, 30 Oct 2017 20:39:11 +0000 (15:39 -0500)]
Match non-executable shared libraries in glob patterns.
Shared libraries in Debian do not have executable permissions. This
patch makes glob patterns work for those. Debian policy 8.1:
"Shared libraries should not be installed executable, since the dynamic
linker does not require this and trying to execute a shared library
usually results in a core dump."
David Smith [Mon, 30 Oct 2017 20:25:21 +0000 (15:25 -0500)]
Add a small improvement to testsuite/semok/twenty.stp
* testsuite/semok/twenty.stp: Instead of piping stap's output to "wc",
instead send it to /dev/null. This uses less memory on the system and
avoids masking the return value of stap (by getting the return value of
"wc").
David Smith [Mon, 30 Oct 2017 18:05:18 +0000 (13:05 -0500)]
Improve the way the web client/server sends files.
* client-http.cxx (http_client): Rename the 'script_files' variable to
'files'.
(http_client::post): No longer assume the first file is a script.
(add_file): Renamed from 'add_script_file'. Just add the file.
(http_client_backend::include_file_or_directory): Rewrite to be like the
nss client.
(http_client_backend::package_request): If we've seen some files the
need to be transferred, package them up into a zip file.
(add_tapsets): Deleted function.
(http_client_backend::add_tmpdir_file): Just remember we added a file.
* client-http.h (client_backend): Remove the 'request_files' variable and
add the 'files_seen' variable.
* httpd/api.cxx (POST): Unzip the client zip file if present.
David Smith [Mon, 30 Oct 2017 17:57:07 +0000 (12:57 -0500)]
Improve the way a script is read from standard input.
* main.cxx (passes_0_4): Instead of passing 'cin' to parse(), pass it the
session 'stdin_script' variable.
* csclient.cxx (create_request): Instead of copying the script from
standard input, copy it from the session 'stdin_script' variable.
* session.h (s): Add the 'stdin_script' variable.
* session.cxx (systemtap_session): Copy the 'stdin_script' variable in the
copy constructor.
* testsuite/systemtap.server/server.exp: Add 2 tests for getting a script
from standard input.
git://sourceware.org / systemtap.git / log