David Smith [Mon, 22 Jan 2018 17:38:12 +0000 (11:38 -0600)]
Make the http server send the module signature back to the client.
* httpd/api.cxx (result_info): Switch from expecting one file to return to
handling multiple files.
(result_info::add_file): New function.
(result_info::generate_response): Add all output files to the "files"
section.
(result_info::generate_file_response): Handle the list of output files.
(build_info::module_build): If the module signing worked, add the
signature file to the results.
David Smith [Fri, 19 Jan 2018 14:24:44 +0000 (08:24 -0600)]
Update the systemtap.server/client.exp test to use a mount namespace.
* testsuite/systemtap.server/client.exp: Use a custom mount namespace to
test signed modules.
* testsuite/systemtap.server/server_privilege.exp: Remove unneeded call to
'systemtap_check_users'.
David Smith [Thu, 18 Jan 2018 20:12:22 +0000 (14:12 -0600)]
Add a new server test that runs modules.
* testsuite/systemtap.server/server_privilege.exp: New test that tests
modules built by the compile server.
* testsuite/lib/server_ns.exp: New file.
* testsuite/lib/systemtap.exp (systemtap_check_users): New function.
(setup_systemtap_environment): Call systemtap_check_users.
(shutdown_server): Call server_ns_cleanup.
* testsuite/config/unix.exp: Add server_ns.exp.
* testsuite/Makefile.am: Pass the SYSCONFDIR down to the tests.
* testsuite/Makefile.in: Regenerated.
* systemtap.spec: Create the 'stapusr', 'stapsys', and 'stapdev' users for
the testsuite.
David Smith [Mon, 15 Jan 2018 17:08:39 +0000 (11:08 -0600)]
Enable python3 support in RHEL > 7.
* systemtap.spec: Expect python3 to exist in RHEL > 7 (according to a
message on fedora-devel entitled "Python3 will be in next major RHEL
release, please adjust %if statements accordingly" on Jan. 11, 2018).
David Smith [Mon, 15 Jan 2018 16:23:45 +0000 (10:23 -0600)]
Fix BZ1525651 by making the task_finder shutdown more robust.
* runtime/linux/task_finder.c (__stap_utrace_detach): New function, based
on stap_utrace_detach(), but lets the caller know if any the task had
the utrace engine attached to it.
(stap_utrace_detach): Just call __stap_utrace_detach().
(stap_utrace_detach_ops): Perform the detaching in a loop, so if the
process list snapshot we operate on becomes out of date, we'll still
detach everything.
(__stp_utrace_task_finder_report_clone): Move the
__stp_tf_handler_start() call to the very start of the function.
(__stp_utrace_task_finder_report_exec): Ditto.
(__stp_utrace_task_finder_target_exec): Ditto.
(__stp_utrace_task_finder_target_exit): Ditto.
(__stp_utrace_task_finder_target_quiesce): Ditto.
(__stp_utrace_task_finder_target_syscall_entry): Ditto.
(__stp_utrace_task_finder_target_syscall_exit): Ditto.
David Smith [Mon, 8 Jan 2018 17:28:25 +0000 (11:28 -0600)]
Add the beginnings of module signing to the web server.
* httpd/main.cxx (parse_cmdline): Add the new '--ssl' argument.
(main): Initialize NSS before starting the server and shut it down
afterwards.
* httpd/api.cxx (parse_cmd_args): Look for the '--privilege' and
'--unprivileged' arguments.
(module_build): Sign the module if needed.
* httpd/api.h (client_request_data): Add the 'privilege' member.
* httpd/server.h (server): Add new member variable 'cert_db_path' and
function get_cert_db_path().
* httpd/server.cxx (server::server): Take the 'cert_db_path' in the
initializer.
* httpd/nss_funcs.cxx: New file.
* httpd/nss_funcs.h: Ditto.
* httpd/Makefile.am: Add the NSS library flags and libraries. Add
nss_funcs.cxx, ../nsscommon.cxx, and ../privilege.cxx to the list of
sources.
* httpd/Makefile.in: Regenerated.
* configure.ac: Make the NSS libraries a requirement for the web server.
* configure: Regenerated.
Frank Ch. Eigler [Mon, 18 Dec 2017 01:20:43 +0000 (20:20 -0500)]
stap-prep: check_error tweaks
Use printf(1), so that \-codes like \n are expanded properly. Add
advice to "# yum erase kernel-debuginfo" in case a debuginfo-install
results in conflicting /usr/lib/debug/.build-id/XX/YYYY files.
David Smith [Fri, 15 Dec 2017 19:38:36 +0000 (13:38 -0600)]
Move repeated web server code to a new function, execute_and_capture().
* httpd/backends.cxx (local_backend::generate_module): Call
execute_and_capture to do all the work.
(docker_backend::generate_module): Ditto.
* httpd/utils.cxx (execute_and_capture): New function.
* httpd/utils.h: Added execute_and_capture() declaration.
David Smith [Fri, 15 Dec 2017 16:37:55 +0000 (10:37 -0600)]
Rename and rework get_uuid_representation().
* httpd/api.cxx (get_uuid): Renamed and reworked from
get_uuid_representation().
* httpd/api.h: Renamed get_uuid() representation.
* httpd/backends.cxx (generate_module): Call the new get_uuid().
David Smith [Wed, 13 Dec 2017 19:07:34 +0000 (13:07 -0600)]
Update the web service fedora package installer to use packages from koji.
* httpd/docker/fedora_install_package.py: Add support from downloading
packages from koji, the fedora build system.
* httpd/docker/fedora.json: Add 'wget' and 'createrepo_c' to the base list
of packages to be installed.
Aaron Merey [Tue, 12 Dec 2017 23:56:55 +0000 (18:56 -0500)]
BPF translator: change map type of internal globals
* bpf-translate.cxx (build_internal_globals): use BPF_MAP_TYPE_HASH
instead of BPF_MAP_TYPE_ARRAY. The former supports atomic updates.
* stapbpf.cxx (init_internal_globals): replace BPF_EXIST with BPF_ANY
since BPF hash maps elements have to be created before attempting
to update them.
David Smith [Tue, 12 Dec 2017 19:42:59 +0000 (13:42 -0600)]
Grab the stap output from the docker container.
* httpd/backends.cxx (docker_backend::generate_module): Cleanup the
stdout/stderr capturing logic. Use "docker cp" to copy any results from
the container image to the server.
* httpd/api.cxx (get_uuid_representation): Make non-static so other code
can use.
* httpd/api.h: Add get_uuid_representation() declaration.
David Smith [Mon, 11 Dec 2017 16:05:50 +0000 (10:05 -0600)]
Add another PR22551 fix by updating the use of timers for the 4.13 kernel.
* runtime/linux/timer_compatibility.h: Update for the
4.13.16-202.fc26.x86_64 kernel, which has the TIMER_TRACE_FLAGMASK
define (which we were using to determine the old vs. new timer
interface).
* tapset-timers.cxx: Ditto.
David Smith [Thu, 7 Dec 2017 22:07:39 +0000 (16:07 -0600)]
Updated several tapsets for the 4.15 kernel.
* tapset/linux/linuxmib.stp: Update the 'DelayedACKs' probes to handle
a missing '$data' parameter.
* tapset/linux/memory.stp: Handle missing '__GFP_COLD' flag.
* tapset/linux/nfsd.stp: Update the 'nfsd.proc4.rename' probe to handle
upstream changes.
* tapset/linux/signal.stp: Update the 'signal.pending' probe to handle a
missing '$sigsestsize' parameter.
Martin Cermak [Fri, 1 Dec 2017 12:43:43 +0000 (13:43 +0100)]
stap: Add shorthand option --bpf for --runtime=bpf
* cmdline.h: Introduce LONG_OPT_RUNTIME_BPF.
* cmdline.cxx: Define --bpf as LONG_OPT_RUNTIME_BPF.
* session.cxx: Set the runtime option, show the feature
in the --version output, document --bpf.
* man/stap.1.in: Document --bpf.
Aaron Merey [Wed, 22 Nov 2017 20:38:40 +0000 (15:38 -0500)]
Fix stapbpf compile error on older kernels.
* configure.ac: Check for the necessary declarations in linux/bpf.h instead of
simply checking that linux/bpf.h exists. This avoids a compile error if
linux/bpf.h exists but does not contain these declarations (as is the case
with some older kernels). Replaced HAVE_LINUX_BPF_H with HAVE_BPF_DECLS.
* Makefile.am, stapbpf/Makefile.am, main.cxx: Replaced HAVE_LINUX_BPF_H with
HAVE_BPF_DECLS.
* rest: changes caused by autoconf, automake and autoheader.
David Smith [Mon, 20 Nov 2017 21:36:03 +0000 (15:36 -0600)]
Actually run the stap command in the docker container.
* httpd/backends.cxx (docker_backend::generate_module): Actually run the
stap command in the docker container.
* httpd/docker/fedora_install_package.py: Renamed from
fedora_install_packages.py. When installing a kernel, automatically
install the kernel-devel package. Install each package's debuginfo.
* httpd/docker/Makefile.am: Handle fedora_install_package.py rename.
* httpd/docker/fedora.json: Ditto.
* httpd/docker/Makefile.in: Regenerated.
Frank Ch. Eigler [Thu, 16 Nov 2017 22:18:57 +0000 (17:18 -0500)]
stap -L: produce yum/dnf disagnostics if kernel-devel missing
stap listing modes previously suppressed the rpm-finder's suggestions,
if not even kernel-devel was installed, leading to an opaque error
message about .../build/.config not being found. Reenable warnings in
this case, for if kernel-devel was sought and missing, stap as a whole
will shut down soon anyway.
David Smith [Thu, 16 Nov 2017 20:18:39 +0000 (14:18 -0600)]
Update the docker container builder for the web server.
* httpd/docker/fedora_install_packages.py: New file.
* httpd/docker/fedora.json: Change from stages to a scheme where we've got
a docker 'header' and 'footer' items. Each package to be installed runs
the 'install' item.
* httpd/docker/stap_build_docker_container.py: Handle changes to the
distro JSON file.
* httpd/backends.cxx (docker_backend::docker_backend): Fix the docker
build script path.
(docker_backend::generate_module): Update the docker build script
arguments.
* httpd/api.cxx (build_info::module_build): Pass the UUID down to the
backends.
* httpd/backends.h (backend_base): Add a 'uuid' parameter to the
generate_module() function declaration.
* httpd/docker/Makefile.am: Installs the fedora install package script.
* httpd/docker/Makefile.in: Regenerated.
David Smith [Mon, 13 Nov 2017 22:34:36 +0000 (16:34 -0600)]
The web server now writes the client request data to a JSON file.
* httpd/api.h ("client_request_data"): Made 'struct client_request_data'
into a class. Added a get_json_object() function declaration.
* httpd/api.cxx: Change the type on all 'client_request_data' variables.
(response build_collection_rh::POST): Lowercase the distro name to make
things easier when searching for a distro.
(client_request_data::get_json_object): New function.
* httpd/backends.h: Change the type on all 'client_request_data'
variables.
* httpd/backends.cxx: Change the type on all 'client_request_data'
variables.
(docker_backend::docker_backend): Lowercase the distro name to make
things easier when searching for a distro.
(docker_backend::generate_module): Grab a JSON representation of the
client_request_data and write it to a file.
David Smith [Fri, 10 Nov 2017 19:46:53 +0000 (13:46 -0600)]
Fix a couple of subtle bugs in the web server POST parameter handling.
* httpd/server.cxx (get_key_values): Expect a 'post_params_t' instead of
a 'struct request' pointer.
(connection_info::postdataiterator): Be sure to only use 'size' bytes of
'data', otherwise you can get junk values.
(server::access_handler): Pass get_key_values a 'post_params_t' instead
of a 'struct request' pointer.
* httpd/server.h: Define and use the 'post_params_t' type.
David Smith [Wed, 8 Nov 2017 22:12:09 +0000 (16:12 -0600)]
Save the package information on the web server side.
* httpd/api.cxx (build_collection_rh::POST): Package up the file
information (name, package name, build id) from the client.
(build_info::parse_cmd_args): Fix the argument handling. We need to
add an argument 0 to make getopt_long() happy. Grab the right perpass
verbosity level as our level.
* httpd/api.h: Add the 'file_info' structure.
David Smith [Wed, 8 Nov 2017 16:41:52 +0000 (10:41 -0600)]
Rework the web client package information to be valid JSON.
* client-http.cxx (http_client::post): Rework the package information to
be valid JSON. Turn the package information inside out so that instead
of one array of multiple fields, we'll have multiplie single-dimension
arrays.
David Smith [Mon, 6 Nov 2017 15:50:50 +0000 (09:50 -0600)]
Move the web server unzip logic to its proper place.
* httpd/api.cxx (build_collection_rh::POST): Move the unzip logic to
build_info::module_build().
(build_info::module_build): Do the unzip here, so that we don't slow
down the thread handling http requests.
David Smith [Fri, 3 Nov 2017 20:06:13 +0000 (15:06 -0500)]
The web server now tries to build a docker container.
* httpd/backends.cxx (local_backend::generate_module): No longer fail if
the stdout/stderr files already exist.
(docker_backend::docker_backend): Clear out the docker path if we
couldn't find the docker executable. Fix choping off the json file
extension.
(docker_backend::generate_module): Run the script that builds the
docker container.
* httpd/api.cxx (parse_cmd_args): New function.
(module_build): Call parse_cmd_args(). Always create the stdout/stderr
files, so the client will always have something to grab.
* httpd/api.h (client_request_data): Add 'verbose' field.
* httpd/Makefile.am: Add ../cmdline.cxx to the list of files to be compiled.
* httpd/Makefile.in: Regenerated.
David Smith [Mon, 30 Oct 2017 20:39:11 +0000 (15:39 -0500)]
Match non-executable shared libraries in glob patterns.
Shared libraries in Debian do not have executable permissions. This
patch makes glob patterns work for those. Debian policy 8.1:
"Shared libraries should not be installed executable, since the dynamic
linker does not require this and trying to execute a shared library
usually results in a core dump."
David Smith [Mon, 30 Oct 2017 20:25:21 +0000 (15:25 -0500)]
Add a small improvement to testsuite/semok/twenty.stp
* testsuite/semok/twenty.stp: Instead of piping stap's output to "wc",
instead send it to /dev/null. This uses less memory on the system and
avoids masking the return value of stap (by getting the return value of
"wc").
David Smith [Mon, 30 Oct 2017 18:05:18 +0000 (13:05 -0500)]
Improve the way the web client/server sends files.
* client-http.cxx (http_client): Rename the 'script_files' variable to
'files'.
(http_client::post): No longer assume the first file is a script.
(add_file): Renamed from 'add_script_file'. Just add the file.
(http_client_backend::include_file_or_directory): Rewrite to be like the
nss client.
(http_client_backend::package_request): If we've seen some files the
need to be transferred, package them up into a zip file.
(add_tapsets): Deleted function.
(http_client_backend::add_tmpdir_file): Just remember we added a file.
* client-http.h (client_backend): Remove the 'request_files' variable and
add the 'files_seen' variable.
* httpd/api.cxx (POST): Unzip the client zip file if present.
David Smith [Mon, 30 Oct 2017 17:57:07 +0000 (12:57 -0500)]
Improve the way a script is read from standard input.
* main.cxx (passes_0_4): Instead of passing 'cin' to parse(), pass it the
session 'stdin_script' variable.
* csclient.cxx (create_request): Instead of copying the script from
standard input, copy it from the session 'stdin_script' variable.
* session.h (s): Add the 'stdin_script' variable.
* session.cxx (systemtap_session): Copy the 'stdin_script' variable in the
copy constructor.
* testsuite/systemtap.server/server.exp: Add 2 tests for getting a script
from standard input.
David Smith [Wed, 25 Oct 2017 13:26:19 +0000 (08:26 -0500)]
Add a minor update to systemtap.stress/tapset_functions.exp.
* testsuite/systemtap.stress/tapset_functions.exp: Change the name of the
"testscript" test to "tapset_functions testscript" test so its name
sticks out a bit more on failure.
David Smith [Fri, 20 Oct 2017 19:52:06 +0000 (14:52 -0500)]
The web server now gives a better error message on an unsupported request.
* httpd/backends.cxx: Provide a new "default" backend, which just returns
a reasonable error message (that this server cannot support the request).
* httpd/api.cxx (module_build): Improved error message.
Frank Ch. Eigler [Fri, 20 Oct 2017 14:01:58 +0000 (10:01 -0400)]
rhbz1504009: let dtrace -G -o /dev/null run, as in autoconf
commit c245153ca193c471a8c broke the ability of dtrace to be tested in
autoconf "-G -o /dev/null" usage, because its output file name was too
simple a function of the input name, and normal users can't write to
/dev/null.dtrace-temp.c . Now we back down to mkstemp, like before,
upon a failure of the simple concatenated name.
David Smith [Thu, 19 Oct 2017 18:24:06 +0000 (13:24 -0500)]
Improve web service startup a bit.
* client-http.cxx (http_client::download): If verbose is >= 4, set the
VERBOSE curl option so that all packets are dumped.
(http_client_backend::find_and_connect_to_server): Instead of trying to
GET the /builds directory (which isn't implemented), grab the base
server directory.
* httpd/server.cxx (server::server): Add a base directory request
handler.
* httpd/server.h: Move 'class server' constructor into server.cxx.
David Smith [Thu, 19 Oct 2017 18:19:04 +0000 (13:19 -0500)]
Add string trim utility functions.
* util.cxx (ltrim): New function.
(rtrim): Ditto.
(trim): Ditto.
(get_distro_info): Trim 'name' and 'version' items.
* util.h: Add trim function declarations.
* tapset-mark.cxx (mark_builder::build): Use new trim functions.
* tapsets.cxx (dwarf_builder::build): Ditto.
David Smith [Mon, 16 Oct 2017 20:30:52 +0000 (15:30 -0500)]
Fix PR22278 by fixing server breakage caused by commit 586c8666a.
* csclient.cxx (create_request): Let the backend add the argument in
include_file_or_directory().
* testsuite/systemtap.server/server.exp: Add 3 simple tests: a filename, a
"-L", and a "-I tapset -e SCRIPT".
* testsuite/lib/systemtap.exp (stap_run_batch): If the "filename" argument
is empty, assume this stap command line doesn't need a file.