David Smith [Wed, 13 Dec 2017 19:07:34 +0000 (13:07 -0600)]
Update the web service fedora package installer to use packages from koji.
* httpd/docker/fedora_install_package.py: Add support from downloading
packages from koji, the fedora build system.
* httpd/docker/fedora.json: Add 'wget' and 'createrepo_c' to the base list
of packages to be installed.
Aaron Merey [Tue, 12 Dec 2017 23:56:55 +0000 (18:56 -0500)]
BPF translator: change map type of internal globals
* bpf-translate.cxx (build_internal_globals): use BPF_MAP_TYPE_HASH
instead of BPF_MAP_TYPE_ARRAY. The former supports atomic updates.
* stapbpf.cxx (init_internal_globals): replace BPF_EXIST with BPF_ANY
since BPF hash maps elements have to be created before attempting
to update them.
David Smith [Tue, 12 Dec 2017 19:42:59 +0000 (13:42 -0600)]
Grab the stap output from the docker container.
* httpd/backends.cxx (docker_backend::generate_module): Cleanup the
stdout/stderr capturing logic. Use "docker cp" to copy any results from
the container image to the server.
* httpd/api.cxx (get_uuid_representation): Make non-static so other code
can use.
* httpd/api.h: Add get_uuid_representation() declaration.
David Smith [Mon, 11 Dec 2017 16:05:50 +0000 (10:05 -0600)]
Add another PR22551 fix by updating the use of timers for the 4.13 kernel.
* runtime/linux/timer_compatibility.h: Update for the
4.13.16-202.fc26.x86_64 kernel, which has the TIMER_TRACE_FLAGMASK
define (which we were using to determine the old vs. new timer
interface).
* tapset-timers.cxx: Ditto.
David Smith [Thu, 7 Dec 2017 22:07:39 +0000 (16:07 -0600)]
Updated several tapsets for the 4.15 kernel.
* tapset/linux/linuxmib.stp: Update the 'DelayedACKs' probes to handle
a missing '$data' parameter.
* tapset/linux/memory.stp: Handle missing '__GFP_COLD' flag.
* tapset/linux/nfsd.stp: Update the 'nfsd.proc4.rename' probe to handle
upstream changes.
* tapset/linux/signal.stp: Update the 'signal.pending' probe to handle a
missing '$sigsestsize' parameter.
Martin Cermak [Fri, 1 Dec 2017 12:43:43 +0000 (13:43 +0100)]
stap: Add shorthand option --bpf for --runtime=bpf
* cmdline.h: Introduce LONG_OPT_RUNTIME_BPF.
* cmdline.cxx: Define --bpf as LONG_OPT_RUNTIME_BPF.
* session.cxx: Set the runtime option, show the feature
in the --version output, document --bpf.
* man/stap.1.in: Document --bpf.
Aaron Merey [Wed, 22 Nov 2017 20:38:40 +0000 (15:38 -0500)]
Fix stapbpf compile error on older kernels.
* configure.ac: Check for the necessary declarations in linux/bpf.h instead of
simply checking that linux/bpf.h exists. This avoids a compile error if
linux/bpf.h exists but does not contain these declarations (as is the case
with some older kernels). Replaced HAVE_LINUX_BPF_H with HAVE_BPF_DECLS.
* Makefile.am, stapbpf/Makefile.am, main.cxx: Replaced HAVE_LINUX_BPF_H with
HAVE_BPF_DECLS.
* rest: changes caused by autoconf, automake and autoheader.
David Smith [Mon, 20 Nov 2017 21:36:03 +0000 (15:36 -0600)]
Actually run the stap command in the docker container.
* httpd/backends.cxx (docker_backend::generate_module): Actually run the
stap command in the docker container.
* httpd/docker/fedora_install_package.py: Renamed from
fedora_install_packages.py. When installing a kernel, automatically
install the kernel-devel package. Install each package's debuginfo.
* httpd/docker/Makefile.am: Handle fedora_install_package.py rename.
* httpd/docker/fedora.json: Ditto.
* httpd/docker/Makefile.in: Regenerated.
Frank Ch. Eigler [Thu, 16 Nov 2017 22:18:57 +0000 (17:18 -0500)]
stap -L: produce yum/dnf disagnostics if kernel-devel missing
stap listing modes previously suppressed the rpm-finder's suggestions,
if not even kernel-devel was installed, leading to an opaque error
message about .../build/.config not being found. Reenable warnings in
this case, for if kernel-devel was sought and missing, stap as a whole
will shut down soon anyway.
David Smith [Thu, 16 Nov 2017 20:18:39 +0000 (14:18 -0600)]
Update the docker container builder for the web server.
* httpd/docker/fedora_install_packages.py: New file.
* httpd/docker/fedora.json: Change from stages to a scheme where we've got
a docker 'header' and 'footer' items. Each package to be installed runs
the 'install' item.
* httpd/docker/stap_build_docker_container.py: Handle changes to the
distro JSON file.
* httpd/backends.cxx (docker_backend::docker_backend): Fix the docker
build script path.
(docker_backend::generate_module): Update the docker build script
arguments.
* httpd/api.cxx (build_info::module_build): Pass the UUID down to the
backends.
* httpd/backends.h (backend_base): Add a 'uuid' parameter to the
generate_module() function declaration.
* httpd/docker/Makefile.am: Installs the fedora install package script.
* httpd/docker/Makefile.in: Regenerated.
David Smith [Mon, 13 Nov 2017 22:34:36 +0000 (16:34 -0600)]
The web server now writes the client request data to a JSON file.
* httpd/api.h ("client_request_data"): Made 'struct client_request_data'
into a class. Added a get_json_object() function declaration.
* httpd/api.cxx: Change the type on all 'client_request_data' variables.
(response build_collection_rh::POST): Lowercase the distro name to make
things easier when searching for a distro.
(client_request_data::get_json_object): New function.
* httpd/backends.h: Change the type on all 'client_request_data'
variables.
* httpd/backends.cxx: Change the type on all 'client_request_data'
variables.
(docker_backend::docker_backend): Lowercase the distro name to make
things easier when searching for a distro.
(docker_backend::generate_module): Grab a JSON representation of the
client_request_data and write it to a file.
David Smith [Fri, 10 Nov 2017 19:46:53 +0000 (13:46 -0600)]
Fix a couple of subtle bugs in the web server POST parameter handling.
* httpd/server.cxx (get_key_values): Expect a 'post_params_t' instead of
a 'struct request' pointer.
(connection_info::postdataiterator): Be sure to only use 'size' bytes of
'data', otherwise you can get junk values.
(server::access_handler): Pass get_key_values a 'post_params_t' instead
of a 'struct request' pointer.
* httpd/server.h: Define and use the 'post_params_t' type.
David Smith [Wed, 8 Nov 2017 22:12:09 +0000 (16:12 -0600)]
Save the package information on the web server side.
* httpd/api.cxx (build_collection_rh::POST): Package up the file
information (name, package name, build id) from the client.
(build_info::parse_cmd_args): Fix the argument handling. We need to
add an argument 0 to make getopt_long() happy. Grab the right perpass
verbosity level as our level.
* httpd/api.h: Add the 'file_info' structure.
David Smith [Wed, 8 Nov 2017 16:41:52 +0000 (10:41 -0600)]
Rework the web client package information to be valid JSON.
* client-http.cxx (http_client::post): Rework the package information to
be valid JSON. Turn the package information inside out so that instead
of one array of multiple fields, we'll have multiplie single-dimension
arrays.
David Smith [Mon, 6 Nov 2017 15:50:50 +0000 (09:50 -0600)]
Move the web server unzip logic to its proper place.
* httpd/api.cxx (build_collection_rh::POST): Move the unzip logic to
build_info::module_build().
(build_info::module_build): Do the unzip here, so that we don't slow
down the thread handling http requests.
David Smith [Fri, 3 Nov 2017 20:06:13 +0000 (15:06 -0500)]
The web server now tries to build a docker container.
* httpd/backends.cxx (local_backend::generate_module): No longer fail if
the stdout/stderr files already exist.
(docker_backend::docker_backend): Clear out the docker path if we
couldn't find the docker executable. Fix choping off the json file
extension.
(docker_backend::generate_module): Run the script that builds the
docker container.
* httpd/api.cxx (parse_cmd_args): New function.
(module_build): Call parse_cmd_args(). Always create the stdout/stderr
files, so the client will always have something to grab.
* httpd/api.h (client_request_data): Add 'verbose' field.
* httpd/Makefile.am: Add ../cmdline.cxx to the list of files to be compiled.
* httpd/Makefile.in: Regenerated.
David Smith [Mon, 30 Oct 2017 20:39:11 +0000 (15:39 -0500)]
Match non-executable shared libraries in glob patterns.
Shared libraries in Debian do not have executable permissions. This
patch makes glob patterns work for those. Debian policy 8.1:
"Shared libraries should not be installed executable, since the dynamic
linker does not require this and trying to execute a shared library
usually results in a core dump."
David Smith [Mon, 30 Oct 2017 20:25:21 +0000 (15:25 -0500)]
Add a small improvement to testsuite/semok/twenty.stp
* testsuite/semok/twenty.stp: Instead of piping stap's output to "wc",
instead send it to /dev/null. This uses less memory on the system and
avoids masking the return value of stap (by getting the return value of
"wc").
David Smith [Mon, 30 Oct 2017 18:05:18 +0000 (13:05 -0500)]
Improve the way the web client/server sends files.
* client-http.cxx (http_client): Rename the 'script_files' variable to
'files'.
(http_client::post): No longer assume the first file is a script.
(add_file): Renamed from 'add_script_file'. Just add the file.
(http_client_backend::include_file_or_directory): Rewrite to be like the
nss client.
(http_client_backend::package_request): If we've seen some files the
need to be transferred, package them up into a zip file.
(add_tapsets): Deleted function.
(http_client_backend::add_tmpdir_file): Just remember we added a file.
* client-http.h (client_backend): Remove the 'request_files' variable and
add the 'files_seen' variable.
* httpd/api.cxx (POST): Unzip the client zip file if present.
David Smith [Mon, 30 Oct 2017 17:57:07 +0000 (12:57 -0500)]
Improve the way a script is read from standard input.
* main.cxx (passes_0_4): Instead of passing 'cin' to parse(), pass it the
session 'stdin_script' variable.
* csclient.cxx (create_request): Instead of copying the script from
standard input, copy it from the session 'stdin_script' variable.
* session.h (s): Add the 'stdin_script' variable.
* session.cxx (systemtap_session): Copy the 'stdin_script' variable in the
copy constructor.
* testsuite/systemtap.server/server.exp: Add 2 tests for getting a script
from standard input.
David Smith [Wed, 25 Oct 2017 13:26:19 +0000 (08:26 -0500)]
Add a minor update to systemtap.stress/tapset_functions.exp.
* testsuite/systemtap.stress/tapset_functions.exp: Change the name of the
"testscript" test to "tapset_functions testscript" test so its name
sticks out a bit more on failure.
David Smith [Fri, 20 Oct 2017 19:52:06 +0000 (14:52 -0500)]
The web server now gives a better error message on an unsupported request.
* httpd/backends.cxx: Provide a new "default" backend, which just returns
a reasonable error message (that this server cannot support the request).
* httpd/api.cxx (module_build): Improved error message.
Frank Ch. Eigler [Fri, 20 Oct 2017 14:01:58 +0000 (10:01 -0400)]
rhbz1504009: let dtrace -G -o /dev/null run, as in autoconf
commit c245153ca193c471a8c broke the ability of dtrace to be tested in
autoconf "-G -o /dev/null" usage, because its output file name was too
simple a function of the input name, and normal users can't write to
/dev/null.dtrace-temp.c . Now we back down to mkstemp, like before,
upon a failure of the simple concatenated name.
David Smith [Thu, 19 Oct 2017 18:24:06 +0000 (13:24 -0500)]
Improve web service startup a bit.
* client-http.cxx (http_client::download): If verbose is >= 4, set the
VERBOSE curl option so that all packets are dumped.
(http_client_backend::find_and_connect_to_server): Instead of trying to
GET the /builds directory (which isn't implemented), grab the base
server directory.
* httpd/server.cxx (server::server): Add a base directory request
handler.
* httpd/server.h: Move 'class server' constructor into server.cxx.
David Smith [Thu, 19 Oct 2017 18:19:04 +0000 (13:19 -0500)]
Add string trim utility functions.
* util.cxx (ltrim): New function.
(rtrim): Ditto.
(trim): Ditto.
(get_distro_info): Trim 'name' and 'version' items.
* util.h: Add trim function declarations.
* tapset-mark.cxx (mark_builder::build): Use new trim functions.
* tapsets.cxx (dwarf_builder::build): Ditto.
David Smith [Mon, 16 Oct 2017 20:30:52 +0000 (15:30 -0500)]
Fix PR22278 by fixing server breakage caused by commit 586c8666a.
* csclient.cxx (create_request): Let the backend add the argument in
include_file_or_directory().
* testsuite/systemtap.server/server.exp: Add 3 simple tests: a filename, a
"-L", and a "-I tapset -e SCRIPT".
* testsuite/lib/systemtap.exp (stap_run_batch): If the "filename" argument
is empty, assume this stap command line doesn't need a file.
BPF translation internals involve data structures that do not track
the staptree level entities they are based on. So when an error
occurs, there's no quick way to mapping it back to the token, for an
ideal error message. translate_bpf_pass() now tries to track a
last-token* as it goes translating all the probe bodies to bpf, so
that a semantic_error can be synthesized if needed.
This is still far from ideal; a single exception causes giving up
trying all the remaining probes; and all begin/end probes are
translated into single larger bpf programs, so it's harder to narrow
down the cause. Baby steps.
Frank Ch. Eigler [Fri, 13 Oct 2017 23:00:37 +0000 (19:00 -0400)]
bpf translator: catch runtime_errors
Some simple test cases can run up against bpf-translate limitations.
Those that are signalled with a runtime_error exception need to be
caught at least in translate_bpf_pass() and a message improvised, so
that there is something more to go on than an anonymous pass-4
failure.
David Smith [Fri, 13 Oct 2017 20:52:44 +0000 (15:52 -0500)]
Make small improvements to the systemtap server testsuite.
* testsuite/systemtap.server/server.exp: When restarting the avahi daemon,
try to fix a restart failure.
* testsuite/systemtap.server/client.exp (list_servers): Add debug output.
* testsuite/lib/systemtap.exp (shutdown_server): Try to ensure that the
systemtap server is really gone before returning.
David Smith [Wed, 11 Oct 2017 15:53:19 +0000 (10:53 -0500)]
Update the server_locale.exp test case to avoid spurious failures.
* testsuite/systemtap.server/server_locale.exp: Handle the case where the
French langpack isn't installed on the system. Do a French sniff test
before trying to use the French locale.
* client-nss.h (client_backend): Removed 'server_tmpdir' declaration that
shadowed the declaration in the base class. This caused the code in
csclient.cxx to get the wrong value for 'server_tmpdir'.
Aaron Merey [Fri, 6 Oct 2017 23:34:24 +0000 (19:34 -0400)]
stapbpf: fix global variable initialization bug.
* bpf-translate.cxx: The translator now adds assignment statements to
the beginning of the "stap_begin" bpf program in order to initialize
global variables with any user-supplied default values. Currently there
is no support for string literals, the translator will still ignore them.
(struct init_block): manages the resources associated with these
assignments.
* tapsets.cxx(sort_for_bpf), tapset-been.cxx (sort_for_bpf): immediately
returns if the probe group pointer is NULL.
* bpf_tests: added new tests and fixed a syntax error in globals1.stp and
globals2.stp.
David Smith [Fri, 6 Oct 2017 15:32:53 +0000 (10:32 -0500)]
Add a PR20516 workaround by using kallsyms_lookup_name to find udelay_simple.
* runtime/linux/common_session_state.h (stp_session_init): Use
kallsyms_lookup_name() to find udelay_simple() if it exists but isn't
exported.
* runtime/linux/runtime.h: Ditto.
* buildrun.cxx (compile_pass): Add an new autoconf test for
udelay_simple().
* runtime/linux/autoconf-udelay_simple.c: New file.