Introducing stap-exporter

[Aaron Merey]

Hello,

A new prometheus-systemtap interoperation mechanism, stap-exporter, has 
recently been added to systemtap. stap-exporter is a small python program
that manages systemtap sessions and relays prometheus metrics from the
sessions to remote requesters on demand. stap-exporter runs as a systemd
service and listens to a configurable TCP port for requests. It is inspired
by myllynen's pmdabcc and cloudflare's ebpf_exporter.

A new probe alias "prometheus" and new tapset macros have also been added to
help make it easy for scripts to produce metrics. Prometheus probes trigger
when stap-exporter receives a request for metrics and prometheus_dump_arrayN
macros are used to produce metrics from arrays with N-element keys. In the
following example script, the prometheus_dump_array2 macro is used to produce
metrics from an array indexed by a cpu number and a thread id and whose values
represent a count of sys_open calls.

$ cat systemtap/stap-exporter/scripts/example.stp
global arr

probe kprobe.function("sys_open") {
  arr[cpu(), tid()]++
}

probe prometheus {
  @prometheus_dump_array2(arr, "count", "cpu", "tid")
}

stap-exporter can be configured to either run scripts at startup or run them
only when requested. stap-exporter can also be configured to run scripts
indefinitely or for a set amount of time. Once stap-exporter launches a session,
its metrics are available as plain text at the URL [HOSTNAME]:[PORT]/[SCRIPTNAME]
(PORT can be selected in stap-exporter's config file).

$ curl [HOSTNAME]:[PORT]/example.stp
count{tid="22111",cpu="0"} 2
count{tid="2245",cpu="2"} 4
count{tid="2362",cpu="3"} 1
[...]

For more information on installing and configuring stap-exporter, see USAGE
and exporter.conf within systemtap/stap-exporter. It should be noted that
stap-exporter does not currently work with systemtap's eBPF runtime. If you
have any feedback to offer, I'd be happy to hear it.

All the best,

Aaron Merey