Bug 17256 - investigate on-the-fly support for other probe types
Summary: investigate on-the-fly support for other probe types
Status: NEW
Alias: None
Product: systemtap
Classification: Unclassified
Component: runtime (show other bugs)
Version: unspecified
: P2 normal
Target Milestone: ---
Assignee: Unassigned
Depends on:
Reported: 2014-08-11 21:10 UTC by Jonathan Lebon
Modified: 2015-05-14 14:18 UTC (History)
0 users

See Also:
Last reconfirmed:

tracepoints: add on-the-fly support (1.43 KB, patch)
2015-05-13 21:06 UTC, Jonathan Lebon
Details | Diff
tracepoint_onthefly.exp: new testcase (2.60 KB, patch)
2015-05-13 21:06 UTC, Jonathan Lebon
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Jonathan Lebon 2014-08-11 21:10:51 UTC
As a follow-up to bug 10995, investigate other probe types where support for on-the-fly operations would be worth adding (e.g. probe types with high overhead cost).


When a probe type supports on-the-fly operations, it means that it is capable of being turned off (or at least put in a lower-overhead state) during runtime depending on the value of the probe condition.

For example, imagine we have:

probe foo if (bar) { ... }

If the underlying probe type of 'foo' has otf support, then it will be turned off (or on) whenever the expression 'bar' transitions from true to false (or false to true).

The way it works underneath is that any probe which writes to a global var involved in the evaluation of 'bar' also sets a flag (need_module_refresh) if the evaluation result has changed. The current value of the evaluation is stored in the 'cond_enabled' field of the affected probe.

Upon exiting the probe, if the context is safe enough, schedule_work() is called so that a work item is added to the system queue (module_refresh_work). This work will eventually call systemtap_module_refresh() in process context, at which point the affected probes are turned on/off as dictated by the 'cond_enabled' fields.

If the context is not safe for workqueue manipulation, then schedule_work() is not called. Instead, an always-running background timer (module_refresh_timer) eventually fires, which does the call to schedule_work() if necessary.


Here are roughly the necessary steps for adding support:

1. Override the otf_supported() and otf_safe_context() functions of the specific derived_probe_group:

- otf_supported() should return true if on-the-fly is supported. This will cause the need_module_refresh flag to be set whenever probes of that type have their conditions changed, which will eventually cause systemtap_module_refresh() to be called.
- otf_safe_context() should return true if calling schedule_work() is safe to do in the context of that probe type. If not, then return true (the background timer will automatically be set to run). In general, hardirq context is safe, while softirq may not be. Probes which fire in process context are most likely safe as well.

Note how otf_supported() and otf_safe_context() are independent of each other. The former determines whether the probe type can be turned on/off, while the latter merely indicates whether calling schedule_work() is a safe thing to do.

Although otf_safe_context() currently defaults to 'false' for mostly all probe types, it may be worth it to go over all the probe types and override it to 'true' wherever appropriate. This will allow us to reduce dependence on the background timer.

2. Write the actual refresh code. You will need to modify the code in emit_module_refresh() for that derived_probe_group (or the code in the function which is called by the code emitted in emit_module_refresh()). In that code, inspect the values of cond_enabled for the probes in your group and turn on/off as needed.

For references, see the following exmaples:
- stapkp_refresh() in runtime/linux/kprobes.c, 
- stapiu_refresh() in runtime/linux/uprobes-inode.c
- hrtimer_derived_probe_group::emit_module_refresh() in tapset-timers.cxx

You might need to create a new field in the probe type specific struct to track whether the probe is enabled or disabled. If you can directly query the kernel that's even better (e.g. for kprobes, we have kprobe_disabled(), while for inode-uprobes, we had to add a 'registered' field to stapiu_consumer).

Use dbug_otf() whenever you turn on/off probes. This will be useful for testing.

3. Write the testcase. The testsuite/systemtap.onthefly directory is where the new testcase would be added. Look at the already existing examples there (hrtimer, kprobes, and uprobes_onthefly.exp) and go from there. The idea is both to verify correct turning on and off in various situations, as well as stress-testing it.

4. Profit

And that's about it! Again, for reference, you can always look back at how otf support was added to kprobes, inode-uprobes, and hrtimer probes.
Comment 1 Jonathan Lebon 2015-05-13 21:05:54 UTC
Here are the patches for adding on-the-fly support for tracepoints. The stress subtests will on very rare occasions crash the computer because it uses a process.end probe on a busy system, which is known to cause crashes, regardless of these patches (see PR17461). Still, it might be better to wait until the above is fixed to verify that these patches cause no issues if run for a whole day.
Comment 2 Jonathan Lebon 2015-05-13 21:06:17 UTC
Created attachment 8313 [details]
tracepoints: add on-the-fly support
Comment 3 Jonathan Lebon 2015-05-13 21:06:48 UTC
Created attachment 8314 [details]
tracepoint_onthefly.exp: new testcase
Comment 4 Jonathan Lebon 2015-05-14 14:18:42 UTC
I merged the testcase in commit 085535d. I setup_kfail'ed the subtests in commit 4f1a4fc so that it can be easily reverted once tracepoint otf support actually goes in.