As a follow-up to bug 10995, investigate other probe types where support for on-the-fly operations would be worth adding (e.g. probe types with high overhead cost).

BASIC IMPLEMENTATION INFO

When a probe type supports on-the-fly operations, it means that it is capable of being turned off (or at least put in a lower-overhead state) during runtime depending on the value of the probe condition.

For example, imagine we have:

probe foo if (bar) { ... }

If the underlying probe type of 'foo' has otf support, then it will be turned off (or on) whenever the expression 'bar' transitions from true to false (or false to true).

The way it works underneath is that any probe which writes to a global var involved in the evaluation of 'bar' also sets a flag (need_module_refresh) if the evaluation result has changed. The current value of the evaluation is stored in the 'cond_enabled' field of the affected probe.

Upon exiting the probe, if the context is safe enough, schedule_work() is called so that a work item is added to the system queue (module_refresh_work). This work will eventually call systemtap_module_refresh() in process context, at which point the affected probes are turned on/off as dictated by the 'cond_enabled' fields.

If the context is not safe for workqueue manipulation, then schedule_work() is not called. Instead, an always-running background timer (module_refresh_timer) eventually fires, which does the call to schedule_work() if necessary.

ADDING SUPPORT FOR NEW PROBE TYPES

Here are roughly the necessary steps for adding support:

1. Override the otf_supported() and otf_safe_context() functions of the specific derived_probe_group:

- otf_supported() should return true if on-the-fly is supported. This will cause the need_module_refresh flag to be set whenever probes of that type have their conditions changed, which will eventually cause systemtap_module_refresh() to be called.
- otf_safe_context() should return true if calling schedule_work() is safe to do in the context of that probe type. If not, then return true (the background timer will automatically be set to run). In general, hardirq context is safe, while softirq may not be. Probes which fire in process context are most likely safe as well.

Note how otf_supported() and otf_safe_context() are independent of each other. The former determines whether the probe type can be turned on/off, while the latter merely indicates whether calling schedule_work() is a safe thing to do.

Although otf_safe_context() currently defaults to 'false' for mostly all probe types, it may be worth it to go over all the probe types and override it to 'true' wherever appropriate. This will allow us to reduce dependence on the background timer.

2. Write the actual refresh code. You will need to modify the code in emit_module_refresh() for that derived_probe_group (or the code in the function which is called by the code emitted in emit_module_refresh()). In that code, inspect the values of cond_enabled for the probes in your group and turn on/off as needed.

For references, see the following exmaples:
- stapkp_refresh() in runtime/linux/kprobes.c, 
- stapiu_refresh() in runtime/linux/uprobes-inode.c
- hrtimer_derived_probe_group::emit_module_refresh() in tapset-timers.cxx

You might need to create a new field in the probe type specific struct to track whether the probe is enabled or disabled. If you can directly query the kernel that's even better (e.g. for kprobes, we have kprobe_disabled(), while for inode-uprobes, we had to add a 'registered' field to stapiu_consumer).

Use dbug_otf() whenever you turn on/off probes. This will be useful for testing.

3. Write the testcase. The testsuite/systemtap.onthefly directory is where the new testcase would be added. Look at the already existing examples there (hrtimer, kprobes, and uprobes_onthefly.exp) and go from there. The idea is both to verify correct turning on and off in various situations, as well as stress-testing it.

4. Profit

And that's about it! Again, for reference, you can always look back at how otf support was added to kprobes, inode-uprobes, and hrtimer probes.