This is the mail archive of the systemtap@sourceware.org mailing list for the systemtap project.


Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]
Other format: [Raw text]

Re: reducing cost of user-space probes


On Mon, Apr 24, 2017 at 6:58 AM, O Mahony, Billy
<billy.o.mahony@intel.com> wrote:
> Hi,
>
> I'm new to systemtap and I am using it to add some probes into a user space application.
>
> The probe is pretty simple - it collects one integer argument and presents a histogram every 3 seconds.
>
> The probe is working fine and I'm getting results that are sensible. The application is a packet processing application that is using a user space io library (DPDK) to read batches of network packets directly into user space.  The probe is called about 750K times per second  (I have 10Gb link with 64B packets which generates 14.8M packets per second - but the batch size (that's the stat I'm tracing) - is about 20 so 750K probe hits per sec.
>
> When the probe is in use I see less performance from the packet processing application - it starts loosing packets at about 90% of it's non-probed throughput.
>
> However, when I run stap I see:
>
>> Pass 4: compiled C into "stap_13723.ko" in 9020usr/980sys/10638real ms
>
> Does this mean that each time the probe is hit that a system call is made to this new .ko module? That would surely mean quite a lot of overhead. If this is correct, can this overhead be avoided for user space probes.

The default "linux" runtime generates source for a kernel module,
compiles and installs it behind the scenes. That's how the default
runtime works. A system call is not made to the kernel module every
time the probe is hit (even it it wanted to, kernel modules can't call
system calls). Systemtap uses a kernel feature called 'uprobes' to
handle user-space probe hits.

> Alternatively is there a way to only execute the script every n times the probe is hit?
>
> Maybe there is a compile time macro that does this or some .stap command that does an early return from the script X% of the time. I searched for 'sample/sampling' in the lang ref but I didn't see anything.

Sure, if you want to exit early, just call "next". It could look
something like the following, if you only want to look at every 10th
function hit:

====
global iterations

probe process("/usr/bin/foo").function("bar")
{
    iterations++
    if (iterations % 10 != 0)
        next

    # ... data collection here
}
====

If you'd like help in making your script run faster, you'll need to
show it to us so that we can make suggestions.

-- 
David Smith
Principal Software Engineer
Red Hat


Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]