This is the mail archive of the
systemtap@sourceware.org
mailing list for the systemtap project.
netlink vs relayfs performance
- From: Martin Hunt <hunt at redhat dot com>
- To: Tom Zanussi <zanussi at us dot ibm dot com>
- Cc: "systemtap at sources dot redhat dot com" <systemtap at sources dot redhat dot com>
- Date: Tue, 02 Aug 2005 03:31:15 -0700
- Subject: netlink vs relayfs performance
- Organization: Red Hat Inc.
(This is preliminary. There are still some optimizations we need to
try.)
I wrote a simple script to send data as fast as it can, 100 bytes per
kprobe, raising the amount of data up to 1GB in 100MB increments. If
the transport dropped any data, I up the buffers and try again. The goal
here is to model what happens when a systemtap program generates too
much output too fast, a worse-case scenario.
Both netlink and relayfs go fine until about 800MB then they require
much larger buffers to get up to 1GB. relayfs uses less than half the
CPU and is faster (37MB/sec vs 24 MB/sec).
I also ran this on my 1.7GHz Pentium-M laptop. It has much lower
kprobes overhead, so it can potentially run this benchmark much faster,
and it did for relayfs, with transfer rates of 44 MB/sec. However with
only 512MB memory and a slow disk, as soon as disk cache is exhausted it
cannot sustain that rate. With 500MB, no amount of relayfs or netlink
buffers will help it. (Netlink managed only 18MB/sec on the laptop)
If you are trying to capture data faster than your disks can write, you
are going to eventually drop data. Both netlink and relayfs can transfer
data faster than disks typically run. However, if the data you are
generating is less than your system's RAM, or if you need to be able to
handle bursts of data, relayfs appears to support capture higher rates
and slows the system down less.
Sample Raw data:
TRANS BENCH for 2.6.9-11.26.EL.huntsmp on x86_64
Red Hat Enterprise Linux WS release 4 (Nahant Update 1)
tiger: 02:23:44 up 46 min, 3 users, load average: 0.06, 0.14, 0.59
processors: 4 (2 physical) Intel(R) Xeon(TM) CPU 2.80GHz
MemTotal: 1025488 kB MemFree: 61608 kB
--------------------------------------
Function call plus kprobe overhead = 1731 ns per call
--------------------------------------
NETLINK with 1MB buffers
_stp_printf on 100 chars = 2408 ns / call system + user time.
_stp_printf of 100MB in 4.14 secs real time.
Transfer rate = 24.15 MB/sec
Transport failures: 0
--------------------------------------
NETLINK with 1MB buffers
_stp_printf on 100 chars = 2440 ns / call system + user time.
_stp_printf of 200MB in 8.34 secs real time.
Transfer rate = 23.98 MB/sec
Transport failures: 0
--------------------------------------
NETLINK with 1MB buffers
_stp_printf on 100 chars = 2501 ns / call system + user time.
_stp_printf of 300MB in 12.70 secs real time.
Transfer rate = 23.62 MB/sec
Transport failures: 956
--------------------------------------
NETLINK with 2MB buffers
_stp_printf on 100 chars = 2332 ns / call system + user time.
_stp_printf of 300MB in 12.19 secs real time.
Transfer rate = 24.61 MB/sec
Transport failures: 0
--------------------------------------
NETLINK with 2MB buffers
_stp_printf on 100 chars = 2396 ns / call system + user time.
_stp_printf of 400MB in 16.51 secs real time.
Transfer rate = 24.23 MB/sec
Transport failures: 0
--------------------------------------
NETLINK with 2MB buffers
_stp_printf on 100 chars = 2397 ns / call system + user time.
_stp_printf of 500MB in 20.65 secs real time.
Transfer rate = 24.21 MB/sec
Transport failures: 0
--------------------------------------
NETLINK with 2MB buffers
_stp_printf on 100 chars = 2397 ns / call system + user time.
_stp_printf of 600MB in 24.78 secs real time.
Transfer rate = 24.21 MB/sec
Transport failures: 0
--------------------------------------
NETLINK with 2MB buffers
_stp_printf on 100 chars = 2492 ns / call system + user time.
_stp_printf of 700MB in 29.58 secs real time.
Transfer rate = 23.66 MB/sec
Transport failures: 26846
--------------------------------------
NETLINK with 4MB buffers
_stp_printf on 100 chars = 2453 ns / call system + user time.
_stp_printf of 700MB in 29.30 secs real time.
Transfer rate = 23.89 MB/sec
Transport failures: 0
--------------------------------------
NETLINK with 4MB buffers
_stp_printf on 100 chars = 2462 ns / call system + user time.
_stp_printf of 800MB in 33.56 secs real time.
Transfer rate = 23.84 MB/sec
Transport failures: 0
--------------------------------------
NETLINK with 4MB buffers
_stp_printf on 100 chars = 2429 ns / call system + user time.
_stp_printf of 900MB in 37.46 secs real time.
Transfer rate = 24.03 MB/sec
Transport failures: 49069
--------------------------------------
NETLINK with 8MB buffers
_stp_printf on 100 chars = 2807 ns / call system + user time.
_stp_printf of 900MB in 40.86 secs real time.
Transfer rate = 22.03 MB/sec
Transport failures: 131036
--------------------------------------
NETLINK with 16MB buffers
_stp_printf on 100 chars = 3476 ns / call system + user time.
_stp_printf of 900MB in 46.88 secs real time.
Transfer rate = 19.20 MB/sec
Transport failures: 170446
--------------------------------------
NETLINK with 32MB buffers
_stp_printf on 100 chars = 2779 ns / call system + user time.
_stp_printf of 900MB in 40.61 secs real time.
Transfer rate = 22.16 MB/sec
Transport failures: 0
--------------------------------------
NETLINK with 32MB buffers
_stp_printf on 100 chars = 2454 ns / call system + user time.
_stp_printf of 1000MB in 41.86 secs real time.
Transfer rate = 23.89 MB/sec
Transport failures: 0
--------------------------------------
RELAYFS with 1MB buffers
_stp_printf on 100 chars = 1029 ns / call system + user time.
_stp_printf of 100MB in 2.76 secs real time.
Transfer rate = 36.23 MB/sec
Transport failures: 0
--------------------------------------
RELAYFS with 1MB buffers
_stp_printf on 100 chars = 1016 ns / call system + user time.
_stp_printf of 200MB in 5.50 secs real time.
Transfer rate = 36.36 MB/sec
Transport failures: 0
--------------------------------------
RELAYFS with 1MB buffers
_stp_printf on 100 chars = 940 ns / call system + user time.
_stp_printf of 300MB in 8.02 secs real time.
Transfer rate = 37.41 MB/sec
Transport failures: 0
--------------------------------------
RELAYFS with 1MB buffers
_stp_printf on 100 chars = 1072 ns / call system + user time.
_stp_printf of 400MB in 11.22 secs real time.
Transfer rate = 35.65 MB/sec
Transport failures: 0
--------------------------------------
RELAYFS with 1MB buffers
_stp_printf on 100 chars = 935 ns / call system + user time.
_stp_printf of 500MB in 13.33 secs real time.
Transfer rate = 37.51 MB/sec
Transport failures: 0
--------------------------------------
RELAYFS with 1MB buffers
_stp_printf on 100 chars = 958 ns / call system + user time.
_stp_printf of 600MB in 16.14 secs real time.
Transfer rate = 37.17 MB/sec
Transport failures: 287
--------------------------------------
RELAYFS with 2MB buffers
_stp_printf on 100 chars = 967 ns / call system + user time.
_stp_printf of 600MB in 16.19 secs real time.
Transfer rate = 37.06 MB/sec
Transport failures: 0
--------------------------------------
RELAYFS with 2MB buffers
_stp_printf on 100 chars = 969 ns / call system + user time.
_stp_printf of 700MB in 18.90 secs real time.
Transfer rate = 37.04 MB/sec
Transport failures: 0
--------------------------------------
RELAYFS with 2MB buffers
_stp_printf on 100 chars = 954 ns / call system + user time.
_stp_printf of 800MB in 21.49 secs real time.
Transfer rate = 37.23 MB/sec
Transport failures: 0
--------------------------------------
RELAYFS with 2MB buffers
_stp_printf on 100 chars = 968 ns / call system + user time.
_stp_printf of 900MB in 24.30 secs real time.
Transfer rate = 37.04 MB/sec
Transport failures: 0
--------------------------------------
RELAYFS with 2MB buffers
_stp_printf on 100 chars = 1030 ns / call system + user time.
_stp_printf of 1000MB in 27.62 secs real time.
Transfer rate = 36.21 MB/sec
Transport failures: 78164
--------------------------------------
RELAYFS with 4MB buffers
_stp_printf on 100 chars = 962 ns / call system + user time.
_stp_printf of 1000MB in 26.94 secs real time.
Transfer rate = 37.12 MB/sec
Transport failures: 58407
--------------------------------------
RELAYFS with 8MB buffers
_stp_printf on 100 chars = 1035 ns / call system + user time.
_stp_printf of 1000MB in 27.68 secs real time.
Transfer rate = 36.13 MB/sec
Transport failures: 285253
--------------------------------------
RELAYFS with 16MB buffers
_stp_printf on 100 chars = 934 ns / call system + user time.
_stp_printf of 1000MB in 26.66 secs real time.
Transfer rate = 37.51 MB/sec
Transport failures: 0
--------------------------------------