This is the mail archive of the
systemtap@sourceware.org
mailing list for the systemtap project.
Kprobes stress test prototype
- From: Masami Hiramatsu <mhiramat at redhat dot com>
- To: systemtap-ml <systemtap at sources dot redhat dot com>, Ananth N Mavinakayanahalli <ananth at in dot ibm dot com>, "Frank Ch. Eigler" <fche at redhat dot com>
- Date: Fri, 26 Jun 2009 14:33:04 -0400
- Subject: Kprobes stress test prototype
Hi,
Here is a script of kprobes stress test which uses
kprobes-ftrace engine. Currently, this doesn't use any
benchmarks for making a load.
Any updates, comments, and reports are welcome!
On KVM/x86-64, I've found some symbols must be marked
as __kprobes or blacklisted by using this script.
<always system hang up>
p irq_return
p start_critical_timing
p time_hardirqs_off
p bad_address
p trace_hardirqs_off_thunk
p restore
<not always, but potentially dangerous>
p start_critical_timings
p stop_critical_timing
p trace_hardirqs_on_thunk
p restore_norax
(I've encountered other odd memory corruptions too.
However, I couldn't identify the reason, it might be a bug
in kprobes or kvm.)
Basic usage is very simple.
Of course, this needs kprobes-ftrace engine patchset which
I posted to lkml (http://lkml.org/lkml/2009/6/1/461).
1. Make a working area and put the script;
$ mkdir stest/
$ cp kprobestest stest/
2. Run it.
$ cd stest/
$ ./kprobestest run
3. If kernel gets trouble, reboot it and run again.
$ cd stest/
$ ./kprobestest run
4. Repeat it, until all symbols are tested.
First, the test lists up all symbols in ".text" section, except
".text.kprobes"(*) and breaks it into sub-lists. Initially, each
sub-lists has 64 symbols and are stored into "gray" dir.
The test passes each sub-list to kprobes-ftrace engine and makes
a load(currently, just sleep 2secs).
If a sub-list hasn't made any trouble, it is moved to "white" dir.
If it causes a kernel panic etc., it is moved to "black" dir.
If "gray" dir becomes empty, it collects sub-lists in "black" dir
and breaks it into smaller sub-lists which has 16 symbols.
The test repeats it until each sub-list has just 1 symbol.
(*) You can also pass limited symbols which you are interested in
instead of all symbols. In the step 2,
$ cd stest/
$ ./kprobestest run SYMBOL_LIST_FILE
The SYMBOL_LIST_FILE must be written in the format of kprobes_events.
This means you have to add a "p " prefix for each symbol. So,
$ cat SYMBOL_LIST_FILE
p symbol1
p symbol2
...
Thank you,
--
Masami Hiramatsu
Software Engineer
Hitachi Computer Products (America), Inc.
Software Solutions Division
e-mail: mhiramat@redhat.com
#!/bin/bash
#
# kprobestest: Kprobes stress test tool
# Written by Masami Hiramatsu <mhiramat@redhat.com>
#
# Usage:
# $ kprobestest run [SYMLIST]
# Run stress test. If SYMLIST file is specified, use it as
# an initial symbol list (This is useful for verifying white list
# after diagnosing all symbols).
#
# $ kprobestest cleanup
# Cleanup all lists
if [ "$1" = "cleanup" ]; then
rm -rf symlist kptext black white gray
echo > /debug/tracing/kprobe_events
exit 0
fi
if [ "$1" != "run" ]; then
echo "Usage: kprobestest run|cleanup [SYMLIST]"
exit 0
fi
if [ -f "$2" ]; then
SYMLIST=$2
fi
NR=64
function nomoreblack () {
echo "no more kprobe blacklisted functions"
exit 0
}
function allfound () {
echo "found all blacklisted functions"
cat black/* | tee blacklist
rm black/*
exit 0
}
function preparing () {
local i=0
local n=0
local outfile=symlist
if [ ! -f symlist ]; then
if [ -f $SYMLIST ]; then
cat $SYMLIST > symlist
else
for sym in `sort /proc/kallsyms | egrep '[0-9a-f]+ [Tt] [^[]*$' | cut -d\ -f 3 ` ;do
[ $sym = "__kprobes_text_start" ] && outfile=kptext && continue
[ $sym = "__kprobes_text_end" ] && outfile=symlist && continue
[ $sym = "_etext" ] && break
echo p $sym >> $outfile
done
fi
[ -d black ] || mkdir black
[ -d white ] || mkdir white
else
[ -z "`ls black`" ] && nomoreblack
for b in black/list-* ; do
NR=`wc -l $b| cut -f1 -d\ `
[ $NR -eq 1 ] && allfound
NR=$((NR/4))
[ $NR -eq 0 ] && NR=1
break
done
cat black/* > symlist
rm black/*
fi
cat symlist | while read ln; do
echo "$ln" >> gray/list-$NR-$i
n=$((n+1))
if [ $n -eq $NR ]; then
n=0
i=$((i+1))
fi
done
sync
}
[ -d gray ] || mkdir gray
[ -z "`ls gray/`" ] && preparing
echo > /debug/tracing/kprobe_events
function execload () {
# do some benchmark
sleep 2
}
for list in `cd gray/; ls`; do
echo -n $list
mv gray/$list black/
sync;sync
echo -n "..sync"
cat black/$list > /debug/tracing/kprobe_events
if [ $? -ne 0 ]; then
echo > /debug/tracing/kprobe_events
echo "..error"
sync;sync
else
execload
echo > /debug/tracing/kprobe_events
sync;sync
echo "..done"
mv black/$list white/
sync;sync
fi
done