This is the mail archive of the systemtap@sourceware.org mailing list for the systemtap project.


Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]
Other format: [Raw text]

Kprobes stress test prototype


Hi,

Here is a script of kprobes stress test which uses
kprobes-ftrace engine. Currently, this doesn't use any
benchmarks for making a load.
Any updates, comments, and reports are welcome!

On KVM/x86-64, I've found some symbols must be marked
as __kprobes or blacklisted by using this script.

<always system hang up>
p irq_return
p start_critical_timing
p time_hardirqs_off
p bad_address
p trace_hardirqs_off_thunk
p restore

<not always, but potentially dangerous>
p start_critical_timings
p stop_critical_timing
p trace_hardirqs_on_thunk
p restore_norax

(I've encountered other odd memory corruptions too.
 However, I couldn't identify the reason, it might be a bug
 in kprobes or kvm.)

Basic usage is very simple.
Of course, this needs kprobes-ftrace engine patchset which
I posted to lkml (http://lkml.org/lkml/2009/6/1/461).

1. Make a working area and put the script;
 $ mkdir stest/
 $ cp kprobestest stest/

2. Run it.
 $ cd stest/
 $ ./kprobestest run

3. If kernel gets trouble, reboot it and run again.
 $ cd stest/
 $ ./kprobestest run

4. Repeat it, until all symbols are tested.

First, the test lists up all symbols in ".text" section, except
".text.kprobes"(*) and breaks it into sub-lists. Initially, each
sub-lists has 64 symbols and are stored into "gray" dir.
The test passes each sub-list to kprobes-ftrace engine and makes
a load(currently, just sleep 2secs).
If a sub-list hasn't made any trouble, it is moved to "white" dir.
If it causes a kernel panic etc., it is moved to "black" dir.

If "gray" dir becomes empty, it collects sub-lists in "black" dir
and breaks it into smaller sub-lists which has 16 symbols.

The test repeats it until each sub-list has just 1 symbol.

(*) You can also pass limited symbols which you are interested in
    instead of all symbols. In the step 2,
   $ cd stest/
   $ ./kprobestest run SYMBOL_LIST_FILE

  The SYMBOL_LIST_FILE must be written in the format of kprobes_events.
  This means you have to add a "p " prefix for each symbol. So,
  $ cat SYMBOL_LIST_FILE
  p symbol1
  p symbol2
  ...


Thank you,

-- 
Masami Hiramatsu

Software Engineer
Hitachi Computer Products (America), Inc.
Software Solutions Division

e-mail: mhiramat@redhat.com

#!/bin/bash
#
#  kprobestest: Kprobes stress test tool
#  Written by Masami Hiramatsu <mhiramat@redhat.com>
#
#  Usage:
#     $ kprobestest run [SYMLIST]
#        Run stress test. If SYMLIST file is specified, use it as 
#        an initial symbol list (This is useful for verifying white list
#        after diagnosing all symbols).
#
#     $ kprobestest cleanup
#        Cleanup all lists

if [ "$1" = "cleanup" ]; then
  rm -rf symlist kptext black white gray
  echo > /debug/tracing/kprobe_events
  exit 0
fi

if [ "$1" != "run" ]; then
  echo "Usage: kprobestest run|cleanup [SYMLIST]"
  exit 0
fi

if [ -f "$2" ]; then
SYMLIST=$2
fi

NR=64

function nomoreblack () {
  echo "no more kprobe blacklisted functions"
  exit 0
}

function allfound () {
  echo "found all blacklisted functions"
  cat black/* | tee blacklist
  rm black/*
  exit 0
}

function preparing () {
local i=0
local n=0
local outfile=symlist
if [ ! -f symlist ]; then
  if [ -f $SYMLIST ]; then
    cat $SYMLIST > symlist
  else
    for sym in `sort /proc/kallsyms | egrep '[0-9a-f]+ [Tt] [^[]*$' | cut -d\  -f 3 ` ;do
      [ $sym  = "__kprobes_text_start" ] && outfile=kptext && continue
      [ $sym  = "__kprobes_text_end" ] && outfile=symlist && continue
      [ $sym  = "_etext" ] && break
      echo p $sym >> $outfile
    done
  fi
  [ -d black ] || mkdir black
  [ -d white ] || mkdir white
else
  [ -z "`ls black`" ] && nomoreblack
  for b in black/list-* ; do
    NR=`wc -l $b| cut -f1 -d\ `
    [ $NR -eq 1 ] && allfound
    NR=$((NR/4))
    [ $NR -eq 0 ] && NR=1
    break
  done
  cat black/* > symlist
  rm black/*
fi
cat symlist | while read ln; do
  echo "$ln" >> gray/list-$NR-$i
  n=$((n+1))
  if [ $n -eq $NR ]; then
    n=0
    i=$((i+1))
  fi
done
sync
}

[ -d gray ] || mkdir gray
[ -z "`ls gray/`" ] && preparing
echo > /debug/tracing/kprobe_events

function execload () {
  # do some benchmark
  sleep 2
}

for list in `cd gray/; ls`; do
echo -n $list
mv gray/$list black/
sync;sync
echo -n "..sync"
cat black/$list > /debug/tracing/kprobe_events
if [ $? -ne 0 ]; then
  echo > /debug/tracing/kprobe_events
  echo "..error"
  sync;sync
else
  execload
  echo > /debug/tracing/kprobe_events
  sync;sync
  echo "..done"
  mv black/$list white/
  sync;sync
fi
done


Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]