Re: Boot-time probing with SystemTap

[William Cohen <wcohen at redhat dot com>]

William Cohen wrote:

<... snip ...>
> Using the lessons from bootchart there can be some things done to make
> SystemTap provide information to quickly focus attention to problem
> areas (BZ#2035).  Want systemtap boot up probing as easy to use as
> bootchart.  Key requirements are:
>
> - Simplify the SystemTap boot probe install steps to a simple command line.
> - Have the startup show the SystemTap bootprobe option as grub entry
> - Have method that automatically shuts down the probe:
>     -user defined script/function test function called when probe started
>         -e.g. stop data collection when particular process starts
>     -when script/function returns kill probe

I have written a couple simple-minded scripts, create_boottap and delete_boottap 
to simplify this process. Both of the scripts need to be run as root because 
they modify grub entries.

create_boottap dir_name exit_condition script.stp "options for stap"

creates a directory "dir_name". In that directory it puts the kernel module, an 
initscript and a directory to store output. The exit_condition is shell command 
 to execute, when exit_condition exits, the systemtap script is killed. 
script.stp is the actual script. Anything else on the line is passed as options 
to stap when building compiling the SystemTap script. Below is an example line:

# ~/bin/create_boottap /boottap3 'sleep 100' \
/home/wcohen/systemtap_write/src/examples/iotime.stp

When rebooting the machine there will be an entry with SystemTap base name in 
the grub menu. Select this and the boot up will be done with the systemtap 
script running. Data will be stored in dir_name/data/boottap.log

delete_boottap dir_name

delete_boottap removes the directory holding the instrumentation module and 
initscript. This will also remove the data in dir_name/data.

The lookingfor script is a little script that can be used to look for a 
particular process running indicating that the instrumentation can shut down. 
This looks once every 5 seconds, so it might not catch short running processes.

-Will
#!/bin/sh
# create_boottap dir_name exit_condition script.stp options_for_stap

PATH=/usr/local/bin:/usr/local/sbin:/bin:/sbin:/usr/bin:/usr/sbin

check_error()
{
  if test $1 != 0; then
    echo $2
    exit $1
  fi
}

DIR_NAME=$1
EXIT_CONDITION=$2
SCRIPT=$3
shift 3
STAP_OPTIONS=$@
SCRIPT_BASE=`basename ${SCRIPT} .stp`

#compile systemtap probe
STAP_COMMAND="stap -k -p4 -m  ${SCRIPT_BASE} ${SCRIPT} ${STAP_OPTIONS}"
OUTPUT=$($STAP_COMMAND 2>&1)
check_error $? "problem building the kernel module ${SCRIPT_BASE}"
STAP_DIR=`echo $OUTPUT | tr -d \" | awk '/Keeping temporary directory/ {print $4}'`
SCRIPT_MODULE="${STAP_DIR}/${SCRIPT_BASE}.ko"

#Create directory and subdirectory data

# should just have init.sh, *.ko, and data dir
mkdir $DIR_NAME
check_error $? "unable to create directory ${DIR_NAME} (Already exists or cannot create)"

mkdir $DIR_NAME/data
check_error $? "unable to create directory ${DIR_NAME}/data (Already exists or cannot create)"

#copy over module.ko
BOOTTAP_MODULE="${DIR_NAME}/${SCRIPT_BASE}.ko"
cp ${SCRIPT_MODULE} ${BOOTTAP_MODULE}
chmod +x ${BOOTTAP_MODULE}

#copy inittap.sh
BOOTTAP_INIT="${DIR_NAME}/inittap.sh"

echo "#!/bin/sh

# Use tmpfs to collect data, since root is still readonly
mount -n -t tmpfs -o size=10M none ${DIR_NAME}/data

# run as long until exit condition satisfied
# Start systemtap daemon & probe
/usr/bin/staprun ${BOOTTAP_MODULE} -c \"${EXIT_CONDITION}\" \
-o ${DIR_NAME}/data/initprocs.log &

# Give daemon time to start collecting...
sleep 3

# Hand off to real init
exec /sbin/init
" >${BOOTTAP_INIT}

check_error $? "unable to write ${BOOTTAP_INIT}"
chmod +x ${BOOTTAP_INIT}

#install grub/lilo option (use script name)

BOOTTITLE="BootTap ${SCRIPT_BASE}"

# Add a new grub/lilo entry
GRUBBY=/sbin/grubby
if [ -x $GRUBBY ]; then
	kernel=$($GRUBBY --default-kernel)
	initrd=$($GRUBBY --info=$kernel | sed -n '/^initrd=/{s/^initrd=//;p;q;}')
	[ ! -z $initrd ] && initrd="--initrd=$initrd"
	$GRUBBY --remove-kernel TITLE="${BOOTTITLE}"
	$GRUBBY --copy-default --add-kernel=$kernel $initrd --args="init=${BOOTTAP_INIT}" --title="${BOOTTITLE}"
fi

#clean up stap temp directory
rm -rf $STAP_DIR

#!/bin/sh
# create_boottap dir_name exit_condition script.stp "options for stap"

PATH=/usr/local/bin:/usr/local/sbin:/bin:/sbin:/usr/bin:/usr/sbin

check_error()
{
  if test $1 != 0; then
    echo $2
    exit $1
  fi
}

DIR_NAME=$1
BOOTTAP_INIT="${DIR_NAME}/inittap.sh"
SCRIPT_MODULE=` find ${DIR_NAME} -path "*.ko"`

# check and make sure that directory looks like a BootTap directory
# look for the .ko file, and the inittap.sh

if [ ! -e "${BOOTTAP_INIT}" ]; then
	echo "No initscript ${BOOTTAP_INIT}" >&2
	exit 1
fi

if [ ! -e "${SCRIPT_MODULE}" ]; then
	echo "No kernel module " >&2
	exit 1
fi

SCRIPT_BASE=`basename ${SCRIPT_MODULE} .ko`
BOOTTITLE="BootTap ${SCRIPT_BASE}"

# Remove the grub/lilo entry
GRUBBY=/sbin/grubby
if [ -x $GRUBBY ]; then
	$GRUBBY --remove-kernel TITLE="${BOOTTITLE}"
fi

# Remove the directory
rm -rf ${DIR_NAME}

#!/bin/sh -x
# a simple program to watch for some set of processes to start up
EXIT_PROGRAMS="$@"

while [ -z "$( /sbin/pidof ${EXIT_PROGRAMS} )" ]; do
	sleep 5
done;