How to do gcov and lcov code coverage

[William Cohen <wcohen at redhat dot com>]

I spent this afternoon making some minor modifications to the Makefile 
for systemtap and writing up how to do the code coverage in systemtap. I 
 have checked in my changes into the cvs repository and have attached 
the writeup. This should make it a bit easier for people to see what 
parts of the translator their tests are exercising.

Let me know if you have problems or comments on the writeup.

-Will
Getting Test Coverage Information on SystemTap

Will Cohen
Jan 6, 2005

When developing software it is very useful to determine which sections
of the program the tests actually exercise.  Having this coverage
information can show which sections of the code are being missed by
the tests.  The GCC compiler has options to compile code to produce
this coverage information when the code is executed. The gcov and lcov
tools can analyze this information and produce either text files or
html that the developers can analyze to determine whether the current
tests are sufficient or whether additional tests are needed to
coverage areas of the program that haven't run.

To enable collection of coverage information the following GCC options
are used to compile code:

 -g -fprofile-arcs -ftest-coverage



Translator Code Coverage Tests

Recent versions the SystemTap makefile has two rules to generate test
coverage information: gcov and lcov. These rules will give code
coverage information on the translator tests. With a little additional
effort test coverage for separate testsuite can also be generated.

Assuming you are building the SystemTap translator in /tmp/obj and the
source is in /tmp/src to get a set of text files describing the code
coverage of the translator tests run the following commands:

cd /tmp/obj
../src/configure
make gcov

Note that the gcov runs the clean rule and removes previous compile
and older coverage files. At the end of the make operation the list of
files generated with the code coverage information is listed. For
example the following was printed at the end of the "make gcov":

-rw-rw-r--  1 wcohen wcohen   7265 Jan  6 16:40 buildrun.cxx.gcov
-rw-rw-r--  1 wcohen wcohen  84829 Jan  6 16:40 elaborate.cxx.gcov
-rw-rw-r--  1 wcohen wcohen  21774 Jan  6 16:40 main.cxx.gcov
-rw-rw-r--  1 wcohen wcohen  84896 Jan  6 16:40 parse.cxx.gcov
-rw-rw-r--  1 wcohen wcohen  68680 Jan  6 16:40 staptree.cxx.gcov
-rw-rw-r--  1 wcohen wcohen 151438 Jan  6 16:40 tapsets.cxx.gcov
-rw-rw-r--  1 wcohen wcohen 154841 Jan  6 16:40 translate.cxx.gcov

The ".gcov" files can be examined using a normal editor. The first
column is the number of times that the line is executed. A decimal
number indicates the number of times that the line of code executes, a
'-' means that it is no code is generated for the line, and "#####"
indicates the generated code was not executed at all. To the right of
the first ':' is line number the remainder of the line is the source
code.

The Linux Test Project (LTP) developed some scripts to convert the
gcov data into something viewable on web pages. The scripts can either
be obtained from the LTP page or installed from a RPM listed in the
References section.  The lcov rule in the SystemTap translator
makefile collects the same data at the gcov, but renders it into HTML
files in the "coverage" subdirectory. A web browser such as Firefox
can be use navigate it and examine the data.

The main advantages of the LCOV information over GCOV are:

-LCOV gives summaries listing the amount of code covered in each file.
-LCOV makes it easy to scan for unexecuted code (highlighted in red).
-LCOV results can be posted on webpage and navigated by web browser.


Tests Coverage of Tests Outside the Build Directory

With little additional effort coverage information can be obtained for
the tests outside the translator's build directory, for example your
own tests or the SystemTap integration testsuite. First, you will need
to build the instrumented stap translator in the build directory with
a "make gcov" or "make lcov". The next step is to set the environment
variables correctly so that the instrumented SystemTap translator,
stap, is used. Assuming the elfutils is bundled into SystemTap, the
build directory for the translator is /home/wcohen/systemtap_write/obj,
and the source directory is /home/wcohen/systemtap_write/src:

export STAP_OBJ=/home/wcohen/systemtap_write/obj
export STAP_SRC=/home/wcohen/systemtap_write/src
export LD_LIBRARY_PATH=${STAP_OBJ}/lib-elfutils:${STAP_OBJ}/lib-elfutils/systemtap:$LD_LIBRARY_PATH
export PATH=${STAP_OBJ}:$PATH
export SYSTEMTAP_TAPSET=${STAP_SRC}/tapset
export SYSTEMTAP_RUNTIME=${STAP_SRC}/runtime

Now we are able to generate coverage information for a test or set of
tests using the SystemTap translator, stap, regardless where the tests
are located. For example generating lcov information of the systemtap
testsuite that build the instrumentation modules and runs them on the
system.

	mkdir -p ./coverage
	lcov --directory $STAP_OBJ --zerocounters
	runtest --tool=systemtap
	lcov --directory $STAP_OBJ --capture --output-file ./coverage/stap.info
	genhtml -o ./coverage ./coverage/stap.info

Once finished you can use a  web browser to examing the files in the
coverage directory.


References

The LTP GCOV extension (lcov), Jan
2006. http://ltp.sourceforge.net/coverage/lcov.php

LCOV RPM for FC4, http://people.redhat.com/wcohen/lcov-1.4-1.src.rpm