RE: [RFC] GDB performance testing infrastructure

["Agovic, Sanimir" <sanimir dot agovic at intel dot com>]

Hello Yao,

I like the overall proposal for a "micro" benchmark suite. Some comments below.

> -----Original Message-----
> From: gdb-patches-owner@sourceware.org [mailto:gdb-patches-owner@sourceware.org] On Behalf
> Of Yao Qi
> Sent: Wednesday, August 14, 2013 03:01 PM
> To: gdb-patches@sourceware.org
> Subject: [RFC] GDB performance testing infrastructure
> 
>   * Remote debugging.  It is slower to read from the remote target, and
>     worse, GDB reads the same memory regions in multiple times, or reads
>     the consecutive memory by multiple packets.
>
Once gdb and gdbserver share most of the target code, the overhead will be
caused by the serial protocol roundtrips. But this will take a while...

>   * Tracepoint.  Tracepoint is designed to be efficient on collecting
>     data in the inferior, so we need performance tests to guarantee that
>     tracepoint is still efficient enough.  Note that we a test
>     `gdb.trace/tspeed.exp', but there are still some rooms to improve.
>
Afaik the tracepoint functionality is quite separated from gdb may be tested
in isolation. Having a generic benchmark framework covering the most parts of
gdb is probably _the_ way to go but I see some room for specialized benchmarks
e.g. for tracepoints with a custom driver. But my knowledge is too vague on
the topic.

>   2. Detect performance regressions.  We collected the performance data
>      of each micro-benchmark, and we need to detect or identify the
>      performance regression by comparing with the previous run.  It is
>      more powerful to associate it with continuous testing.
> 
Something really simple, so simple that one could run it silently with every
make invokation. For a newcomer, it took me some time to get used to make
check e.g. setup, run, and interpret the tests with various settings. Something 
simpler would help to run it more often.

> 
> 2 Known works
> =============
> 
>   * [LNT] It was written for LLVM, but is *designed* to be usable for
>     the performance testing of any software.  It is written in python,
>     well-documented and easy to set up.  LNT spawn the compiler first
>     and then target program, record the time usages of compiler and
>     target program in json format.  No interaction is involved.  The
>     performance data collection in LNT is relatively simple, because it
>     is targeted to compiler.  The performance testing part is done, and
>     the next step is to show the data and detect performance
>     regressions.  LNT does a lot work here.  The performance data in
>     json format can be imported to a database, and shown through [web].
>     The performance regression will be highlighted in red.
> 
>   * [lldb] LLDB has a [performance.py] to measure the speed and memory
>     usage of LLDB.  It captures the internal events, feeds some events
>     and record the time usages.  It handles interactions by consuming
>     debugging events, and take some actions accordingly.  It only
>     collects performance data, doesn't detect performance regressions.
> 
>   * libstdc++-v3 There is directory performance in
>     libstdc++-v3/testsuite/ and a header testsuite_performance.h in
>     testsuite/util/.  Test cases are compiled with the header, and run
>     with some large data set, to calculate the time usage.  It is
>     suitable for performance testing for a library.
> 
I like to add the Machine Interface (MI) to the list, but it is quite rudimentary:

$ gdb -interpreter mi -q debugee
[...]
-enable-timings
^done
(gdb)
-break-insert -f main
^done,bkpt={...},time={wallclock="0.00656",user="0.00000",system="0.00000"}
[...]
(gdb)
-exec-step
^running
*running,thread-id="1"
(gdb)
*stopped,[...],time={wallclock="0.19425",user="0.09700",system="0.04200"}
(gdb)

With -enable-timings[1] enabled, every result record has a time triple
appended, even for async[2] ones. If we come up with a full mi parser
one could run tests w/o timings. A mi result is quite json-ish.

(To be honest I do not know how timings are composed of =D)

In addition there are some tools for plotting benchmark results[3].

[1] http://sourceware.org/gdb/onlinedocs/gdb/GDB_002fMI-Miscellaneous-Commands.html
[2] https://sourceware.org/gdb/onlinedocs/gdb/GDB_002fMI-Async-Records.html
[3] http://speed.pypy.org/

-Sanimir
Intel GmbH
Dornacher Strasse 1
85622 Feldkirchen/Muenchen, Deutschland
Sitz der Gesellschaft: Feldkirchen bei Muenchen
Geschaeftsfuehrer: Christian Lamprechter, Hannes Schwaderer, Douglas Lusk
Registergericht: Muenchen HRB 47456
Ust.-IdNr./VAT Registration No.: DE129385895
Citibank Frankfurt a.M. (BLZ 502 109 00) 600119052