memory consumption, playing with valgrind

[Mark Wielaard <mjw at redhat dot com>]

Hi,

I was playing with valgrind a bit and noticed we were leaking massive
amounts of tokens in the parser. Valgrind showed ~2MB of definitely lost
memory (and another ~1MB of indirectly lost memory) caused by token
allocations that were just dropped on the floor. In general we don't
dispose of datastructures, so it is expected that a lot of memory is
still in use at the end of the program. But since all datastructures are
still "life" there really shouldn't be definitely lost memory to which
there are no pointers at all.

I pushed a simple patch that "swallows" any (well most, the ones that
were obvious) tokens produced but not used in parse.cxx. Which reduced
the amount of lost memory by ~3MB. It doesn't try to do anything fancy,
it just makes sure that if we are dropping a token on the floor
immediately after inspection we clean it up.

commit 731a5359e2f226a0a7fa253265cf59d06c4356bf
Author: Mark Wielaard <mjw@redhat.com>

    parse.cxx swallow tokens we are definitely not using.

Before the patch we had (for "hello world", -p4):

==12545==    definitely lost: 2,470,128 bytes in 51,408 blocks
==12545==    indirectly lost: 14,180,805 bytes in 319,624 blocks

After the patch we have:

==14782==    definitely lost: 18,856 bytes in 220 blocks
==14782==    indirectly lost: 12,432,436 bytes in 264,176 blocks

So there is some more memory to be saved. But nothing that looked really
as simple as just cleaning up the unused tokens. If you are interested
run with valgrind --track-orgin=yes to see where valgrind thinks memory
is being allocated and then dropped (warning, lots of output).

It might be helpful to do some real cleanup of datastructures we don't
use anymore to help valgrind point out which memory really is being
leaked. For example cleaning up the stapfiles from the session after
PASS 2 might be a good idea. But it is somewhat hard to track
"ownership" of all structures, so that might not be trivial. Also our
usage of gcc in PASS 4 uses a lot of memory anyway, more than any other
PASS it seems, so it might not really be that beneficial anyway. But I
was somewhat surprised even a simple 'probe begin { log("Hello,
World!"); exit(); }' script allocates more than 30MB. Which seems
somewhat excessive (this mainly seems to come from the fact that we
parse the complete tapset library files, of which there are now 90+).

Note that when using valgrind it only works up to PASS 4. PASS 5 uses
setuid helpers and valgrind cannot translate/emulate those.

Valgrind didn't show any bad memory usage, so that is good. It did
notice some corner cases in elfutils that used some uninitialized memory
or off-by-one array accesses (but none apparently triggered by normal
systemtap runs). Will fix those next.

Cheers,

Mark