This is the mail archive of the
libc-alpha@sourceware.org
mailing list for the glibc project.
[PATCH v2 1/2] mtrace: record backtrace of memory allocation/deallocation
- From: Kyeong Yoo <kyeong dot yoo at alliedtelesis dot co dot nz>
- To: libc-alpha at sourceware dot org
- Date: Tue, 20 Nov 2018 17:46:42 +1300
- Subject: [PATCH v2 1/2] mtrace: record backtrace of memory allocation/deallocation
When a memory leak is detected by mtrace, it is still difficult
to identify the exact source code causing that issue if the program
uses libraries or common functions.
Now a backtrace of the calling functions for memory allocation and
deallocation is also recorded. This new backtrace is displayed in
a separate line with '#' prefix prior to the original trace record
with '@' prefix.
This extra backtrace is turned off by default and only turned on
if valid number is set to the environment variable MALLOC_TRACE_LEVEL.
Valid value in MALLOC_TRACE_LEVEL is between 1 and 15, which indicates
the maximum number of recent function calls to display.
The companion mtrace.pl script is also updated to parse backtrace info.
And a new command argument "Maps" is added to mtrace.pl, which should
be a copy of the program's maps file ("/proc/PID/maps"). This helps
to interpret addresses from shared libraries.
Also to make mtrace more useful for cross-compilation, two new command
options are added: --prefix PREFIX and --addr2line ADDR2LINE.
Relevant documents are updated. Note typo in a related example for
mtrace() is also corrected.
---
ChangeLog | 19 +++++++
NEWS | 4 +
malloc/mtrace.c | 65 +++++++++++++++++++++++
malloc/mtrace.pl | 145 +++++++++++++++++++++++++++++++++++++++++-----------
manual/memory.texi | 75 ++++++++++++++++++++++++---
5 files changed, 271 insertions(+), 37 deletions(-)
diff --git a/ChangeLog b/ChangeLog
index d340866c43..1ea5f010f0 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,3 +1,22 @@
+2018-11-20 Kyeong Yoo <kyeong.yoo@alliedtelesis.co.nz>
+
+ * malloc/mtrace.c: New environment variable MALLOC_TRACE_LEVEL is
+ defined to enable backtrace output from mtrace(). If the value set
+ in MALLOC_TRACE_LEVEL is between 1 and 15 when mtrace() is called,
+ extra backtrace info is recorded to the mtrace output.
+ * malloc/mtrace.c (tr_backtrace): Print backtrace of the function
+ calls up to the specified call level.
+ * malloc/mtrace.c (tr_freehook, tr_mallochook,tr_reallochook,
+ tr_memalignhook): Record backtrace if needed.
+ * malloc/mtrace.c (mtrace): Parse MALLOC_TRACE_LEVEL environment
+ variable. Valid number is between 1 and 15.
+ * malloc/mtrace.c (muntrace): Reset backtrace level to 0.
+ * malloc/mtrace.pl: Adjust to print backtrace of function calls.
+ * malloc/mtrace.pl: Accept new command argument for "Maps" which makes
+ possible to resolve addresses from shared libraries.
+ * malloc/mtrace.pl: Add "--prefix=PREFIX" and "--addr2line=CMD" options
+ for cross-compilation.
+
2018-11-20 Joseph Myers <joseph@codesourcery.com>
* conform/conformtest.py (ElementTest.run): Use unique identifiers
diff --git a/NEWS b/NEWS
index f488821af1..b025c49bfd 100644
--- a/NEWS
+++ b/NEWS
@@ -35,6 +35,10 @@ Major new features:
different directory. This is a GNU extension and similar to the
Solaris function of the same name.
+* mtrace() records backtraces if MALLOC_TRACE_LEVEL environment variable
+ is set. mtrace Perl script is updated to specify maps file, prefix directory
+ and addr2line program to support cross-compilation case.
+
Deprecated and removed features, and other changes affecting compatibility:
* The glibc.tune tunable namespace has been renamed to glibc.cpu and the
diff --git a/malloc/mtrace.c b/malloc/mtrace.c
index 9064f209ec..5ab6c5d4fe 100644
--- a/malloc/mtrace.c
+++ b/malloc/mtrace.c
@@ -24,6 +24,7 @@
# include <mcheck.h>
# include <libc-lock.h>
#endif
+#include <execinfo.h>
#include <dlfcn.h>
#include <fcntl.h>
@@ -44,8 +45,12 @@
#define TRACE_BUFFER_SIZE 512
+#define MAX_BACKTRACE_LEVEL 15
+
static FILE *mallstream;
static const char mallenv[] = "MALLOC_TRACE";
+static const char mall_level_env[] = "MALLOC_TRACE_LEVEL";
+static int mall_trace_level = 0;
static char *malloc_trace_buffer;
__libc_lock_define_initialized (static, lock);
@@ -61,6 +66,9 @@ static void *(*tr_old_realloc_hook) (void *ptr, size_t size,
static void *(*tr_old_memalign_hook) (size_t __alignment, size_t __size,
const void *);
+static void *tr_mallochook (size_t size, const void *caller);
+static void *tr_reallochook (void *ptr, size_t size, const void *caller);
+
/* This function is called when the block being alloc'd, realloc'd, or
freed has an address matching the variable "mallwatch". In a debugger,
set "mallwatch" to the address of interest, then put a breakpoint on
@@ -108,6 +116,26 @@ tr_where (const void *caller, Dl_info *info)
}
}
+static void
+tr_backtrace (void)
+{
+ void *bt_addrs[MAX_BACKTRACE_LEVEL + 2];
+ size_t bt_size;
+
+ bt_size = __backtrace (bt_addrs, mall_trace_level + 2);
+
+ /* Print backtrace (skip the first two) */
+ if (bt_size > 2)
+ {
+ size_t i;
+
+ fprintf (mallstream, "# %p", bt_addrs[2]);
+ for (i = 3; i < bt_size; i++)
+ fprintf (mallstream, ",%p", bt_addrs[i]);
+ fprintf (mallstream, "\n");
+ }
+}
+
static Dl_info *
lock_and_info (const void *caller, Dl_info *mem)
{
@@ -127,6 +155,18 @@ tr_freehook (void *ptr, const void *caller)
if (ptr == NULL)
return;
+ /* Print backtrace */
+ if (mall_trace_level > 0)
+ {
+ __free_hook = tr_old_free_hook;
+ __malloc_hook = tr_old_malloc_hook;
+ __realloc_hook = tr_old_realloc_hook;
+ tr_backtrace ();
+ __free_hook = tr_freehook;
+ __malloc_hook = tr_mallochook;
+ __realloc_hook = tr_reallochook;
+ }
+
Dl_info mem;
Dl_info *info = lock_and_info (caller, &mem);
tr_where (caller, info);
@@ -160,6 +200,11 @@ tr_mallochook (size_t size, const void *caller)
hdr = (void *) (*tr_old_malloc_hook)(size, caller);
else
hdr = (void *) malloc (size);
+
+ /* Print backtrace */
+ if (mall_trace_level > 0)
+ tr_backtrace ();
+
__malloc_hook = tr_mallochook;
tr_where (caller, info);
@@ -192,6 +237,11 @@ tr_reallochook (void *ptr, size_t size, const void *caller)
hdr = (void *) (*tr_old_realloc_hook)(ptr, size, caller);
else
hdr = (void *) realloc (ptr, size);
+
+ /* Collect backtrace */
+ if (mall_trace_level > 0)
+ tr_backtrace ();
+
__free_hook = tr_freehook;
__malloc_hook = tr_mallochook;
__realloc_hook = tr_reallochook;
@@ -236,6 +286,11 @@ tr_memalignhook (size_t alignment, size_t size, const void *caller)
hdr = (void *) (*tr_old_memalign_hook)(alignment, size, caller);
else
hdr = (void *) memalign (alignment, size);
+
+ /* Collect backtrace */
+ if (mall_trace_level > 0)
+ tr_backtrace ();
+
__memalign_hook = tr_memalignhook;
__malloc_hook = tr_mallochook;
@@ -321,6 +376,15 @@ mtrace (void)
__dso_handle);
}
#endif
+
+ /* Check backtrace level */
+ const char *level_str = getenv (mall_level_env);
+ if (level_str != NULL)
+ {
+ int num = atoi (level_str);
+ if (0 < num && num <= MAX_BACKTRACE_LEVEL)
+ mall_trace_level = num;
+ }
}
else
free (mtb);
@@ -338,6 +402,7 @@ muntrace (void)
file. */
FILE *f = mallstream;
mallstream = NULL;
+ mall_trace_level = 0;
__free_hook = tr_old_free_hook;
__malloc_hook = tr_old_malloc_hook;
__realloc_hook = tr_old_realloc_hook;
diff --git a/malloc/mtrace.pl b/malloc/mtrace.pl
index fe9f546000..22294bf20c 100644
--- a/malloc/mtrace.pl
+++ b/malloc/mtrace.pl
@@ -24,49 +24,56 @@ $VERSION = "@VERSION@";
$PKGVERSION = "@PKGVERSION@";
$REPORT_BUGS_TO = '@REPORT_BUGS_TO@';
$progname = $0;
+$prefix = "";
+$addr2line_cmd = "addr2line";
+
+use Class::Struct;
+use File::Basename;
+use Getopt::Long;
+
+struct Map => {
+ start => '$', # start address
+ end => '$', # end address
+ path => '$', # library path
+};
+
+sub version {
+ print "mtrace $PKGVERSION$VERSION\n";
+ print "Copyright (C) 2018 Free Software Foundation, Inc.\n";
+ print "This is free software; see the source for copying conditions. There is NO\n";
+ print "warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.\n";
+ print "Written by Ulrich Drepper <drepper\@gnu.org>\n";
+ exit 0;
+}
sub usage {
- print "Usage: mtrace [OPTION]... [Binary] MtraceData\n";
- print " --help print this help, then exit\n";
- print " --version print version number, then exit\n";
+ print "Usage: mtrace [OPTION]... [Binary] MtraceData [Maps]\n";
+ print " --help print this help, then exit\n";
+ print " --version print version number, then exit\n";
+ print " --prefix=PREFIX Root directory for library search (cross-compilation)\n";
+ print " --addr2line=CMD Specify addr2line command (cross-compilation)\n";
print "\n";
print "For bug reporting instructions, please see:\n";
print "$REPORT_BUGS_TO.\n";
exit 0;
}
-# We expect two arguments:
+# We expect three arguments:
# #1: the complete path to the binary
# #2: the mtrace data filename
+# #3: the memory map of the process (/proc/PID/maps)
# The usual options are also recognized.
-arglist: while (@ARGV) {
- if ($ARGV[0] eq "--v" || $ARGV[0] eq "--ve" || $ARGV[0] eq "--ver" ||
- $ARGV[0] eq "--vers" || $ARGV[0] eq "--versi" ||
- $ARGV[0] eq "--versio" || $ARGV[0] eq "--version") {
- print "mtrace $PKGVERSION$VERSION\n";
- print "Copyright (C) 2018 Free Software Foundation, Inc.\n";
- print "This is free software; see the source for copying conditions. There is NO\n";
- print "warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.\n";
- print "Written by Ulrich Drepper <drepper\@gnu.org>\n";
-
- exit 0;
- } elsif ($ARGV[0] eq "--h" || $ARGV[0] eq "--he" || $ARGV[0] eq "--hel" ||
- $ARGV[0] eq "--help") {
- &usage;
- } elsif ($ARGV[0] =~ /^-/) {
- print "$progname: unrecognized option `$ARGV[0]'\n";
- print "Try `$progname --help' for more information.\n";
- exit 1;
- } else {
- last arglist;
- }
-}
+GetOptions ("help" => \&usage,
+ "version" => \&version,
+ "prefix=s" => \$prefix, # string
+ "addr2line=s" => \$addr2line_cmd, # string
+ ) or die "Try `$progname --help' for more information.\n";
if ($#ARGV == 0) {
$binary="";
$data=$ARGV[0];
-} elsif ($#ARGV == 1) {
+} elsif ($#ARGV == 1 or $#ARGV == 2) {
$binary=$ARGV[0];
$data=$ARGV[1];
@@ -84,10 +91,71 @@ if ($#ARGV == 0) {
}
close (LOCS);
}
+
+ # Parse maps file
+ if (defined $ARGV[2]) {
+ open(MAPS, "<$ARGV[2]") || die "Cannot open maps file";
+ my $prog_base = basename($prog);
+ while (<MAPS>) {
+ chop;
+ my @cols = split (' ');
+ # Find executable memory segment for library (except the program itself)
+ if ($#cols == 5 and $cols[1] =~ /^..x.$/ and basename($cols[5]) ne $prog_base) {
+ my $path = $prefix . $cols[5];
+ if (-e $path and $cols[0] =~ /^(.*)-(.*)$/) {
+ my $m = Map->new();
+ $m->start(hex($1));
+ $m->end(hex($2));
+ $m->path($path);
+ push(@maps, $m);
+ }
+ }
+ }
+ close (MAPS);
+ }
} else {
die "Wrong number of arguments, run $progname --help for help.";
}
+sub find_address_map {
+ my $addr = pop(@_);
+ foreach my $m (@maps) {
+ return $m if ($m->start <= $addr and $addr <= $m->end);
+ }
+ return undef;
+}
+
+sub print_backtrace {
+ foreach my $addr (split (',', pop(@_))) {
+ printf (" # %-16s %s\n", $addr, &addr2line(hex($addr)));
+ }
+}
+
+sub addr2line {
+ my $addr = pop(@_);
+ return $cache{$addr} if (exists $cache{$addr});
+
+ my $executable;
+ my $searchaddr;
+ my $m = find_address_map($addr);
+ if ($m) {
+ $executable = $m->path;
+ $searchaddr = sprintf ("%#x", $addr - $m->start);
+ } else {
+ $executable = $binary;
+ $searchaddr = sprintf ("%#x", $addr);
+ }
+ if ($executable ne "" && open (ADDR, "$addr2line_cmd -e $executable $searchaddr|")) {
+ my $line = <ADDR>;
+ chomp $line;
+ close (ADDR);
+ $cache{$addr} = $line;
+ return $cache{$addr};
+ }
+ $cache{$addr} = "unknown";
+ return $cache{$addr};
+}
+
sub location {
my $str = pop(@_);
return $str if ($str eq "");
@@ -95,7 +163,7 @@ sub location {
my $addr = $1;
my $fct = $2;
return $cache{$addr} if (exists $cache{$addr});
- if ($binary ne "" && open (ADDR, "addr2line -e $binary $addr|")) {
+ if ($binary ne "" && open (ADDR, "$addr2line_cmd -e $binary $addr|")) {
my $line = <ADDR>;
chomp $line;
close (ADDR);
@@ -116,7 +184,7 @@ sub location {
$searchaddr = $addr;
$prog = $binary;
}
- if ($binary ne "" && open (ADDR, "addr2line -e $prog $searchaddr|")) {
+ if ($binary ne "" && open (ADDR, "$addr2line_cmd -e $prog $searchaddr|")) {
my $line = <ADDR>;
chomp $line;
close (ADDR);
@@ -129,7 +197,7 @@ sub location {
} elsif ($str =~ /^.*[[](0x[^]]*)]$/) {
my $addr = $1;
return $cache{$addr} if (exists $cache{$addr});
- if ($binary ne "" && open (ADDR, "addr2line -e $binary $addr|")) {
+ if ($binary ne "" && open (ADDR, "$addr2line_cmd -e $binary $addr|")) {
my $line = <ADDR>;
chomp $line;
close (ADDR);
@@ -144,6 +212,7 @@ sub location {
}
$nr=0;
+$bt="";
open(DATA, "<$data") || die "Cannot open mtrace data file";
while (<DATA>) {
my @cols = split (' ');
@@ -167,9 +236,11 @@ while (<DATA>) {
printf ("+ %#0@XXX@x Alloc %d duplicate: %s %s\n",
hex($allocaddr), $nr, &location($addrwas{$allocaddr}),
$where);
+ &print_backtrace ($bt) if ($bt ne "");
} elsif ($allocaddr =~ /^0x/) {
$allocated{$allocaddr}=$howmuch;
$addrwas{$allocaddr}=$where;
+ $backtrace{$allocaddr}=$bt if ($bt ne "");
}
last SWITCH;
}
@@ -177,9 +248,11 @@ while (<DATA>) {
if (defined $allocated{$allocaddr}) {
undef $allocated{$allocaddr};
undef $addrwas{$allocaddr};
+ undef $backtrace{$allocaddr};
} else {
printf ("- %#0@XXX@x Free %d was never alloc'd %s\n",
hex($allocaddr), $nr, &location($where));
+ &print_backtrace ($bt) if ($bt ne "");
}
last SWITCH;
}
@@ -187,9 +260,11 @@ while (<DATA>) {
if (defined $allocated{$allocaddr}) {
undef $allocated{$allocaddr};
undef $addrwas{$allocaddr};
+ undef $backtrace{$allocaddr};
} else {
printf ("- %#0@XXX@x Realloc %d was never alloc'd %s\n",
hex($allocaddr), $nr, &location($where));
+ &print_backtrace ($bt) if ($bt ne "");
}
last SWITCH;
}
@@ -198,9 +273,11 @@ while (<DATA>) {
printf ("+ %#0@XXX@x Realloc %d duplicate: %#010x %s %s\n",
hex($allocaddr), $nr, $allocated{$allocaddr},
&location($addrwas{$allocaddr}), &location($where));
+ &print_backtrace ($bt) if ($bt ne "");
} else {
$allocated{$allocaddr}=$howmuch;
$addrwas{$allocaddr}=$where;
+ $backtrace{$allocaddr}=$bt if ($bt ne "");
}
last SWITCH;
}
@@ -213,6 +290,13 @@ while (<DATA>) {
last SWITCH;
}
}
+
+ # Save backtrace info (if any) for the next line
+ if ($cols[0] eq "#") {
+ $bt=$cols[1];
+ } else {
+ $bt="";
+ }
}
close (DATA);
@@ -229,6 +313,7 @@ if ($#addrs >= 0) {
}
printf ("%#0@XXX@x %#8x at %s\n", hex($addr), $allocated{$addr},
&location($addrwas{$addr}));
+ &print_backtrace ($backtrace{$addr}) if (defined $backtrace{$addr});
}
}
}
diff --git a/manual/memory.texi b/manual/memory.texi
index a1435aad1a..25aefc4820 100644
--- a/manual/memory.texi
+++ b/manual/memory.texi
@@ -1709,6 +1709,11 @@ functions are traced and protocolled into the file. There is now of
course a speed penalty for all calls to the traced functions so tracing
should not be enabled during normal use.
+To generate more detailed backtrace information of function calls to
+@code{malloc} etc, set an environment variable named @code{MALLOC_TRACE_LEVEL}
+with a number between 1 and 15. This indicates the maximum number of
+function pointers to generate in the backtrace information.
+
This function is a GNU extension and generally not available on other
systems. The prototype can be found in @file{mcheck.h}.
@end deftypefun
@@ -1830,13 +1835,13 @@ If you take a look at the output it will look similar to this:
@example
= Start
-@ [0x8048209] - 0x8064cc8
-@ [0x8048209] - 0x8064ce0
-@ [0x8048209] - 0x8064cf8
-@ [0x80481eb] + 0x8064c48 0x14
-@ [0x80481eb] + 0x8064c60 0x14
-@ [0x80481eb] + 0x8064c78 0x14
-@ [0x80481eb] + 0x8064c90 0x14
+@@ [0x8048209] - 0x8064cc8
+@@ [0x8048209] - 0x8064ce0
+@@ [0x8048209] - 0x8064cf8
+@@ [0x80481eb] + 0x8064c48 0x14
+@@ [0x80481eb] + 0x8064c60 0x14
+@@ [0x80481eb] + 0x8064c78 0x14
+@@ [0x80481eb] + 0x8064c90 0x14
= End
@end example
@@ -1915,6 +1920,62 @@ from line 33 in the source file @file{/home/drepper/tst-mtrace.c} four
times without freeing this memory before the program terminates.
Whether this is a real problem remains to be investigated.
+@node Interpreting the traces with backtrace
+@subsubsection Interpreting the traces with backtrace
+
+If backtrace information is generated by setting an environment variable
+named @code{MALLOC_TRACE_LEVEL} (for example,
+@code{export MALLOC_TRACE_LEVEL=3}), the output will look similar to this:
+
+@example
+= Start
+# 0x4009b4,0x400866,0x4008f3
+@@ [0x4009b4] - 0x184e280
+# 0x400879,0x4008f3,0x400923
+@@ [0x400879] + 0x184e280 0x14
+= End
+@end example
+
+A list of pointers are printed with '#' mark before each memory
+allocation/deallocation, which indicates the return addresses from the
+corresponding stack frame.
+
+@example
+drepper$ mtrace tst errlog
+- 0x0184e280 Free 19 was never alloc'd /home/drepper/tst.c:51
+ # 0x4009b4 /home/drepper/tst.c:51
+ # 0x400866 /home/drepper/tst.c:13
+ # 0x4008f3 /home/drepper/tst.c:26
+
+Memory not freed:
+-----------------
+ Address Size Caller
+0x0184e280 0x14 at /home/drepper/tst.c:13
+ # 0x400879 /home/drepper/tst.c:13
+ # 0x4008f3 /home/drepper/tst.c:26
+ # 0x400923 /home/drepper/tst.c:36
+@end example
+
+When this mtrace output is processed, it shows better picture of where
+the memory allocation/deallocation was called.
+
+@node Use mtrace for cross-compilation
+@subsubsection Use mtrace for cross-compilation
+
+For cross-compilation, @code{mtrace} can generate meaningful output
+with additional command line options @code{--prefix=PREFIX} and
+@code{--addr2line=CMD} along with @code{/proc/<PID>maps} file copied
+from the target device.
+
+For example, if you have cross-compiled libraries exists in @code{~/staging}
+and compiled for powerpc, you can run command like this:
+
+@example
+drepper$ mtrace --prefix=~/staging \
+ --addr2line=powerpc64-hardfloat-linux-gnu-addr2line \
+ tst errlog maps
+@end example
+
@node Replacing malloc
@subsection Replacing @code{malloc}