Bug 4630 - assertion failure in do_check_malloc_state caused by the non-continuous case in sysMALLOc()
Summary: assertion failure in do_check_malloc_state caused by the non-continuous case ...
Status: RESOLVED WORKSFORME
Alias: None
Product: glibc
Classification: Unclassified
Component: libc (show other bugs)
Version: unspecified
: P2 normal
Target Milestone: ---
Assignee: Ulrich Drepper
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2007-06-12 14:26 UTC by Philipp Tomsich
Modified: 2014-07-04 16:22 UTC (History)
2 users (show)

See Also:
Host: mipsel-linux-gnu
Target: mipsel-linux-gnu
Build: i686-linux-gnu
Last reconfirmed:
fweimer: security-


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Philipp Tomsich 2007-06-12 14:26:06 UTC
We've hit the assertion "malloc.c:2884: do_check_malloc_state: Assertion `(unsigned long)(av-
>system_mem) <= (unsigned long)(av->max_system_mem)' failed." with glibc-2.6 (with 
MALLOC_DEBUG defined to 1) on Linux 2.6. 

Our application frequently uses sbrk()/mmap() directly and exercises the non-continuous case in 
sYSMALLOc(). In rare cases, do_check_malloc_state() will be called through _int_free() at malloc.c:3333 
with (av->system_mem) > (av->max_system_mem), when av->system_mem is increased in 
sYSMALLOc at malloc.c:3299, malloc.c:3232 and malloc.c:3188. 

I'm including a patch to adjust av->max_system_mem to a consistent malloc state (earlier than at 
malloc.c:3352) if _int_free() will be called. The patch will introduce no overhead if MALLOC_DEBUG is 
not defined.

--- glibc-2.6-vanilla/malloc/malloc.c   2007-05-15 03:51:17.000000000 +0200
+++ glibc-2.6/malloc/malloc.c   2007-06-12 16:05:48.000000000 +0200
@@ -3330,6 +3330,17 @@
 
           /* If possible, release the rest. */
           if (old_size >= MINSIZE) {
+
+#if MALLOC_DEBUG
+           /* Restore a consistent state early to avoid failures with
+              debug builds _int_free() calls sYSTRIm() (and sYSTRIm()
+              asserts on inconistencies) */
+
+           if ((unsigned long)(av->system_mem) > 
+               (unsigned long)(av->max_system_mem))
+             av->max_system_mem = av->system_mem;
+#endif
+
             _int_free(av, chunk2mem(old_top));
           }
 
--- glibc-2.6-vanilla/ChangeLog 2007-05-15 08:49:29.000000000 +0200
+++ glibc-2.6/ChangeLog 2007-06-12 16:04:55.000000000 +0200
@@ -0,0 +1,7 @@
+2007-06-12  Philipp Tomsich  <philipp.tomsich@theobroma-systems.com>
+
+       * malloc/malloc.c (sYSMALLOc): Added an early adjustment
+       (conditional to DEBUG_MALLOC) of av->max_system_mem for the
+       non-continuous case to avoid triggering an assertion in
+       check_malloc_state().
+
Comment 1 Andreas Jaeger 2007-06-16 17:26:05 UTC
Wolfram, could you comment on this?
Comment 2 Wolfram Gloger 2007-06-18 15:00:09 UTC
I just checked glibc-2.6 malloc.c and current ptmalloc2.
The only way I can see that _int_free() calls malloc_check_state()
is through malloc_consolidate() and that only after initializing
state.  So, this seems rather strange.  Could you please post a
complete backtrace from the failed assertion?

I'm not dismissing your claim, but I'd like to understand the
problem completely before patching.

Please also note that compiling glibc with MALLOC_DEBUG costs
an awful lot of performance.  I believe it happened by mistake
recently for a released version, but that should be corrected
by now.
Comment 3 Ulrich Drepper 2007-08-22 06:43:13 UTC
NO reply in 2+ months.  Closing.