[PATCH][BZ 15089] Fixes malloc_trim always trims for large padding

[Fernando J V da Silva <fernandojvdasilva at gmail dot com>]

Hi all!

I'm starting learning about glibc code and I've began trying to fix
the bug 15089. I've put some comments on Bugzilla and I wrote a patch
(attached) for fixing it, based on the solution proposed by Thiago
Ize' comment in Bugzilla as well
(https://sourceware.org/bugzilla/show_bug.cgi?id=15089).

That solution really fixes the problem, but the systrim() code still
mixes some signed and unsigned operations to calculate the size used
to call sbrk(), which may be dangerous (and in fact caused this bug).
I think the most correct solution would be to refactor this function
for only using unsigned variables but, to be honest, I haven't tried
it as I'm new on that (may be someone could give me some advices about
it?).

I would be glad if someone could have some time to comment it! :-)

Cheers,

Fernando J V da Silva

From c63ae43d50864a648a7797816fdade0b2d107122 Mon Sep 17 00:00:00 2001
From: Fernando J. V. da Silva <fernandojvdasilva@gmail.com>
Date: Wed, 20 Nov 2013 01:56:36 -0200
Subject: [PATCH] Fixes BZ 15089: malloc_trim always trim for large padding.

At systrim(), if the heap top size is bigger than requested pad, then
assign 0 to variable extra, which avoids sbrk() to be called. This
solution is based on Thiago Ize's proposal made on Bugzilla comments.
---
 malloc/malloc.c |    4 +++-
 1 files changed, 3 insertions(+), 1 deletions(-)

diff --git a/malloc/malloc.c b/malloc/malloc.c
index be472b2..09691a1 100644
--- a/malloc/malloc.c
+++ b/malloc/malloc.c
@@ -2712,12 +2712,14 @@ static int systrim(size_t pad, mstate av)
   char* current_brk;     /* address returned by pre-check sbrk call */
   char* new_brk;         /* address returned by post-check sbrk call */
   size_t pagesz;
+  long  top_area;
 
   pagesz = GLRO(dl_pagesize);
   top_size = chunksize(av->top);
 
   /* Release in pagesize units, keeping at least one page */
-  extra = (top_size - pad - MINSIZE - 1) & ~(pagesz - 1);
+  top_area = top_size - MINSIZE - 1;
+  extra = (top_area <= pad)?0:(top_area - pad) & ~(pagesz - 1);
 
   if (extra > 0) {
 
-- 
1.7.1