sbrk() does not set errno on overflow

[Cyril Hrubis]

Hi!
I've been looking at LTP test failures on x86_64 and 32bit test binaries
and I've found that testcase for sbrk() fails. What the test does is to
call sbrk() repeatedly until it fails and checks that the return value
is -1 and errno set to ENOMEM.

Now this works fine on x86_64 because we hit ENOMEM from kernel brk()
before address space overflows. But with 32bit binary we hit overflow
check in glibc which does not seem to set errno.

Here is what I would do for a fix, I will create a proper patch with ChangeLog
etc. if this change is OK for glibc.

diff --git a/misc/sbrk.c b/misc/sbrk.c
index 87b5472..7ff3921 100644
--- a/misc/sbrk.c
+++ b/misc/sbrk.c
@@ -50,8 +50,10 @@ __sbrk (intptr_t increment)
   if ((increment > 0
        ? ((uintptr_t) oldbrk + (uintptr_t) increment < (uintptr_t) oldbrk)
        : ((uintptr_t) oldbrk < (uintptr_t) -increment))
-      || __brk (oldbrk + increment) < 0)
+      || __brk (oldbrk + increment) < 0) {
+    __set_errno (ENOMEM);
     return (void *) -1;
+  }

   return oldbrk;
 }

-- 
Cyril Hrubis
chrubis@suse.cz