[Bug libc/12232] New: setrlimit interferes with malloc

["bkorb at gnu dot org" <sourceware-bugzilla at sourceware dot org>]

http://sourceware.org/bugzilla/show_bug.cgi?id=12232

           Summary: setrlimit interferes with malloc
           Product: glibc
           Version: unspecified
            Status: NEW
          Severity: critical
          Priority: P2
         Component: libc
        AssignedTo: drepper.fsp@gmail.com
        ReportedBy: bkorb@gnu.org


The following code comes from gnulib.  It often but does
not always fail.  The problem may be in the OS or in
glibc but I cannot tell because _int_malloc() is too
twisty for me to readily follow.

int
main (int argc, char *argv[])
{
  struct rlimit limit;
  int arg;
  int repeat;
  void * memory;
  if (getrlimit (RLIMIT_AS, &limit) < 0)
    return 77;
  if (limit.rlim_max == RLIM_INFINITY || limit.rlim_max > MAX_ALLOC_TOTAL)
    limit.rlim_max = MAX_ALLOC_TOTAL;
  limit.rlim_cur = limit.rlim_max;
  if (setrlimit (RLIMIT_AS, &limit) < 0)
    return 77;
  memory = malloc (0x88);

By adding this line of code before the "getrlimit" call:
  free (malloc (0x88));

the problem is not seen.  So, if you use malloc() before the
setrlimit call, you are fine.  There is an interaction issue
with possible problems on both sides of the OS/user space divide.


$ uname -a;/lib64/libc.so.6 |head -1;cat /etc/SuSE-release
Linux beethoven 2.6.34.7-0.4-desktop #1 SMP PREEMPT 2010-10-07 19:07:51 +0200
x86_64 x86_64 x86_64 GNU/Linux
GNU C Library stable release version 2.11.2 (20100531), by Roland McGrath et
al.
openSUSE 11.3 (x86_64)
VERSION = 11.3

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.