This problem is related to:
http://securityreason.com/achievement_securityalert/89
http://ftp.netbsd.org/pub/NetBSD/security/advisories/NetBSD-SA2010-008.txt.asc
http://www.oracle.com/technetwork/topics/security/cpujan2011-194091.html
http://support.avaya.com/css/P8/documents/100127892
http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2010-4754
http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2010-4755
http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2010-4756
http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2011-0418

and is mildly security relevant.  The glob implementation (I checked git head)
contains some calls to malloc where the argument is calculated in a way that
integer overflow or wraparound might occur, in effect allocating less memory
than intended, and hence writing to unallocated or unrelated memory.  In
particular I believe these calls to be problematic:

          pglob->gl_pathv = (char **) malloc ((pglob->gl_offs + 1)
                                              * sizeof (char *));

  (gl_offs is size_t, the multiplication by 4/8 can introduce a wraparound,
   leading to the malloc to succeed but with less memory allocated than
   intended. this could be replaced with calloc as the resulting memory is
   cleared anyway)

          new_gl_pathv
            = (char **) realloc (pglob->gl_pathv,
                                 (newcount + 1 + 1) * sizeof (char *));

  (same problem as above, but even worse as newcount is declared as int,
   so on overflow anything might happen)

              new_gl_pathv = (char **) realloc (pglob->gl_pathv,
                                                (newcount + 2)
                                                * sizeof (char *));

  (same as above)

With properly constructed patterns using repeated application of braces
such wraparounds can easily be reproduced.