An easy way to reproduce is: $ echo | grep -E "(.*)\1{4}+" Segmentation fault (core dumped) $ This bug has been verified to exist in glibc 2.11.1 shipped with Ubuntu 10.04, as well as the latest version from git repository. It may have security implications as shown in the description of CVE-2010-4051 and CVE-2010-4052.
This is not really a vulnerability in glibc; in various forms, it is common to pretty much any regular expression engine. In general, applications should not pass to regcomp regular expressions coming from untrusted sources. The glibc implementations ensures that "good" regular expressions, in particular not including very high repetition counts or backreferences, do not cause anomalous stack usage in either regcomp or regexec. This is usually a sufficient guarantee.
Back references are impossible to implement efficiently, but the glibc implementation should at least avoid a stack overflow.
Flagging as security-, per the documented Security Exceptions for regcomp: https://sourceware.org/glibc/wiki/Security%20Exceptions