system and popen fail in case of big application

[Zack Weinberg]

On Sun, Sep 9, 2018 at 6:16 PM Sergey Melnikov
<melnikov.sergey.v@gmail.com> wrote:
>
> Hi all,
>
> I'm developing JNI code for a Java application. My code works fine in
> case of small heap size. Nevertheless, if I set heap size to something
> reasonably big (like 20Gb on my dev PC with 32Gb of RAM), 'popen' and
> 'system' glibc calls don't work with ENOMEM errno.
>
> So, my question is if does it worth to rewrite existing popen/system
> calls with posix_spawn as StackOverflow recommends? If so, I may
> implement and contribute this to glibc.

What would be worthwhile is to change popen and system so that they
internally use *vfork*, on systems where vfork is not the same as
fork. posix_spawn is just an elaborate and inconvenient wrapper around
vfork, and I don't think we should be using it internally or
encouraging its use by others.  Also, unlike vfork, it does not
guarantee to avoid the problem you are experiencing.

zw