Re: system and popen fail in case of big application

[Adhemerval Zanella <adhemerval dot zanella at linaro dot org>]

On 09/09/2018 19:44, Zack Weinberg wrote:
> On Sun, Sep 9, 2018 at 6:16 PM Sergey Melnikov
> <melnikov.sergey.v@gmail.com> wrote:
>>
>> Hi all,
>>
>> I'm developing JNI code for a Java application. My code works fine in
>> case of small heap size. Nevertheless, if I set heap size to something
>> reasonably big (like 20Gb on my dev PC with 32Gb of RAM), 'popen' and
>> 'system' glibc calls don't work with ENOMEM errno.
>>
>> So, my question is if does it worth to rewrite existing popen/system
>> calls with posix_spawn as StackOverflow recommends? If so, I may
>> implement and contribute this to glibc.
> 
> What would be worthwhile is to change popen and system so that they
> internally use *vfork*, on systems where vfork is not the same as
> fork. posix_spawn is just an elaborate and inconvenient wrapper around
> vfork, and I don't think we should be using it internally or
> encouraging its use by others.  Also, unlike vfork, it does not
> guarantee to avoid the problem you are experiencing.

We changed posix_spawn to exactly avoid call vfork due its inherent usage
issue with signal handlers (BZ#14750), cancellation state being shared 
with the parent (BZ#10354), and the vfork limitation of possible parent 
clobber due stack spilling (which would require some specific compiler
support to mark the call).  

I can't really see why you think not using posix_spawn, which addresses
all this issues, would be better to either user a problematic interface
(vfork) or replicate the logic all over arch-specific popen implementation
(since clone is Linux specific and it would require us to maintain a
generic implementation which uses fork and another one which uses
CLONE_VFORK with all the required machinery).

In fact think using posix_spawn should require a minimum extra memory
for each call (at least 32kb due BZ#21253 and I still think this change
is quite pessimist due current somewhat broken -fstack-check support).

So I would suggest otherwise: to use implement popen using posix_spawn
as generic code instead of current fork and avoid using the broken
vfork interface (which has been removed from POSIX 2008).