Using vfork() in do_system() instead of fork()

[David Wuertele]

In glibc, sysdeps/posix/system.c:do_system() implements the system()
command with fork() and execve().  This causes a problem for my
embedded product because the fork() copies the process VM doubling the
used memory and tirggering the OOM killer.  This would normally not be a
problem, but my team has decided for various reasons to use
/proc/sys/vm/overcommit_memory == 0.

The application developers want to use system(), despite the fact that
their quite large app already allocates most of the system memory.
They say that as long as the system call doesn't need more than the
remaining free RAM, it should succeed.  I agree.

If glibc implemented do_system() with vfork() instead of fork(), it
would prevent the duplication of the parent process's VM that would
never be used anyway.  Is there any reason to not replace fork() with
vfork()?

Dave