This is the mail archive of the cygwin mailing list for the Cygwin project.

Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]
Other format: [Raw text]

Re: Excessive thrashing when popen()ing after a large malloc()

Hash: SHA1

According to Loic Grenie on 11/14/2006 4:12 AM:
>    The subjects says it all: when a process has a large memory space,
>   a popen() triggers a long disk thrashing. The result can clearly be
>   seen iwth the allegated cygtharsh program (running with 1GB of memory,
>   change the size of the malloc to roughly half your memory size):

Indeed - the fork/exec model of the current popen implementation is harsh
on large memory spaces, compared to your proposed spawn model.  While the
idea may have merit, you have some work to do before a patch can be accepted.

>   to something similar to (warning: untested, needs a char *cmd; at the
>   beggining):

Hint - popen is implemented in newlib, not cygwin, so you are posting to
the wrong list.  Unless you provide a 'diff -u' patch, it is very
difficult to see your changes in context.  And admitting that your changes
are untested is not a good sign for getting it approved.

>   if (cmd = malloc(strlen(program)+64)) == NULL) {
>     (void)close(pdes[1]);
>     (void)close(pdes[2]);
>     free(cur);
>     return (NULL);
>   }
>   if (*type == 'r') {
>     if (pdes[1] != STDOUT_FILENO)
>       sprintf(cmd, "sh -c '%s' >&%d %d>&-", program, pdes[1], pdes[1]);
>     else
>       sprintf(cmd, "sh -c '%s'", program);

This mishandles a program that contains embedded '.

>     if (pdes[0] != STDOUT_FILENO)
>       sprintf(cmd+strlen(cmd), " %d>&-", pdes[0]);

Won't work if cmd already contains redirections.  If you intend for your
redirection to occur first, you will have to do some more work.

>   }
>   else {
>     if (pdes[0] != STDIN_FILENO)
>       sprintf(cmd, "sh -c '%s' <&%d %d>&- %d>&-", program,
>               pdes[0], pdes[0], pdes[1]);

I'm not sure your redirections are correct here.

>     else
>       sprintf(cmd, "sh -c '%s' %d>&-", program, pdes[1])
>   }
>   pid = spawnl(_P_NOWAIT, _PATH_BSHELL, "sh", "-c", cmd, NULL);

Why are you going through two levels of sh?  That seems like a waste to
me; the whole idea of using spawn is to avoid a fork(), but when you
invoke "sh" "-c" "sh -c 'cmd'", you are right back to a fork.  True, the
new invocation of sh uses less memory than the 1 GB process that invoked
popen, so less thrashing will occur, but your whole approach seems
fundamentally flawed if you are trying to use spawn to avoid a fork.

- --
Life is short - so eat dessert first!

Eric Blake   
Version: GnuPG v1.4.5 (Cygwin)
Comment: Public key at
Comment: Using GnuPG with Mozilla -


Unsubscribe info:
Problem reports:

Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]