This is the mail archive of the
mailing list for the Cygwin project.
Re: Running 4096 parallel cat processes on remote share results in only 1018 succeeding
- From: Linda Walsh <cygwin at tlinx dot org>
- To: "cygwin at cygwin dot com" <cygwin at cygwin dot com>
- Date: Thu, 09 Oct 2014 20:05:45 -0700
- Subject: Re: Running 4096 parallel cat processes on remote share results in only 1018 succeeding
- Authentication-results: sourceware.org; auth=none
- References: <113885868 dot 183866 dot 1412807725028 dot JavaMail dot yahoo at jws10675 dot mail dot bf1 dot yahoo dot com>
Nathan Fairchild wrote:
When I run a script like so:
cat: /u/pe/env_files/transpath.map: No such file or directory
./run_many.sh: fork: retry: Resource temporarily unavailable
./run_many.sh: fork: Resource temporarily unavailable
$ grep -l PATH out* | wc -l
I think I'm probably hitting the 256 process limit because of the I/O slowdown the network presents? I don't get this issue running on (much faster) local disk.
You are only reading from the net? or are you copying
to the to the net too? I.e. what is your CWD? is out$i.log
on the net or local? I tried it locally and couldn't
reproduce your symptoms.
Your problem is more likely the server hosting the remote file system.
While you can write files locally that fast, the remote server adds
enough "ms"/file delay that it can't keep up. Even processing your
Network requests take cpu time.
/i/fdlims> cat mrun
ulimit -n 3200
for i in $(seq $1)
do exec cat mrun >/tmp/tmp$i.log&
/i/fdlims> ll /tmp/tmp*|wc
4096 28672 191405
I'm not sure how accurate the ulimit command inside
cygwin is... may be accurate, just saying I don't know.
Problem reports: http://cygwin.com/problems.html
Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple