slow handling of large sets of files?

Ken Sheldon Ken_Sheldon@ACM.org
Tue Feb 1 20:23:00 GMT 2005


On Tue, 2005-02-01 at 14:51 -0500, Larry Hall wrote:
> At 02:28 PM 2/1/2005, you wrote:
> >I too have been seeing a problem with very slow file access in large
> >directories.
> >
> >Specifically,
> >
> >On a Cygwin/Win2k box, I have a mirror of an FTP site.  The site has 2.5
> >million files spread between 100 directories.  
> >(20,000 - 30,000 files per directory)  I have previously run this number
> >of files in an NT4 NTFS filesystem without significant performance
> >problems.
> >
> >In this site, operations like these are __VERY__ slow.
> >ls ${some_dir}
> >ls ${some_dir}/${some_path}
> >cp ${some_file} ${some_path}
> >cp -R ${some_path_with _only_a_few_files} ${some_path}
> >
> >
> >If I look at the performance monitor, I can see a queue depth of 1-2 and
> >300-500 disk reads per second.  (That's real.  It's a fast array)  The
> >reads appear to be single-block reads, as the throughput during these
> >events is 1.5 - 3MB/sec.
> >
> >I am beginning to think the disk activity relates to NTFS permission
> >checking, which can be complex under Win2k.
> >
> >I don't know how to debug or tune this.
> >
> >Any ideas?
> 
> 
> Have you looked at the '-x', '-X', and '-E' flags of mount?
> 

Hmm... Thats an interesting guess, but neither -e, -X nor -x seem to
have any measurable effect.  
 
$ /bin/ls -F 
00/ 07/ 14/ 21/ 28/ 35/ 42/ 49/ 56/ 63/ 70/ 77/ 84/ 91/ 98/
01/ 08/ 15/ 22/ 29/ 36/ 43/ 50/ 57/ 64/ 71/ 78/ 85/ 92/ 99/
02/ 09/ 16/ 23/ 30/ 37/ 44/ 51/ 58/ 65/ 72/ 79/ 86/ 93/
03/ 10/ 17/ 24/ 31/ 38/ 45/ 52/ 59/ 66/ 73/ 80/ 87/ 94/
04/ 11/ 18/ 25/ 32/ 39/ 46/ 53/ 60/ 67/ 74/ 81/ 88/ 95/
05/ 12/ 19/ 26/ 33/ 40/ 47/ 54/ 61/ 68/ 75/ 82/ 89/ 96/
06/ 13/ 20/ 27/ 34/ 41/ 48/ 55/ 62/ 69/ 76/ 83/ 90/ 97/
 
$ time /bin/ls 04 | wc -l
29551
 
real    0m53.297s
user    0m0.232s
sys     0m0.733s
 
More information:
Not only Cygwin apps incur this large performance penalty.  Something
similar happens with the cmd.exe prompt command "DIR", with the windows
file explorer, or with IIS (FTP server).  This only seems to happen in
the directory structures created by my CygWin scripts (using apps: tar,
wget, cp)


--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/



More information about the Cygwin mailing list