slow handling of large sets of files?
Wed Feb 2 01:08:00 GMT 2005
On Tuesday, February 01, 2005 7:53 PM [EST], Brian Dessent wrote:
> Ken Sheldon wrote:
>> Not only Cygwin apps incur this large performance penalty.
>> Something similar happens with the cmd.exe prompt command "DIR",
>> with the windows file explorer, or with IIS (FTP server). This
>> only seems to happen in the directory structures created by my
>> CygWin scripts (using apps: tar, wget, cp)
> If I had to take a wild guess I'd say that's because when a normal
> win32 app creates a file it usually inherits its ACL from the parent
> directory, and presumably NTFS is tuned for this in some way so that
> checking thousands of such files that all inherit ACLs is fast.
> However, when cygwin creates a file or dir it usually sets the
> permissions explicitly to match what you would expect on a POSIX
> system (i.e. 644 or 755.) Try creating your files/folders with
> "nontsec" set and see if the cygwin-created trees are as slow as
> native-created ones.
NTFS in general just isn't very fast when you really start using some
of the advanced features. Its just not tuned very well for large
amounts of files. A perfect example of this is when you need to
delete alot of files and directories all at once - it takes an awfully
long time to do when compared to ext2/3 or reiserfs, and I've noticed
that sometimes the system doesn't properly clean up the directory
indexes, leaving stale ACL entries and such (run chkdsk, boot, do a
bunch of filesystem operations such as deleting files, then run chkdsk
again, and most likely, it will report even more errors).
MFT fragmentation seems to compound already existing problems.
The Summit Open Source Development Group
http://www.sosdg.org / http://www.ahbl.org
Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
Problem reports: http://cygwin.com/problems.html
More information about the Cygwin