Cygwin Filesystem Performance degradation 1.7.5 vs 1.7.7, and methods for improving performance

[Derry Shribman]

Hi,

 >> - GFA: GetFileAttributes(): This NT API allows getting information
 >
 > You meant NtQueryFullAttributesFile, apparently...

Yep: As you can see the code called the right function 
(NtQueryFullAttributesFile()) - its just the printed text that was incorrect.

 >> on a file by name without having to open a handle to the file. This
 >> misses information such as st_ino and st_nlink, but is added here
 >> for sake of comparison. Theoretically this should have been the
 >> fastest API, but on Win7 it is actually slower that
 >> CreateFile(dir)+QueryDirectoryFile+CloseFile. GFA takes 0.01ms on XP
 >> and 0.16 on Win7 (strange...).
 >
 > Not really.  If you use sysinternal's procmon, you'll see what happens.
 > On XP, the NtQueryFullAttributesFile is a simple FASIO_NETWORK_QUERY_OPEN.
 > On Windows 7, the FASTIO_NETWORK_QUERY_OPEN returns with FAST_IO_DISALLOWED
 > and the OS has to fall back to NtOpenFile/NtQueryInformationFile/NtClose.
 > Don't ask me why this occurs on W7.  Maybe FASTIO_NETWORK_QUERY_OPEN
 > interferes badly with NTFS transactions, I don't know.  I only know that
 > I never saw a FASTIO_NETWORK_QUERY_OPEN succeed on W7 so far.

Your correct.
But thats not exactly what Yoni meant. What Yoni was referring to is that 
assuming NT kernel implemented NtQueryFullAttributesFile() efficiently, it 
should have at been at least as fast as 
NtOpenFile/NtQueryInformationFile/NtClose, since in its worse case it needs to 
exactly was we did in user mode (open+query+close) just doing it all in the 
kernel should have at least saved the system-call time (1 system call instead of 3).

But apparently, as you told, according to procmon: NtQueryFullAttributesFile() 
is not implemented efficiently in the NT kernel.

 > I'm missing a comparison using the latest Cygwin from CVS.  That's
 > much more interesting than 1.7.5.

All the comparisons where done BOTH on 1.7.5 AND on 1.7.7!
Here are the 1.7.7 results again:
XP:
 >>     lstat(1.7.7 unpatched) 3588 files stat() 3570ms, per file: 0.9951ms
 >>     lstat(1.7.7 patched) 3588 files stat() 3374ms, per file: 0.9404ms
Win7:
 >>     lstat(1.7.7 unpatched) 3459 files stat()  777ms, per file: 0.2246ms
 >>     lstat(1.7.7 patched) 3459 files stat()  631ms, per file: 0.1824ms

Yoni and I prepared this performance testing program in a very easy to use 
manner: trivial to compile, and trivial to run. No need for DDK or anything special.

Could you compile and run it and report the results on your PCs?

 > You also never replied to my mail describing the suffix problem when
 > using the NtQueryDirectoryFile function:
 > http://cygwin.com/ml/cygwin-patches/2010-q3/msg00073.html

Yoni was busy with preparing the stat performance test program, so I will answer 
this one:

Calling NtQueryDirectoryFile() is implemented in the kernel by 
IRP_MJ_DIRECTORY_CONTROL, and RestartScan is flags |= SL_RESTART_SCAN, 
ReturnSingleEntry is flags |= SL_RETURN_SINGLE_ENTRY.

What is missing is being able to set flags |= SL_INDEX_SPECIFIED when calling 
from user-mode.

So - as you wrote: This means that NtQueryDirectoryFile() should be called with 
a 'fresh' handle every time.

Thats why in the performance test application Yoni wrote, a new handle was 
called for every NtQueryDirectoryFile() - so that this was taken into account.

 >> - QDF() MUCH MUCH MUCH faster than QIF(35x faster on XP, 4x faster on Win7).
 >
 > Dunno why XP is so slow.  Are you sure there's no BLODA kind of software
 > on your system which is to blame for the difference?  I'm only seldom
 > using XP, given that I test on W7 in the first place, but I don't
 > remember that XP was so slow.  Most of the time XP was the fastest of
 > the compared systems.

We have NOD32 anti-virus installed on all XP PCs at our work place. So that may 
probably be one of the reasons for the XP slowdown. Can you run the test program 
on an XP at your place and see how results differ?

The Win7 tests are with the built-in Microsoft Win7 AV.

 > However, it's easy to make a speed comparsion which ignores the bunch of
 > problems Cygwin is fighting against.  What about the problems of various
 > filesystems, for instance?

Cygwin anyway today caches the volume type. So it can know whether it is 
NTFS/FAT without a system call.

So at the first stage, until validated on different other obscure filesystems, 
it can be enabled just for NTFS/FAT.

Solving the performance problem for NTFS/FAT is solving 99%.

 > This requires at least one open handle to the volume to be sure to have
 > the right filesystem type.  So, whatever you do, you need to open a
 > handle first.  Either to the parent dir or to the file.  This is kind of
 > a chicken-egg problem.

This is exactly what Yoni cached in the patch he submitted to cygwin-patches!

 >> for NTFS and FAT volumes ('white list'). Since these are implemented
 >> in NTFS.SYS and FAT.SYS - which we know are implemented correctly,
 >> there is no problem, and this covers 99% of the FS system calls.
 >
 > I doubt that.  There are *lots* of people out there using Samba or
 > a NAS of whatever kind.

The question is not the percentage of systems with additional file systems. The 
question is the percentage of stat() calls per file-system.

Even people with Samba (the most common additional file system on Windows after 
NTFS/FAT/UDFS) - they run Cygwin which is installed on their C: drive 
(C:/cygwin) - which is nearly 100% of the cases an NTFS volume.

Its very rare (if not just about impossible) that someone will install Cygwin on 
a remote Samba/NAS and 'boot' it from there!

And since Cygwin nearly 100% of the cases 'boots' from an NTFS volume, nearly 
all the stat() calls are on that volume: all the applications are there (PATH 
will point to C:/cygwin/bin and in C:/Windows and other locations on C:), the 
temporary files are put there (C:/cygwin/tmp), conf files are there 
(C:/cygwin/etc) etc etc etc.

So if you sum of the stat() calls on a PC with Samba/NAS drive, nearly all the 
calls will still be to the C:/cygwin directory.

 > Anyway, I'm in the process of creating a symlink_info::xcheck function
 > which uses NtQueryDirectoryFile with a single filter expression with
 > wildcards.  The goal is to implement xstat in the first place.  Like cgf
 > I'm not yet sure it's really such a terribly good idea to allow people
 > to use that function via a $CYGWIN setting.  But that isn't set in stone.

Sounds good!

Derry

On 10/3/2010 9:53 PM, Corinna Vinschen wrote:
> On Oct  3 18:15, Yoni Londner wrote:
>> Hi,
>>
>> Following is a cygwin stat performance testing appilication.
>>
>> It compares Cygwin's stat() with Native NT.
>> [...]
>> - GFA: GetFileAttributes(): This NT API allows getting information
>
> You meant NtQueryFullAttributesFile, apparently...
>
>> on a file by name without having to open a handle to the file. This
>> misses information such as st_ino and st_nlink, but is added here
>> for sake of comparison. Theoretically this should have been the
>> fastest API, but on Win7 it is actually slower that
>> CreateFile(dir)+QueryDirectoryFile+CloseFile. GFA takes 0.01ms on XP
>> and 0.16 on Win7 (strange...).
>
> Not really.  If you use sysinternal's procmon, you'll see what happens.
> On XP, the NtQueryFullAttributesFile is a simple FASIO_NETWORK_QUERY_OPEN.
> On Windows 7, the FASTIO_NETWORK_QUERY_OPEN returns with FAST_IO_DISALLOWED
> and the OS has to fall back to NtOpenFile/NtQueryInformationFile/NtClose.
> Don't ask me why this occurs on W7.  Maybe FASTIO_NETWORK_QUERY_OPEN
> interferes badly with NTFS transactions, I don't know.  I only know that
> I never saw a FASTIO_NETWORK_QUERY_OPEN succeed on W7 so far.
>
>> [...]
>> And here are the results of my tests on /bin ~3500 files:
>>     XP: CYGWIN_NT-5.1 yoni 1.7.5(0.228/5/3) 2010-07-18 14:53 i686 Cygwin
>>     lstat(1.7.5 unpatched) 3587 files stat() 490.2ms, per file: 0.1367ms
>>     lstat(1.7.5 patched) 3585 files stat() 78.12ms, per file: 0.02179ms
>>     lstat(1.7.7 unpatched) 3588 files stat() 3570ms, per file: 0.9951ms
>>     lstat(1.7.7 patched) 3588 files stat() 3374ms, per file: 0.9404ms
>>     GFA 3585 files stat() 38.09ms, per file: 0.01062ms
>>     QDF 3585 files stat() 105.5ms, per file: 0.02942ms
>>     QIF 3585 files stat() 3309ms, per file: 0.9229ms
>>
>>     Win7: CYGWIN_NT-6.1 bush-PC 1.7.5(0.228/5/3) 2010-07-18 14:53
>> i686 Cygwin
>>     lstat(1.7.5 unpatched) 3457 files stat() 934.1ms, per file: 0.2702ms
>>     lstat(1.7.5 patched) 3455 files stat()  634ms, per file: 0.1835ms
>>     lstat(1.7.7 unpatched) 3459 files stat()  777ms, per file: 0.2246ms
>>     lstat(1.7.7 patched) 3459 files stat()  631ms, per file: 0.1824ms
>>     GFA 3455 files stat()  574ms, per file: 0.1661ms
>>     QDF 3455 files stat()  159ms, per file: 0.04602ms
>>     QIF 3455 files stat()  599ms, per file: 0.1734ms
>
> I'm missing a comparison using the latest Cygwin from CVS.  That's
> much more interesting than 1.7.5.
>
> However, it's easy to make a speed comparsion which ignores the bunch of
> problems Cygwin is fighting against.  What about the problems of various
> filesystems, for instance?
>
> You also never replied to my mail describing the suffix problem when
> using the NtQueryDirectoryFile function:
> http://cygwin.com/ml/cygwin-patches/2010-q3/msg00073.html
>
>> - QDF() MUCH MUCH MUCH faster than QIF(35x faster on XP, 4x faster on Win7).
>
> 4 times slower only?  That's not too bad.
>
> Dunno why XP is so slow.  Are you sure there's no BLODA kind of software
> on your system which is to blame for the difference?  I'm only seldom
> using XP, given that I test on W7 in the first place, but I don't
> remember that XP was so slow.  Most of the time XP was the fastest of
> the compared systems.
>
> Well, except for the handling of long pathnames.  There's a bug in NTFS
> in all NT 5.x kernels which results in a quadratic access time to files
> relative to the length of the path.  I reported this bug upstream in
> 2008.  Unfortunately it got the "confirmed, won't fix" stamp, since
> Vista had already been released, and the NT 6.X kernel didn't have this
> problem anymore.
>
>> The only information we 'loose' by using QDF() vs. QIF() is st_nlink.
>
> No, that's incorrect.  You're also losing POSIX permissions since you
> neglected to read the ACL.
>
>> Regaring compatiblity for loadable filesystems on Windows (MVFS
>> etc...): Since cygwin knows the volume information, it can use QDF()
>> for NTFS and FAT volumes.
>
> This requires at least one open handle to the volume to be sure to have
> the right filesystem type.  So, whatever you do, you need to open a
> handle first.  Either to the parent dir or to the file.  This is kind of
> a chicken-egg problem.
> You're also still neglecting permissions, or stuff like the missing
> implementation of FileIdBothDirectoryInformation.
>
>> for NTFS and FAT volumes ('white list'). Since these are implemented
>> in NTFS.SYS and FAT.SYS - which we know are implemented correctly,
>> there is no problem, and this covers 99% of the FS system calls.
>
> I doubt that.  There are *lots* of people out there using Samba or
> a NAS of whatever kind.
>
> Anyway, I'm in the process of creating a symlink_info::xcheck function
> which uses NtQueryDirectoryFile with a single filter expression with
> wildcards.  The goal is to implement xstat in the first place.  Like cgf
> I'm not yet sure it's really such a terribly good idea to allow people
> to use that function via a $CYGWIN setting.  But that isn't set in stone.
>
> However, I can work on this only next week and then I'm unavailable
> until late November so this may take some time.
>
>
> Corinna
>