Cygwin Filesystem Performance degradation 1.7.5 vs 1.7.7, and methods for improving performance

[Yoni Londner]

Hi,
I did a performnace comparison with latest version from CVS, on WinXP.

lstat(1.7.5 unpatched) 3587 files stat() 490.2ms, per file: 0.1367ms
lstat(1.7.5 patched) 3585 files stat() 78.12ms, per file: 0.02179ms
lstat(1.7.7 unpatched) 3588 files stat() 3570ms, per file: 0.9951ms
lstat(1.7.7 patched) 3588 files stat() 3374ms, per file: 0.9404ms
lstat(1.7.8 unpatched)3584 files stat() 3360ms, per file: 0.9376ms

As you can see, there is no performance improvement.

Yoni.


On 4/10/2010 12:23 PM, Corinna Vinschen wrote:
> On Oct  3 23:33, Derry Shribman wrote:
>>> I'm missing a comparison using the latest Cygwin from CVS.  That's
>>> much more interesting than 1.7.5.
>>
>> All the comparisons where done BOTH on 1.7.5 AND on 1.7.7!
>> Here are the 1.7.7 results again:
>
> I saw the numbers for 1.7.7 but I was talking about current CVS.  Due to
> Yoni's mail to cygwin-patches I already changed quite a few things.  A
> comparison should take the latest incarnation of the code into account
> since, obviously, development is going into this direction.
>
>> Could you compile and run it and report the results on your PCs?
>
> Compile the latest Cygwin from CVS and do the comparison on the machines
> you already used.  What sense does it make to add YA machine with
> different hardware?  Especially since all my Windows machines except for
> a single netbook are running in VMs.
>
>>> You also never replied to my mail describing the suffix problem when
>>> using the NtQueryDirectoryFile function:
>>> http://cygwin.com/ml/cygwin-patches/2010-q3/msg00073.html
>>
>> Yoni was busy with preparing the stat performance test program, so I
>> will answer this one:
>>
>> Calling NtQueryDirectoryFile() is implemented in the kernel by
>> IRP_MJ_DIRECTORY_CONTROL, and RestartScan is flags |=
>> SL_RESTART_SCAN, ReturnSingleEntry is flags |=
>> SL_RETURN_SINGLE_ENTRY.
>>
>> What is missing is being able to set flags |= SL_INDEX_SPECIFIED
>> when calling from user-mode.
>
> The WDK description does not say that SL_INDEX_SPECIFIED provides
> this functionality and it also specifies that using the FileIndex
> does not make any sense on NTFS.  Which in turn makes sense, given
> that NTFS keeps directories always in name sorted order.
>
>> So - as you wrote: This means that NtQueryDirectoryFile() should be
>> called with a 'fresh' handle every time.
>>
>> Thats why in the performance test application Yoni wrote, a new
>> handle was called for every NtQueryDirectoryFile() - so that this
>> was taken into account.
>
> That's not really the same.  The test application does not take .lnk and
> .exe suffixes into account.  It just stats what it already got from a
> former readdir.  My private test within the Cygwin code itself was not
> as positive and in fact slowed down `ls -l' since now there was YA
> handle to open.  But anyway.  Maybe the method using a filter with
> wildcards is a winner.
>
>> We have NOD32 anti-virus installed on all XP PCs at our work place.
>> So that may probably be one of the reasons for the XP slowdown. Can
>> you run the test program on an XP at your place and see how results
>> differ?
>
> Well, I can try, but it's a VM, too.
>
>>> However, it's easy to make a speed comparsion which ignores the bunch of
>>> problems Cygwin is fighting against.  What about the problems of various
>>> filesystems, for instance?
>>
>> Cygwin anyway today caches the volume type. So it can know whether
>> it is NTFS/FAT without a system call.
>
> No, not really.  You can make this point for the drive which hosts the
> systemroot, but every other drive (or junction) can change at any given
> time.
>
>>> This requires at least one open handle to the volume to be sure to have
>>> the right filesystem type.  So, whatever you do, you need to open a
>>> handle first.  Either to the parent dir or to the file.  This is kind of
>>> a chicken-egg problem.
>>
>> This is exactly what Yoni cached in the patch he submitted to cygwin-patches!
>
> I didn't look into the patch for the well-known reason.  The description
> in Yoni's mail implies that the caching is based on the path, just like
> in 1.5.x.  That's correct only in an environment which you control
> yourself, but it's not necessarily correct for any other environment.
> That's why I added a new caching algorithm to current CVS which is based
> on the results of a single FileFsVolumeInformation result.  This still
> requires an open handle, either of the file itself, or of it's parent
> dir.
>
>
> Corinna
>