Question about slow access to file information

[Christian Franke]

Eliot Moss via Cygwin wrote:
> On 1/15/2023 3:38 AM, Christian Franke via Cygwin wrote:
>> Eliot Moss via Cygwin wrote:
>>> I have a separate drive mounted this way:
>>>
>>> d:/ /cygdrive/d ntfs binary,posix=0,user,noacl,auto 0 0
>>>
>>> One thing I use it for is to store backup files.  These tend to be 2 Gb
>>> chunks, and there can be hundreds of them in the backup directory. 
>>> (The drive
>>> is 5Tb.)  The Windows Disk Management tool describes it as NTFS, 
>>> Basic Data
>>> Partition.
>>>
>>> Doing ls (for example) takes a very perceptible numbers of seconds 
>>> (though
>>> whatever takes a long time seems to be cached, at least for a while, 
>>> since a
>>> second ls soon after is fast).
>>
>> The problem is the 'noacl' mount option and the fact that POSIX only 
>> offers the *stat*() functions to retrieve file information. These 
>> functions always need to provide the full file information, even if 
>> only a small subset is needed.
>>
>> To determine the 'x'-permission bits in the 'stat.st_mode' field on a 
>> 'noacl'-mount, Cygwin reads the first bytes of most files (all except 
>> *.exe, *.lnk, *.com). The 'x' bits are set if the file starts with 
>> "#!" (script), ":\n" (?) or "MZ" (Windows executable).
>>
>> On 'noacl' mounts, this behavior could be suppressed by 'exec' or 
>> 'noexec' mount options.
>
> Interesting.  I removed the noacl from /etc/fstab and restarted all 
> Cygwin processes.
> The mount program now shows that drive without noacl.  It still takes 
> surprisingly
> long to ls if I have not done so recently.  The directory contains 
> ~1200 files.

This depends on storage device, sometimes (HDD) on filesystem 
fragmentation and always on 'ls' options. Plain '/bin/ls' without any 
arguments does not call stat(). 'ls -s' or 'ls --color=yes' call stat() 
for each file. 'ls -l' additionally calls getfacl() for each file if on 
an 'acl' mount. The latter is apparently slower than expected, see below.

Here a quick test on a directory with 10000 ~3KB files on a NTFS USB 
drive connected via USB-2 (~28MB/s raw read speed). The first test of 
each mount variant was done immediately after connecting the drive:

$ TIMEFORMAT='%R'

1. mount [-o acl]

$ time ls -l > /dev/null
4.282
$ time ls -l > /dev/null
1.322
$ time ls -s > /dev/null
0.404
$ time ls > /dev/null
0.032


2. mount -o noacl

$ time ls -l > /dev/null
13.452
$ time ls -l > /dev/null
0.789
$ time ls -s > /dev/null
0.764
$ time ls > /dev/null
0.033


3. mount -o noacl,noexec

$ time ls -l > /dev/null
3.215
$ time ls -l > /dev/null
0.368
$ time ls -s > /dev/null
0.355
$ time ls > /dev/null
0.032

-- 
Regards,
Christian