ls/stat on OneDrive causes download of files

[Jeffrey Altman]

On 3/8/2024 7:52 AM, Thomas Wolff via Cygwin wrote:
>
> Am 08.03.2024 um 11:37 schrieb Corinna Vinschen via Cygwin:
>> Hi Jeffrey,
>>
>> On Mar  6 13:55, Jeffrey Altman via Cygwin wrote:
>>> On 3/6/2024 12:19 PM, Corinna Vinschen via Cygwin wrote:
>>>> We can add an explicit call to
>>>>
>>>>     RtlSetProcessPlaceholderCompatibilityMode 
>>>> (PHCM_EXPOSE_PLACEHOLDERS);
>>>> [...]
>>> Files and directories that are placeholders should have either the
>>> FILE_ATTRIBUTE_RECALL_ON_DATA_ACCESS or 
>>> FILE_ATTRIBUTE_RECALL_ON_OPEN file
>>> attributes set. When these attributes are set, applications and mini 
>>> filters
>>> are advised not to "read" or "open" the files or directories unless 
>>> they
>>> absolutely need to because doing so will cause the placeholder to be
>>> replaced by an object containing the actual data which might take a 
>>> long
>>> time to fetch,
>> Yesterday I stumbled over a certain NtCreateFile flag:
>>
>>    FILE_OPEN_NO_RECALL (0x00400000)
>>
>>      Instructs any filters that perform offline storage or 
>> virtualization
>>      to not recall the contents of the file as a result of this open.
>>
>> MS-CIFS described it like this:
>>
>>    FILE_OPEN_NO_RECALL
>>    0x00400000
>>
>>      In a hierarchical storage management environment, this option
>>      requests that the file SHOULD NOT be recalled from tertiary storage
>>      such as tape. A file recall can take up to several minutes in a
>>      hierarchical storage management environment. The clients can 
>> specify
>>      this option to avoid such delays.
>>
>> This sounds like we could simply add this flag to all NtOpenFile
>> used for path conversion or stat-like calls, without having to care
>> for any file attributes specificially.
>>
>> Does that make sense?
> Sounds good, without even studying the other details...
> I speculate some more handling would still be needed to avoid executable
> detection via magic tags.
>
Agreed.   FILE_OPEN_NO_RECALL has been defined for at least a decade but 
was not documented by Microsoft relatively recently.

Another suggestion would be to try opening the file with 
FILE_READ_ATTRIBUTES instead of GENERIC_READ if the file data is not 
required.  See

https://github.com/microsoft/BuildXL/commit/4fb8e7ce07d243ccd95de0d66da551538a794493

Jeffrey Altman