Performance optimization in av::fixup - use buffered IO, not mapped file

Ryan Johnson ryan.johnson@cs.utoronto.ca
Wed Dec 12 16:57:00 GMT 2012


On 12/12/2012 11:47 AM, Eric Blake wrote:
> On 12/12/2012 08:39 AM, Ryan Johnson wrote:
>
>> Does gcc/ld/whatever know the final file size before the first write?
> No, but does it need to?  posix_fallocate() does not change file
> contents; it merely says that anywhere there was previously a hole must
> now be guaranteed to be backed by disk.  So gcc would write the file as
> usual, and then just before close()ing the fd, do a final
> posix_fallocate(fd, 0, len) with len determined by the final file size.
>
>> You have to posix_fallocate the entire file before any write that might
>> create a hole, because the sparse flag poisons the loader,
> Is there really a flag stuck into the file when it becomes sparse?
>
>> and persists
>> even if all gaps are later filled. For example, if I invoke the
>> following commands:
>>
>> cp --sparse=always $(which emacs-nox) sparse
>> cp --sparse=never $(which emacs-nox) dense
>> for f in sparse dense; do echo $f; time ./$f -Q --batch --eval
>> '(kill-emacs)'; done
>> cp --sparse=never dense sparse
>> for f in sparse dense; do echo $f; time ./$f -Q --batch --eval
>> '(kill-emacs)'; done
>> du dense sparse
> This doesn't point to a flag in the file, so much as cached information
> (the file system is remembering that 'sparse' used to be sparse, even if
> it is no longer sparse).  But your point about a file being cached at
> some point while it is sparse, even if it is later made non-sparse, is
> interesting.
>
>> The relevant output is:
>>> sparse
>>> real    0m1.791s
>>>
>>> dense
>>> real    0m0.606s
>>>
>>> sparse
>>> real    0m3.158s
>>>
>>> dense
>>> real    0m0.081s
>>>
>>> 16728   dense
>>> 16768   sparse
>> Given that we're talking about cygwin-specific patches for emacs and
>> binutils anyway, would it be better to add a cygwin-specific fcntl call
>> that clears the file's sparse flag?
> What flag is there to clear?  Your cp demonstration showed that even
> when we do a byte-for-byte copy of every byte (and the file is
> non-sparse), the file system cache remembers that it used to be sparse.
>   How do we defeat that file system cache?
See [1] and $2.3.59 of [2]. There is a metadata flag associated with the 
file itself, independent of the file's contents. You can even truncate a 
sparse file to zero bytes, then grow it back out with dd, and it will 
remain "sparse."

[1] http://msdn.microsoft.com/en-us/library/aa365564.aspx
[2] 
http://download.microsoft.com/download/9/5/E/95EF66AF-9026-4BB0-A41D-A4F81802D92C/%5BMS-FSCC%5D.pdf

Ryan



More information about the Cygwin-developers mailing list