Performance optimization in av::fixup - use buffered IO, not mapped file

Corinna Vinschen
Thu Dec 13 10:40:00 GMT 2012

On Dec 13 01:44, Daniel Colascione wrote:
> On 12/13/12 1:09 AM, Corinna Vinschen wrote:
> > cygexec is not such a good solution in the long run.  For 64 bit Cygwin
> > we *have* to know if the process to execute is a 32 or 64 bit process
> > so we can't quite avoid the hook_or_detect_cygwin test.
> Today, I run with /bin mounted cygexec. Will I not be able to do
> that once I have mixed 32- and 64-bit Cygwin binaries in PATH?

Depends on the way "cygexec" is implemented.  At execve time we cannot
pass over internal data (cygheap in the first place) the same way as
today if the parent is 32 and the child is 64 bit or vice versa.  In
these cases we need to use another technique, still to be developed.
Whatever this method, obviously this requires us to know the target CPU
of an executable at execve time.  This requires us *at least* to call
GetBinaryType, which, I suppose, is nothing else but a CreateFile,
ReadFile,Close with a check for the machine type in the IMAGE_NT_HEADERS
file header...  which is pretty much exactly what av:fixup does to
figure out the executable type.  See the extended version in the 64 bit

> > so rewriting this code to use ReadFile's isn't that simple.  The
> > question here is this.  Assuming we change MapViewOfFile to
> > VirtualAlloc
> Why VirtualAlloc? For smaller allocations, alloca should suffice,
> and for pathologically large ones, we could just use the normal
> process heap.

alloca is kind of dangerous since we have a lot of stack pressure.
We could use tmp_pathbuf for the first 64K and malloc for the buffer
in hook_or_detect_cygwin.

> > /ReadFile, will that make things slower?
> Not having benchmarked a ReadFile version of av::fixup, it's hard to
> make a definitive statement. Still, the ReadFile version of my test
> program had no effect on overall execution time. Using ReadFile
> instead of a section view shouldn't affect the total number of IOs
> either (assuming readahead is disabled by FILE_FLAG_RANDOM_ACCESS).
> The performance penalty would be an extra memcpy, which, for the IO
> sizes we're talking about, is noise.
> This approach still makes me nervous. I'd be worried that 1) using
> ReadFile wouldn't actually eliminate the redundant loads in some
> cases, and 2) there might be other hidden performance problems with
> sparse files. Examples of the latter problem might be cache flushes
> caused by programs that mmap sparse data files and antivirus
> programs that mmap all executables prior to execution in order to do
> some kind of scanning.

Valid point.  One has to wonder why sparse files were implemented at
all, if nobody seems to care for them, of course...

(Same with transactional NTFS, btw.  It's the only way to implement
 really POSIX compliant unlink() or rename(), but it's implementation
 is slow, and it's supposed to die again in the near future.  Oh well)

> > After a night's sleep I tend to agree with you, though.  I found an old
> > discussion in the cygwin archives from 2003 which also handled sparse
> > file slowness.  It seems this Windows code hasn't changed a lot since
> > then.
> I saw that thread as well, but I didn't see any mention of sparse
> file caching behavior. The thread didn't seem all that productive
> (or pleasant).

Well, it lead to the 128K test.  Memory was still sparse in 2003 ;)

> > Cygwin is used a lot for development, and it is famed for its slowness.
> > If we can notably speed up normal operation by disabling the automatic
> > sparse handling in lseek/write, I think we should do it.  For those
> > requiring sparseness on NTFS, we can add a "sparse" mount option and
> > run the code in lseek/write only if that mount flag is set.  And
> > for symmetry we should probbaly do the same in ftruncate.
> What about using the automatic sparse handling in lseek/lwrite and
> ftruncate only when the file being operated on is already sparse?

That doesn't make sense.  If the file is already sparse, there's no
reason to set the sparse flag in write or ftruncate again.  Also, if you
set the sparse flag only on already sparse files, you will never be able
to create sparse files.


Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Project Co-Leader          cygwin AT cygwin DOT com
Red Hat

More information about the Cygwin-developers mailing list