cp vs native copy performance; suboptimal st_blksize?
Fri Dec 29 14:55:00 GMT 2006
I was investigating a severe performance difference between cp and xcopy
of large files from/to local disk, and found the following:
xcopy uses 64k I/Os
cmd copy uses 64k I/Os
explorer uses 64k I/Os
cp uses 8k I/Os (coreutils-6.7-2)
cp uses 1k I/Os (coreutils-5.3.0-9)
The 8k in cp (coreutils-6.7-2) comes from a built in floor in the copy.c
code not present in a previous cp (coreutils-5.3.0-9).
Typical timings for a 28M file produced via:
dd if=/dev/zero of=28Mfile bs=28M count=1
$ time cp-5-3-0-9 28Mfile dst
$ time cp-6-7-2 28Mfile dst
I'm not quite sure how to account for the real time difference, but I
assure you it is real and repeatable.
$ time xcopy 28Mfile dst
1 File(s) copied
as a sanity check:
$ time dd if=28Mfile of=dst/28Mfile bs=1k
28672+0 records in
28672+0 records out
29360128 bytes (29 MB) copied, 3.64 seconds, 8.1 MB/s
$ time dd if=28Mfile of=dst/28Mfile bs=8k
3584+0 records in
3584+0 records out
29360128 bytes (29 MB) copied, 2.562 seconds, 11.5 MB/s
Likewise missing real time that is repeatable.
$ time dd if=28Mfile of=dst/28Mfile bs=64k
448+0 records in
448+0 records out
29360128 bytes (29 MB) copied, 0.875 seconds, 33.6 MB/s
Ah, much closer!
So, it does indeed appear to be I/O size dependent.
Current cp gets its initial buffer size from the st_blksize field
returned by fstat, which is currently set to 1k. I have found several
references on the web that state 64k is the "native" I/O size for the
disk cache manager. This might also fix the infamous USB transfer
rate issue as discussed here:
Should we change st_blksize to 64k for performance reasons?
The benefit is non-trivial, IMHO.
If so, what scope should this change be (for disk files, raw devices,
mail slots, etc.) (memory devices already use 64k for st_blksize)?
Lead Realtime Software Engineer
VITAL - Visual Simulation Systems
the best safety device in any aircraft is a well-trained crew...
Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
Problem reports: http://cygwin.com/problems.html
More information about the Cygwin