cygrunsrv + sshd + rsync = 20 times too slow -- throttled?
Takashi Yano
takashi.yano@nifty.ne.jp
Fri Aug 27 11:24:40 GMT 2021
On Thu, 26 Aug 2021 18:18:29 -0400
Ken Brown wrote:
> On 8/26/2021 11:56 AM, Ken Brown via Cygwin wrote:
> > On 8/25/2021 5:29 PM, Takashi Yano wrote:
> >> On Wed, 25 Aug 2021 13:52:19 -0400
> >> Ken Brown wrote:
> >>> On 8/25/2021 7:18 AM, Takashi Yano via Cygwin wrote:
> >>>> On Tue, 24 Aug 2021 12:49:52 -0700
> >>>> Chris Roehrig wrote:
> >>>>> I have a network of Windows, Linux and Mac machines and I use rsync to
> >>>>> synchronize various directories between them.
> >>>>>
> >>>>> I'm trying to figure out why my rsync transfers are so slow (<4 MB/s) only
> >>>>> when the remote endpoint is Cygwin rsync over sshd (with both a Linux or
> >>>>> Cygwin rsync client). In all other scenarios, I get the full 100MB/s as
> >>>>> expected from gigabit ethernet. This has been an ongoing problem for me
> >>>>> for a couple of years over several Windows and Cygwin versions, and I'd
> >>>>> like to try to fix it.
> >>>>>
> >>>>> If I run rsync --daemon --no-detach under mintty in the foreground on the
> >>>>> remote Windows endpoint, I get the full 100 MB/s transfers, so it seems
> >>>>> like it has something to do with rsync.exe running in the background under
> >>>>> the cygrunsrv+sshd service (which was installed normally using
> >>>>> ssh-host-config).
> >>>>>
> >>>>> If I do:
> >>>>> pv /dev/zero | ssh $WINHOST "cat > /dev/null"
> >>>>> or even
> >>>>> pv /dev/urandom | ssh $WINHOST md5sum
> >>>>> I also get the full 100 MB/s transfers, so it doesn't look like sshd itself
> >>>>> is being throttled by bandwidth or CPU.
> >>>>>
> >>>>> The machines have less than 15% CPU utilization while transferring, with
> >>>>> each of the 4 cores less than 30%, so it doesn't look to be CPU issue.
> >>>>> In Task Manager, sshd.exe and rsync.exe seem to be running normally using
> >>>>> only few percent CPU, and show Power Throttling=Disabled,
> >>>>> Priority=Normal. Setting their Priority to High doesn't seem to change
> >>>>> things.
> >>>>>
> >>>>> Looking in Resource Monitor on the remote endpoint, the network usage is
> >>>>> pretty much a flat horizontal line at about 18 Mbps (2.5 MB/s), so it sure
> >>>>> looks to me as if rsync is somehow being bandwidth-throttled when run in
> >>>>> the background under cygsshd.
> >>>>>
> >>>>> It's almost as if rsync has an implicit --bwlimit override when it is run
> >>>>> from cygrunsrv+sshd (I've tried --bwlimit=0 on the client which makes no
> >>>>> difference).
> >>>>>
> >>>>>
> >>>>> Any ideas? Not sure where to go from here.
> >>>>
> >>>> In cygwin, just scp is very slow.
> >>>>
> >>>> The transfer speed in my environment is as follows.
> >>>> The tests were done with 100MB of test.dat file.
> >>>>
> >>>> (1-1) From cygwin-PC,
> >>>> [yano@cygwin-PC ~]$ scp test.dat yano@linux-server:.
> >>>> yano@linux-server's password:
> >>>> test.dat 100% 100MB 4.0MB/s 00:24
> >>>> [yano@cygwin-PC ~]$ scp yano@linux-server:test.dat .
> >>>> yano@linux-server's password:
> >>>> test.dat 100% 100MB 8.0MB/s 00:12
> >>>>
> >>>> (1-2) From linux-server,
> >>>> yano@linux-server:~$ scp yano@cygwin-PC:test.dat .
> >>>> yano@cygwin-PC's password:
> >>>> test.dat 100% 100MB 4.0MB/s 00:24
> >>>> yano@linux-server:~$ scp test.dat yano@cygwin-PC:.
> >>>> yano@cygwin-PC's password:
> >>>> test.dat 100% 100MB 4.1MB/s 00:24
> >>>>
> >>>>
> >>>> I looked into this problem, and noticed that this is caused
> >>>> by cygwin pipe implementation. Pipe in cygwin is configured
> >>>> with FILE_FLAG_OVERLAPPED.
> >>>>
> >>>> If the pipe is configured without FILE_FLAG_OVERLAPPED,
> >>>> the transfer speed is much improved as follows.
> >>>>
> >>>>
> >>>> (2-1) From cygwin-PC,
> >>>> [yano@cygwin-PC ~]$ scp test.dat yano@linux-server:.
> >>>> yano@linux-server's password:
> >>>> test.dat 100% 100MB 85.5MB/s 00:01
> >>>> [yano@cygwin-PC ~]$ scp yano@linux-server:test.dat .
> >>>> yano@linux-server's password:
> >>>> test.dat 100% 100MB 69.7MB/s 00:01
> >>>>
> >>>> (2-2) From linux-server,
> >>>> yano@linux-server:~$ scp yano@cygwin-PC:test.dat .
> >>>> yano@cygwin-PC's password:
> >>>> test.dat 100% 100MB 80.1MB/s 00:01
> >>>> yano@linux-server:~$ scp test.dat yano@cygwin-PC:.
> >>>> yano@cygwin-PC's password:
> >>>> test.dat 100% 100MB 57.7MB/s 00:01
> >>>>
> >>>> I am not sure why this happens and how to fix this.
> >>>
> >>> A couple years ago I had an idea for changing the pipe implementation to avoid
> >>> overlapped I/O:
> >>>
> >>> https://cygwin.com/pipermail/cygwin-patches/2019q2/009393.html
> >>> https://cygwin.com/pipermail/cygwin-patches/2019q2/009423.html
> >>>
> >>> I never followed up on it. But if you think it might help with this problem, I
> >>> could dust it off and try to finish it.
> >>
> >> Interesting.
> >>
> >> It will be also helpfull for:
> >> https://cygwin.com/pipermail/cygwin/2021-March/247987.html
> >> which seems to be the same issue with
> >> https://stackoverflow.com/questions/10385424/good-alternatives-to-cygwin-cygwin-doesnt-support-natively-support-win32-app
> >>
> >>
> >> However, I wonder why scp dislikes overlapped I/O.
> >
> > I agree that it would be good to understand this. When I first proposed the
> > change, I was thinking in terms of code simplification. If it also improves
> > performance (which we don't know yet), it becomes a higher priority, but in that
> > case it would be nice to understand why it improves performance.
>
> Hi Takashi,
>
> In case you want to try out my proposed change, I've just rebased the patches to
> the current master and pushed them to a new topic/pipe branch.
Hi Ken,
Thanks much! I tested topic/pipe branch.
[yano@cygwin-PC ~]$ scp test.dat yano@linux-server:.
yano@linux-server's password:
test.dat 100% 100MB 95.9MB/s 00:01
[yano@cygwin-PC ~]$ scp yano@linux-server:test.dat .
yano@linux-server's password:
test.dat 100% 100MB 8.0MB/s 00:12
yano@linux-server:~$ scp yano@cygwin-PC:test.dat .
yano@cygwin-PC's password:
test.dat 100% 100MB 109.7MB/s 00:00
yano@linux-server:~$ scp test.dat yano@cygwin-PC:.
yano@cygwin-PC's password:
test.dat 100% 100MB 31.4MB/s 00:03
As shown above, outgoing transfer-rate has been improved upto near
theoretical limit. However, incoming transfer-rate is not improved
much.
I digged further and found the first patch attached solves the issue
as follows.
[yano@cygwin-PC ~]$ scp yano@linux-server:test.dat .
yano@linux-server's password:
test.dat 100% 100MB 112.8MB/s 00:00
yano@linux-server2:~$ scp test.dat yano@cygwin-PC:.
yano@cygwin-PC's password:
test.dat 100% 100MB 102.5MB/s 00:00
I also tested the case:
> >> https://cygwin.com/pipermail/cygwin/2021-March/247987.html
> >> which seems to be the same issue with
> >> https://stackoverflow.com/questions/10385424/good-alternatives-to-cygwin-cygwin-doesnt-support-natively-support-win32-app
Unfortunately, topic/pipe does not help.
I confirmed that applying the second patch attached, which reverts
to create() rather than nt_create(), and setting CYGWIN=pipe_byte
fixes the problem.
What do you think of this alternative implementation which does
not use nt_create()?
--
Takashi Yano <takashi.yano@nifty.ne.jp>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0001-Cygwin-select-Improve-select-poll-response.patch
Type: application/octet-stream
Size: 2460 bytes
Desc: not available
URL: <https://cygwin.com/pipermail/cygwin/attachments/20210827/9f8a739f/attachment.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0002-Cygwin-pipe-Revert-to-create-rather-than-nt_create.patch
Type: application/octet-stream
Size: 2368 bytes
Desc: not available
URL: <https://cygwin.com/pipermail/cygwin/attachments/20210827/9f8a739f/attachment-0001.obj>
More information about the Cygwin
mailing list