[1.7.0-50] scp progress counter flies through first 175 MB or so

[Corinna Vinschen]

On Jun 25 10:05, Christopher Faylor wrote:
> On Thu, Jun 25, 2009 at 12:10:39PM +0200, Corinna Vinschen wrote:
> >On Jun 24 13:17, Andrew Schulman wrote:
> >> Here's an odd one.
> >> 
> >> Using openssh 5.2p1-2 with Cygwin 1.7.0-50, when I scp any file, the
> >> progress counter appears to show ridiculously fast transfer rates, e.g.
> >> about 50 MB/s over a 750 KB/s connection, for the first 175 MB or so. After
> >> that the counter settles down to normal speed.  Then when the counter
> >> reaches the end, it "hangs" at 100% for the remaining time while the copy
> >> finishes.
> >> 
> >> At first I thought that the copy itself was being corrupted in the first
> >> 175 MB, but I'm no longer able to reproduce that.  I believe now that the
> >> copy is good and it's only the progress counter that's wrong.
> >> 
> >> When I revert to Cygwin 1.7.0-49, this problem doesn't occur.
> >
> >I can reproduce that copying a file via scp from a Windows machine to
> >a Linux box.
> >
> >It looks like the pipes between the local scp and the local ssh are now
> >exchanging the data quicker at the start than the ssh socket can send
> >them to the remote machine.  On my XP machine, scp advances quickly by
> >about 260 Megs (hard to tell, maybe it's exaclty 256 Megs for some
> >reason?), then keeps the advance roughly at that value until scp
> >finished.  At the end scp is just waiting for ssh which still has to
> >send the 256/260 Megs of data.
> >
> >This is really weird, given that Cygwin does not create such a big
> >buffer for the pipe.  Consequentially Task Manager claims that the
> >memory is neither taken by scp, nor by ssh.  Both processes have normal
> >VM sizes < 10 Megs.  Per Task manager the memory is paged Kernel Memory.
> >A strange side effect is that the entire time taken by the data 
> >transmission is longer than with -49, by almost exactly the time it
> >takes to empty the big kernel cache.
> >
> >Puzzeling.
> 
> Is ssh using non-blocking pipes opened for write?  Until a week or two
> ago, Cygwin didn't support those and treated the non-blocking write as a
> blocking write.

scp switches the pipes to non-blocking and then tries to do blocking io
on its own, using the poll() function.  It calls a function called scpio
which basically work like this:

  scpio (io_function, fd, buf, size)
  {
    for (offset = 0; offset < size;) {
      r = io_function (fd, buf + offset, size - offset);
      [...]
      if (r < 0) {
	if (errno == EINTR)
	  continue;
	if (errno == EAGAIN || errno == EWOULDBLOCK) {
	  poll (fd, 1, -1) /* Use poll() for blocking */
	  continue;
	}
	[...]
      }
      offset += r;
    }
  }

Looks like scp now stumbles over the pipe select() implementation.

Btw., scp knows a `-l bandwidth' parameter to allow to specify a maximum
throughput.  If you set this to a value which corresponds to the maximum
throughput of the sending socket in ssh, then you can "optimize" the 
problem away and you're back to the old timings.


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Project Co-Leader          cygwin AT cygwin DOT com
Red Hat

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple