Possible Bug or limitation in Cygwin 1.7 and Rsync and file number limit
lists@trcintl.com
lists@trcintl.com
Fri Jun 19 18:21:00 GMT 2009
I'm not sure if this is a bug or a limitation that can be worked around with a setting somewhere, but I have found a problem with cygwin 1.7 while using rsync. I have been using rsync and cygwin v1.5 for quite some time. I recently started testing cygwin v1.7 and I ran into a problem with an apparent limit on the number of files cygwin 1.7 along with rsync can process in a directory. First let me say that I don't store this many files in a directory manually but rather we have a program that does.
I have not been able to identify the exact number of files, but it seems to be somewhere over 6,000 in a single directory. To help sort this out, I whipped up the following perl code that creates 9,000 dummy files and puts them in a c:\temp directory. Code is as follows if anyone would like to try and reproduce this:
my $path="c:/temp/";
my $number_of_files=9000;
my $base_file_name="file";
my $count=1;
my $file_name = $path . $base_file_name . $count;
while ($count <= $number_of_files) {
open FILE, ">$file_name" or die $!;
$count++;
$file_name = $path . $base_file_name . $count;
print FILE "asdfasdf";
close(FILE);
}
For those familiar with rsync, here are the options I am using in rsync and by the way, I am currently reproducing this by copying from and to the same workstation, however this same issue has been reproduced on more than one machine. This doesn't appear to be an rsync problem but just the same, I have tried several different options for rsync and none seem to make a difference.
--recursive
--human-readable
--delete
--times
--ignore-errors
--stats
--progress
-v -v
If I then run rsync 3.0.5 using cygwin 1.7, the processor will spike to 100% utilization and it will process several thousand files before it just locks up with the processor still spiked at 100%. No errors are shown, it just hangs. If I then delete the directory I am using rsync to copy to and repeat the same procedure with the same version of rsync only this time using cygwin 1.5.25-15, the processor will again spike to 100% however the copy will finish as expected and everything returns to normal. Again, I have been able to repeat this on more than one computer.
One more interesting thing. If I first using rsync and cygwin v1.5 to copy the files and THEN come back and use rsync and cygwin 1.7, it make it all the way through.
See the attached cygcheck.out as per posting instructions. Note I have also tried this on machines with much stronger hardware with the exact same results. ALso note that the machine never runs out of memory so that doesn't appear to be the problem either.
Is this a bug in cygwin 1.7 or is there some setting that will allow me to work around this limitation?
Thank you.
- Kyle
-------------- next part --------------
Cygwin Configuration Diagnostics
Current System Time: Thu Jun 18 13:39:33 2009
Windows XP Professional Ver 5.1 Build 2600 Service Pack 3
Path: C:\WINDOWS\system32
C:\WINDOWS
C:\WINDOWS\System32\Wbem
Output from C:\cygwin_1.7\bin\id.exe (nontsec)
UID: 10500(administrator) GID: 10513(Domain Users)
0(root) 544(Administrators) 545(Users)
10513(Domain Users)
Output from C:\cygwin_1.7\bin\id.exe (ntsec)
UID: 10500(administrator) GID: 10513(Domain Users)
0(root) 544(Administrators) 545(Users)
10513(Domain Users)
SysDir: C:\WINDOWS\system32
WinDir: C:\WINDOWS
Path = 'C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem'
ALLUSERSPROFILE = 'C:\Documents and Settings\All Users'
APPDATA = 'C:\Documents and Settings\Administrator.TRC\Application Data'
CLIENTNAME = 'Console'
CommonProgramFiles = 'C:\Program Files\Common Files'
COMPUTERNAME = 'TEMP'
ComSpec = 'C:\WINDOWS\system32\cmd.exe'
FP_NO_HOST_CHECK = 'NO'
HOMEDRIVE = 'C:'
HOMEPATH = '\Documents and Settings\Administrator.TRC'
LOGONSERVER = '\\TRCBT1'
NUMBER_OF_PROCESSORS = '1'
OS = 'Windows_NT'
PATHEXT = '.COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH'
PROCESSOR_ARCHITECTURE = 'x86'
PROCESSOR_IDENTIFIER = 'x86 Family 6 Model 5 Stepping 2, GenuineIntel'
PROCESSOR_LEVEL = '6'
PROCESSOR_REVISION = '0502'
ProgramFiles = 'C:\Program Files'
PROMPT = '$P$G'
SESSIONNAME = 'Console'
SystemDrive = 'C:'
SystemRoot = 'C:\WINDOWS'
TEMP = 'C:\DOCUME~1\ADMINI~1.TRC\LOCALS~1\Temp'
TMP = 'C:\DOCUME~1\ADMINI~1.TRC\LOCALS~1\Temp'
USERDNSDOMAIN = 'TRCINTL.COM'
USERDOMAIN = 'TRC'
USERNAME = 'administrator'
USERPROFILE = 'C:\Documents and Settings\Administrator.TRC'
windir = 'C:\WINDOWS'
HKEY_CURRENT_USER\Software\Cygnus Solutions\Cygwin
HKEY_CURRENT_USER\Software\Cygnus Solutions\Cygwin\mounts v2
HKEY_CURRENT_USER\Software\Cygnus Solutions\Cygwin\Program Options
HKEY_CURRENT_USER\Software\Cygwin
HKEY_CURRENT_USER\Software\Cygwin\Program Options
HKEY_CURRENT_USER\Software\Cygwin\setup
HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Explorer\MenuOrder\Start Menu\Programs\Cygwin
(default) = (unsupported type)
HKEY_LOCAL_MACHINE\SOFTWARE\Cygnus Solutions\Cygwin
HKEY_LOCAL_MACHINE\SOFTWARE\Cygnus Solutions\Cygwin\mounts v2
(default) = '/cygdrive'
cygdrive flags = 0x00000022
HKEY_LOCAL_MACHINE\SOFTWARE\Cygnus Solutions\Cygwin\mounts v2\/
(default) = 'C:\cygwin'
flags = 0x0000000a
HKEY_LOCAL_MACHINE\SOFTWARE\Cygnus Solutions\Cygwin\mounts v2\/usr/bin
(default) = 'C:\cygwin/bin'
flags = 0x0000000a
HKEY_LOCAL_MACHINE\SOFTWARE\Cygnus Solutions\Cygwin\mounts v2\/usr/lib
(default) = 'C:\cygwin/lib'
flags = 0x0000000a
HKEY_LOCAL_MACHINE\SOFTWARE\Cygnus Solutions\Cygwin\Program Options
HKEY_LOCAL_MACHINE\SOFTWARE\Cygwin
HKEY_LOCAL_MACHINE\SOFTWARE\Cygwin\Program Options
HKEY_LOCAL_MACHINE\SOFTWARE\Cygwin\setup
(default) = 'C:\cygwin_1.7'
obcaseinsensitive set to 1
a: fd FAT 1Mb 6% CP UN
c: hd NTFS 8048Mb 68% CP CS UN PA FC
d: cd N/A N/A
z: net NTFS 547127Mb 58% CP CS UN PA FC Root
C:\cygwin_1.7 / system binmode
. /cygdrive user binmode,cygdrive,noacl,posix=0
Not Found: awk
Not Found: bash
Not Found: cat
Not Found: cp
Not Found: cpp (good!)
Not Found: crontab
Found: C:\WINDOWS\system32\find.exe
Not Found: gcc
Not Found: gdb
Not Found: grep
Not Found: kill
Not Found: ld
Not Found: ls
Not Found: make
Not Found: mv
Not Found: patch
Not Found: perl
Not Found: rm
Not Found: sed
Not Found: ssh
Not Found: sh
Not Found: tar
Not Found: test
Not Found: vi
Not Found: vim
61k 2009/03/02 .\cygbz2-1.dll - os=4.0 img=1.0 sys=4.0
"cygbz2-1.dll" v0.0 ts=2009/3/1 21:11
40k 2009/03/01 .\cygform-8.dll - os=4.0 img=1.0 sys=4.0
"cygform-8.dll" v0.0 ts=2009/3/1 0:32
41k 2009/03/27 .\cygform-9.dll - os=4.0 img=1.0 sys=4.0
"cygform-9.dll" v0.0 ts=2009/3/27 0:02
42k 2009/03/12 .\cyggcc_s-1.dll - os=4.0 img=1.0 sys=4.0
"cyggcc_s-1.dll" v0.0 ts=2009/3/6 5:54
315k 2009/06/06 .\cyggmp-3.dll - os=4.0 img=1.0 sys=4.0
"cyggmp-3.dll" v0.0 ts=2009/6/6 6:23
24k 2008/11/29 .\cyghistory6.dll - os=4.0 img=1.0 sys=4.0
"cyghistory6.dll" v0.0 ts=2008/11/29 8:30
270k 2009/05/07 .\cygicons-0.dll - os=4.0 img=1.0 sys=4.0
"cygicons-0.dll" v0.0 ts=2009/5/6 19:58
981k 2009/05/30 .\cygiconv-2.dll - os=4.0 img=1.0 sys=4.0
"cygiconv-2.dll" v0.0 ts=2009/5/30 14:05
31k 2005/11/20 .\cygintl-3.dll - os=4.0 img=1.0 sys=4.0
"cygintl-3.dll" v0.0 ts=2005/11/19 20:04
31k 2009/04/03 .\cygintl-8.dll - os=4.0 img=1.0 sys=4.0
"cygintl-8.dll" v0.0 ts=2009/4/2 20:15
5k 2009/06/08 .\cyglsa.dll - os=4.0 img=1.0 sys=4.0
"cyglsa.dll" v0.0 ts=2009/6/8 15:14
9k 2009/06/08 .\cyglsa64.dll - os=4.0 img=0.0 sys=5.2
21k 2009/03/01 .\cygmenu-8.dll - os=4.0 img=1.0 sys=4.0
"cygmenu-8.dll" v0.0 ts=2009/3/1 0:31
21k 2009/03/27 .\cygmenu-9.dll - os=4.0 img=1.0 sys=4.0
"cygmenu-9.dll" v0.0 ts=2009/3/27 0:01
211k 2009/06/06 .\cygmp-3.dll - os=4.0 img=1.0 sys=4.0
"cygmp-3.dll" v0.0 ts=2009/6/6 6:23
66k 2009/03/01 .\cygncurses++-8.dll - os=4.0 img=1.0 sys=4.0
"cygncurses++-8.dll" v0.0 ts=2009/3/1 0:39
334k 2009/03/27 .\cygncurses++-9.dll - os=4.0 img=1.0 sys=4.0
"cygncurses++-9.dll" v0.0 ts=2009/3/27 0:12
237k 2009/03/01 .\cygncurses-8.dll - os=4.0 img=1.0 sys=4.0
"cygncurses-8.dll" v0.0 ts=2009/3/1 0:28
164k 2009/03/27 .\cygncurses-9.dll - os=4.0 img=1.0 sys=4.0
"cygncurses-9.dll" v0.0 ts=2009/3/26 23:58
11k 2009/03/01 .\cygpanel-8.dll - os=4.0 img=1.0 sys=4.0
"cygpanel-8.dll" v0.0 ts=2009/3/1 0:30
11k 2009/03/27 .\cygpanel-9.dll - os=4.0 img=1.0 sys=4.0
"cygpanel-9.dll" v0.0 ts=2009/3/27 0:00
190k 2009/04/17 .\cygpcre-0.dll - os=4.0 img=1.0 sys=4.0
"cygpcre-0.dll" v0.0 ts=2009/4/17 11:07
7k 2009/04/17 .\cygpcreposix-0.dll - os=4.0 img=1.0 sys=4.0
"cygpcreposix-0.dll" v0.0 ts=2009/4/17 11:07
22k 2002/06/09 .\cygpopt-0.dll - os=4.0 img=1.0 sys=4.0
"cygpopt-0.dll" v0.0 ts=2002/6/9 0:45
155k 2008/11/29 .\cygreadline6.dll - os=4.0 img=1.0 sys=4.0
"cygreadline6.dll" v0.0 ts=2008/11/29 8:30
66k 2009/03/27 .\cygtic-9.dll - os=4.0 img=1.0 sys=4.0
"cygtic-9.dll" v0.0 ts=2009/3/26 23:54
64k 2009/03/02 .\cygz.dll - os=4.0 img=1.0 sys=4.0
"cygz.dll" v0.0 ts=2009/3/1 19:33
2376k 2009/06/08 .\cygwin1.dll - os=4.0 img=1.0 sys=4.0
"cygwin1.dll" v0.0 ts=2009/6/8 15:13
Cygwin DLL version info:
DLL version: 1.7.0
DLL epoch: 19
DLL old termios: 5
DLL malloc env: 28
Cygwin conv: 181
API major: 0
API minor: 210
Shared data: 5
DLL identifier: cygwin1
Mount registry: 3
Cygwin registry name: Cygwin
Program options name: Program Options
Cygdrive default prefix:
Build date: Mon Jun 8 22:13:39 CEST 2009
Shared id: cygwin1S5
61k 2009/03/02 C:\cygwin_1.7\bin\cygbz2-1.dll - os=4.0 img=1.0 sys=4.0
"cygbz2-1.dll" v0.0 ts=2009/3/1 21:11
40k 2009/03/01 C:\cygwin_1.7\bin\cygform-8.dll - os=4.0 img=1.0 sys=4.0
"cygform-8.dll" v0.0 ts=2009/3/1 0:32
41k 2009/03/27 C:\cygwin_1.7\bin\cygform-9.dll - os=4.0 img=1.0 sys=4.0
"cygform-9.dll" v0.0 ts=2009/3/27 0:02
42k 2009/03/12 C:\cygwin_1.7\bin\cyggcc_s-1.dll - os=4.0 img=1.0 sys=4.0
"cyggcc_s-1.dll" v0.0 ts=2009/3/6 5:54
315k 2009/06/06 C:\cygwin_1.7\bin\cyggmp-3.dll - os=4.0 img=1.0 sys=4.0
"cyggmp-3.dll" v0.0 ts=2009/6/6 6:23
24k 2008/11/29 C:\cygwin_1.7\bin\cyghistory6.dll - os=4.0 img=1.0 sys=4.0
"cyghistory6.dll" v0.0 ts=2008/11/29 8:30
270k 2009/05/07 C:\cygwin_1.7\bin\cygicons-0.dll - os=4.0 img=1.0 sys=4.0
"cygicons-0.dll" v0.0 ts=2009/5/6 19:58
981k 2009/05/30 C:\cygwin_1.7\bin\cygiconv-2.dll - os=4.0 img=1.0 sys=4.0
"cygiconv-2.dll" v0.0 ts=2009/5/30 14:05
31k 2005/11/20 C:\cygwin_1.7\bin\cygintl-3.dll - os=4.0 img=1.0 sys=4.0
"cygintl-3.dll" v0.0 ts=2005/11/19 20:04
31k 2009/04/03 C:\cygwin_1.7\bin\cygintl-8.dll - os=4.0 img=1.0 sys=4.0
"cygintl-8.dll" v0.0 ts=2009/4/2 20:15
5k 2009/06/08 C:\cygwin_1.7\bin\cyglsa.dll - os=4.0 img=1.0 sys=4.0
"cyglsa.dll" v0.0 ts=2009/6/8 15:14
9k 2009/06/08 C:\cygwin_1.7\bin\cyglsa64.dll - os=4.0 img=0.0 sys=5.2
21k 2009/03/01 C:\cygwin_1.7\bin\cygmenu-8.dll - os=4.0 img=1.0 sys=4.0
"cygmenu-8.dll" v0.0 ts=2009/3/1 0:31
21k 2009/03/27 C:\cygwin_1.7\bin\cygmenu-9.dll - os=4.0 img=1.0 sys=4.0
"cygmenu-9.dll" v0.0 ts=2009/3/27 0:01
211k 2009/06/06 C:\cygwin_1.7\bin\cygmp-3.dll - os=4.0 img=1.0 sys=4.0
"cygmp-3.dll" v0.0 ts=2009/6/6 6:23
66k 2009/03/01 C:\cygwin_1.7\bin\cygncurses++-8.dll - os=4.0 img=1.0 sys=4.0
"cygncurses++-8.dll" v0.0 ts=2009/3/1 0:39
334k 2009/03/27 C:\cygwin_1.7\bin\cygncurses++-9.dll - os=4.0 img=1.0 sys=4.0
"cygncurses++-9.dll" v0.0 ts=2009/3/27 0:12
237k 2009/03/01 C:\cygwin_1.7\bin\cygncurses-8.dll - os=4.0 img=1.0 sys=4.0
"cygncurses-8.dll" v0.0 ts=2009/3/1 0:28
164k 2009/03/27 C:\cygwin_1.7\bin\cygncurses-9.dll - os=4.0 img=1.0 sys=4.0
"cygncurses-9.dll" v0.0 ts=2009/3/26 23:58
11k 2009/03/01 C:\cygwin_1.7\bin\cygpanel-8.dll - os=4.0 img=1.0 sys=4.0
"cygpanel-8.dll" v0.0 ts=2009/3/1 0:30
11k 2009/03/27 C:\cygwin_1.7\bin\cygpanel-9.dll - os=4.0 img=1.0 sys=4.0
"cygpanel-9.dll" v0.0 ts=2009/3/27 0:00
190k 2009/04/17 C:\cygwin_1.7\bin\cygpcre-0.dll - os=4.0 img=1.0 sys=4.0
"cygpcre-0.dll" v0.0 ts=2009/4/17 11:07
7k 2009/04/17 C:\cygwin_1.7\bin\cygpcreposix-0.dll - os=4.0 img=1.0 sys=4.0
"cygpcreposix-0.dll" v0.0 ts=2009/4/17 11:07
22k 2002/06/09 C:\cygwin_1.7\bin\cygpopt-0.dll - os=4.0 img=1.0 sys=4.0
"cygpopt-0.dll" v0.0 ts=2002/6/9 0:45
155k 2008/11/29 C:\cygwin_1.7\bin\cygreadline6.dll - os=4.0 img=1.0 sys=4.0
"cygreadline6.dll" v0.0 ts=2008/11/29 8:30
66k 2009/03/27 C:\cygwin_1.7\bin\cygtic-9.dll - os=4.0 img=1.0 sys=4.0
"cygtic-9.dll" v0.0 ts=2009/3/26 23:54
64k 2009/03/02 C:\cygwin_1.7\bin\cygz.dll - os=4.0 img=1.0 sys=4.0
"cygz.dll" v0.0 ts=2009/3/1 19:33
2376k 2009/06/08 C:\cygwin_1.7\bin\cygwin1.dll - os=4.0 img=1.0 sys=4.0
"cygwin1.dll" v0.0 ts=2009/6/8 15:13
Cygwin DLL version info:
DLL version: 1.7.0
DLL epoch: 19
DLL old termios: 5
DLL malloc env: 28
Cygwin conv: 181
API major: 0
API minor: 210
Shared data: 5
DLL identifier: cygwin1
Mount registry: 3
Cygwin registry name: Cygwin
Program options name: Program Options
Cygdrive default prefix:
Build date: Mon Jun 8 22:13:39 CEST 2009
Shared id: cygwin1S5
Warning: There are multiple cygwin1.dlls on your path
Can't find the cygrunsrv utility, skipping services check.
Cygwin Package Information
Last downloaded files to: C:\Documents and Settings\Administrator.TRC\Desktop
Last downloaded files from: ftp://mirror.mcs.anl.gov/pub/cygwin/
Package Version
_update-info-dir 00823-1
alternatives 1.3.30c-10
ash 20040127-4
base-cygwin 2.0-1
base-files 3.8-3
base-passwd 3.1-1
bash 3.2.48-21
bzip2 1.0.5-10
coreutils 7.0-2
cygutils 1.4.0-1
cygwin 1.7.0-49
cygwin-doc 1.5-1
editrights 1.01-2
findutils 4.5.4-1
gawk 3.1.6-2
grep 2.5.3-1
groff 1.19.2-2
gzip 1.3.12-2
ipc-utils 1.0-1
less 429-1
libbz2_1 1.0.5-10
libgcc1 4.3.2-2
libgmp3 4.3.1-3
libiconv2 1.13-10
libintl3 0.14.5-1
libintl8 0.17-11
libncurses8 5.5-10
libncurses9 5.7-14
libpcre0 7.9-1
libpopt0 1.6.4-4
libreadline6 5.2.13-11
login 1.10-10
man 1.6e-1
rebase 3.0-2
rsync 3.0.5-1
run 1.1.10-1
sed 4.1.5-2
tar 1.22-1
termcap 20050421-1
terminfo 5.7_20090228-10
terminfo0 5.5_20061104-10
texinfo 4.13-3
tzcode 2008h-1
which 2.20-2
zlib0 1.2.3-10
Use -h to see help about each section
-------------- next part --------------
--
Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
Problem reports: http://cygwin.com/problems.html
Documentation: http://cygwin.com/docs.html
FAQ: http://cygwin.com/faq/
More information about the Cygwin
mailing list