Observations about Cygwin's md5 checksums

Luke Kendall luke.kendall@cisra.canon.com.au
Tue Jul 8 03:12:00 GMT 2014


On 07/07/14 20:05, Marco Atzeri wrote:
> On 07/07/2014 06:38, Luke Kendall wrote:
>> Here are five observations about md5 checksums in Cygwin.  I share it in
>> case it may be of some small interest to a few people.  Please note that
>> I may be wrong; if so, I'm happy to be corrected.
>>
>> 1) For each package, Cygwin stores the md5sum for the components of the
>> main parts of the package in the setup.ini file.  The exception is the
>> setup.hint file: its md5 sum is not recorded in setup.ini.
>
> setup.ini is built using the setup.hint's of the several packages.
> No further usage outside the www.cygwin.com server.

Thanks for the explanation, Marco.  Can I check what you mean?
I think you're saying that setup.hint has no function once setup.ini has 
been created.
(You're not saying "setup.ini shouldn't be used for other purposes.")

>> 2) In each zip file for each package, an md5.sum file is almost always
>> provided.  But not always. (*)
>>
>> 3) These md5.sum files list all the components of the package (including
>> setup.hint), but these md5 sums are not reliable: they often don't match
>> the actual md5 checksum (of the file itself, or of course the md5 stored
>> for it in setup.ini).(**)
>
> during upload of new files the creation of md5.sum is out of sync
> with the directory content. Md5.sum is updated 1 time per hour
> If the mirror sync before the creation of the md5sum it has still the 
> old version.
>

Does that mean that if the md5.sum file were created at the same time 
that the package contents were updated, there would be no possible way 
for out-of-sync md5.sum files to be provided?  Do you think the current 
process could be improved?

I have observed that the errors in the mirrors persist for weeks or more.

Anyway, by changing our checking process to use the information in 
setup.ini instead of the md5.sum files which can be wrong (for many 
weeks), we can bypass the incorrect md5.sum files.

>> 4) The most common file to have the wrong md5 checksum is setup.hint
>>
>> 5) It's not rare for files to be mentioned in a package's md5.sum which
>> are be absent from the package itself.(***)
>>
>> I'm curious about the purpose of having the md5.sum file in each
>> package.  Is it a relic of a previous system?
>>
>> The above observations are based on a few weeks of mirroring and
>> automatically checking the md5 sums of what we downloaded.  The main
>> site we used was aarnet.edu.au (IIRC); recently we changed to
>> mirrors.kernel.org, but from my ad hoc checks there wasn't much
>> difference between the two).
>>
>> Regards,
>>
>> luke
>>
>> (*)
>> For mirrors.kernel.org last night:
>>
>> Worrying: X11/khronos-opengl-registry has no md5.sum file
>> Worrying: X11/xlaunch has no md5.sum file
>> Worrying: cygwin64-gcc/cygwin64-gcc-debuginfo has no md5.sum file
>> Worrying: git/git-oodiff has no md5.sum file
>> Worrying: git/stgit has no md5.sum file
>> Worrying: git/tig has no md5.sum file
>> Worrying: man has no md5.sum file
>> Worrying: python/python-paramiko has no md5.sum file
>
> some of this directory does not exist anymore on www.cygwin.com

Fair enough.  But from what you explained above, it seems strange that 
no md5.sum file exists.  Is it another problem caused by not providing 
the updated md5.sum file at the same time that a package is updated?  I 
wonder if there's a race condition: files in the package are updated, 
one by one, the md5.sum file removed, the new md5 sums computed and the 
file written - and in the meantime, someone may have mirrored the 
partially-updated package?

>
>>
>> (**)
>> $ grep FAILED [path-omitted]/x86/cygwin-archive-incomplete.txt | wc
>>       55     110    1463
>> $ grep "^setup.hint: FAILED"
>> [path-omitted]/x86/cygwin-archive-incomplete.txt | wc
>>       28      56     560
>>
>> (***)
>> $ wc -l < [path-omitted]/x86/cygwin-archive-all-missing-files.txt
>> 406
>>
>>
>
> Regards
> MArco
>
>
> -- 
> Problem reports:       http://cygwin.com/problems.html
> FAQ:                   http://cygwin.com/faq/
> Documentation:         http://cygwin.com/docs.html
> Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
>

Thanks for taking the time to explain, Marco.

regards,

luke

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple



More information about the Cygwin mailing list