Re: setup

[Achim Gratz <Stromeko at nexgo dot de>]

Achim Gratz writes:
> Pushed.

I've found another (and hopefully the last, except for the mostly
dormant gzip code) 16kiB buffer to be increased to 64kiB.  Also, when
installing from a local directory without a setup.ini file there
actually is a need to skip the checksum validation.  For this to work we
also need to record whether we actually set a checksum and testing the
first byte for zero is not correct for doing this.  As detailed in the
design for Bitcoin, there is an exponetnially decreasing probability for
the leading bits to all be zero.  Current Cygwin has more than 80
packages with the first eight bits or more all zero (I've seen 15
leading zero bits in my local mirror).

>From 9de3a769ae38b1e54ec99e5f105f832a74becfab Mon Sep 17 00:00:00 2001
From: Achim Gratz <Stromeko@Stromeko.DE>
Date: Tue, 4 Aug 2015 19:39:23 +0200
Subject: [PATCH 1/2] Increase read buffer size for MD5 checksumming to 64kiB

	* install.cc (md5_one): Change buffer size from 16kiB to 64kiB for
	faster reading.
---
 ChangeLog  | 7 ++++++-
 install.cc | 2 +-
 2 files changed, 7 insertions(+), 2 deletions(-)

diff --git a/ChangeLog b/ChangeLog
index 355fd23..334bd9a 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,4 +1,9 @@
-2015-08-03  Achim Gratz  <ASSI <Stromeko@NexGo.DE>>
+2015-08-04  Achim Gratz  <Stromeko@NexGo.DE>
+
+	* install.cc (md5_one): Change buffer size from 16kiB to 64kiB for
+	faster reading.
+
+2015-08-03  Achim Gratz  <Stromeko@NexGo.DE>
 
 	* inilex.ll: Introduce HEX and B64 definitions, use them in the
 	rules section.  Parse both SHA512 and SHA512-Base64URL checksums
diff --git a/install.cc b/install.cc
index 1e69564..8c1589b 100644
--- a/install.cc
+++ b/install.cc
@@ -1010,7 +1010,7 @@ md5_one (const packagesource& pkgsource)
   Progress.SetText4 ("Progress:");
   Progress.SetBar1 (0);
 
-  unsigned char buffer[16384];
+  unsigned char buffer[64 * 1024];
   ssize_t count;
   while ((count = thefile->read (buffer, sizeof (buffer))) > 0)
     {
-- 
2.4.6

>From d2f2501580391154fbd5331a932271ce17880680 Mon Sep 17 00:00:00 2001
From: Achim Gratz <Stromeko@Stromeko.DE>
Date: Tue, 4 Aug 2015 19:48:56 +0200
Subject: [PATCH 2/2] Properly record SHA512 checksum presence and skip
 validation for ad-hoc installs

	* package_source.h (packagesource): Add boolean member variable
	sha512_isSet to record whether an SHA512 checksum has been set.
	(packagesource): Initialize sha512_isSet to false.

	* IniDBBuilderPackage.cc (buildInstallSHA512, buildSourceSHA512):
	Only set the SHA512 checksum when it was previously unset like it
	is done for MD5 checksums.  That will generally be the checksum
	recorded on the package line in setup.ini, any further checksums
	in separate lines will thus be ignored.

	* install.cc (chksum_one): Conditionalize the comparison of the
	SHA512 checksum on whether or not it was previously set.  Check
	SHA512 checksum first since it is the default now.  This is
	necessary for ad-hoc installs from local disk without a setup.ini
	file.  Output a warning when the checksum was not be verified
	because neither a MD5 nor a SHA512 checksum was set.
---
 ChangeLog              | 19 +++++++++++++++++++
 IniDBBuilderPackage.cc |  8 ++++++--
 install.cc             |  8 ++++++--
 package_source.h       |  2 ++
 4 files changed, 33 insertions(+), 4 deletions(-)

diff --git a/ChangeLog b/ChangeLog
index 334bd9a..03f91b4 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,5 +1,24 @@
 2015-08-04  Achim Gratz  <Stromeko@NexGo.DE>
 
+	* package_source.h (packagesource): Add boolean member variable
+	sha512_isSet to record whether an SHA512 checksum has been set.
+	(packagesource): Initialize sha512_isSet to false.
+
+	* IniDBBuilderPackage.cc (buildInstallSHA512, buildSourceSHA512):
+	Only set the SHA512 checksum when it was previously unset like it
+	is done for MD5 checksums.  That will generally be the checksum
+	recorded on the package line in setup.ini, any further checksums
+	in separate lines will thus be ignored.
+
+	* install.cc (chksum_one): Conditionalize the comparison of the
+	SHA512 checksum on whether or not it was previously set.  Check
+	SHA512 checksum first since it is the default now.  This is
+	necessary for ad-hoc installs from local disk without a setup.ini
+	file.  Output a warning when the checksum was not be verified
+	because neither a MD5 nor a SHA512 checksum was set.
+
+2015-08-04  Achim Gratz  <Stromeko@NexGo.DE>
+
 	* install.cc (md5_one): Change buffer size from 16kiB to 64kiB for
 	faster reading.
 
diff --git a/IniDBBuilderPackage.cc b/IniDBBuilderPackage.cc
index 7ee2af4..b41955a 100644
--- a/IniDBBuilderPackage.cc
+++ b/IniDBBuilderPackage.cc
@@ -268,15 +268,19 @@ IniDBBuilderPackage::buildInstallSize (const std::string &size)
 void
 IniDBBuilderPackage::buildInstallSHA512 (unsigned char const *sha512)
 {
-  if (sha512)
+  if (sha512 && cbpv.source()->sha512_isSet) {
     memcpy (cbpv.source()->sha512sum, sha512, sizeof cbpv.source()->sha512sum);
+    cbpv.source()->sha512_isSet = true;
+  }
 }
 
 void
 IniDBBuilderPackage::buildSourceSHA512 (unsigned char const *sha512)
 {
-  if (sha512)
+  if (sha512 && cbpv.source()->sha512_isSet) {
     memcpy (cspv.source()->sha512sum, sha512, sizeof cspv.source()->sha512sum);
+    cbpv.source()->sha512_isSet = true;
+  }
 }
 
 void
diff --git a/install.cc b/install.cc
index 8c1589b..a5c4b10 100644
--- a/install.cc
+++ b/install.cc
@@ -1044,8 +1044,12 @@ chksum_one (const packagesource& pkgsource)
 {
   if (!pkgsource.Cached ())
     return;
-  if (pkgsource.md5.isSet())
+  if (pkgsource.sha512_isSet)
+    sha512_one (pkgsource);
+  else if (pkgsource.md5.isSet())
     md5_one (pkgsource);
   else
-    sha512_one (pkgsource);
+    Log (LOG_BABBLE) << "No checksum recorded for " << pkgsource.Base ()
+		     << ", cannot determine integrity of package!"
+		     << endLog;
 }
diff --git a/package_source.h b/package_source.h
index 79d357b..997ccf8 100644
--- a/package_source.h
+++ b/package_source.h
@@ -62,6 +62,7 @@ public:
    _installedSize (0)
   {
     memset (sha512sum, 0, sizeof sha512sum);
+    sha512_isSet = false;
   };
   /* how big is the source file */
   size_t size;
@@ -107,6 +108,7 @@ public:
   virtual void set_canonical (char const *);
   virtual void set_cached (const std::string& );
   unsigned char sha512sum[SHA512_DIGEST_LENGTH];
+  bool sha512_isSet;
   MD5Sum md5;
   typedef std::vector <site> sitestype;
   sitestype sites;
-- 
2.4.6


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptation for Waldorf microQ V2.22R2:
http://Synth.Stromeko.net/Downloads.html#WaldorfSDada