Problem with Setup and multiple URLs with same host

[Peter A. Castro]

Greetings, All,

  I tend to just be a lurker on the Cygwin lists, but a user of the
Cygwin Time Machine had a problem with the latest Setup version (2.893)
that made me think a little about how Setup treats URLs.  I'd email Jon
Turney off list, and, after some discussion, felt it should be brought
up for discussion.

  The user, call him "Mike", was getting a failure trying to fetch a
specific circa of Cygwin from the Cygwin Time Machine.  The setup.log
showed something I hadn't seen before, which was that multiple URLs
were being accessed.  Here's a piece of the log:

2018/11/05 17:10:45 Starting cygwin install, version 2.893
2018/11/05 17:10:45 User has backup/restore rights
2018/11/05 17:10:45 Current Directory: C:\cygwin-install\cygwin-time-machine
2018/11/05 17:10:45 Could not open service McShield for query, start and stop. McAfee may not be installed, or we don't have access.
2018/11/05 17:10:49 source: download
2018/11/05 17:10:50 Selected local directory: C:\cygwin-install\cygwin-time-machine
2018/11/05 17:10:51 net: Direct
2018/11/05 17:11:24 site: http://ctm.crouchingtigerhiddenfruitbat.org/pub/cygwin/circa/64bit/
2018/11/05 17:11:24 site: http://ctm.crouchingtigerhiddenfruitbat.org/pub/cygwin/circa/64bit/2014/04/06/120212/
2018/11/05 17:11:24 site: http://ctm.crouchingtigerhiddenfruitbat.org/pub/cygwin/circa/64bit/index.html /
2018/11/05 17:11:27 HTTP status 404 fetching http://ctm.crouchingtigerhiddenfruitbat.org/pub/cygwin/circa/64bit/2014/04/06/120212/x86_64/setup.xz.sig
2018/11/05 17:11:28 HTTP status 404 fetching http://ctm.crouchingtigerhiddenfruitbat.org/pub/cygwin/circa/64bit/index.html /x86_64/setup.xz.sig
2018/11/05 17:11:28 HTTP status 404 fetching http://ctm.crouchingtigerhiddenfruitbat.org/pub/cygwin/circa/64bit/index.html /x86_64/setup.xz
2018/11/05 17:11:28 HTTP status 404 fetching http://ctm.crouchingtigerhiddenfruitbat.org/pub/cygwin/circa/64bit/index.html /x86_64/setup.bz2.sig
2018/11/05 17:11:29 HTTP status 404 fetching http://ctm.crouchingtigerhiddenfruitbat.org/pub/cygwin/circa/64bit/index.html /x86_64/setup.bz2
2018/11/05 17:11:29 HTTP status 404 fetching http://ctm.crouchingtigerhiddenfruitbat.org/pub/cygwin/circa/64bit/index.html /x86_64/setup.ini.sig
2018/11/05 17:11:29 HTTP status 404 fetching http://ctm.crouchingtigerhiddenfruitbat.org/pub/cygwin/circa/64bit/index.html /x86_64/setup.ini
2018/11/05 17:11:29 mbox note: Unable to get setup from <http://ctm.crouchingtigerhiddenfruitbat.org/pub/cygwin/circa/64bit/index.html />
2018/11/05 17:11:33 Ending cygwin install

Now, I did not know that Mike had previously tried adding several
incorrect URLs before, but the end result was that he'd added three URLs
all having the same location:

http://ctm.crouchingtigerhiddenfruitbat.org/pub/cygwin/circa/64bit/
http://ctm.crouchingtigerhiddenfruitbat.org/pub/cygwin/circa/64bit/2014/04/06/120212/
http://ctm.crouchingtigerhiddenfruitbat.org/pub/cygwin/circa/64bit/index.html /

These were recorded, together, in setup.rc under "last-mirror", because
they were added as User mirrors, but they are displayed in the Chooser
as only one location.

Now, as it turns out, two of these will resolve to a valid
setup.ini{.xz} (being qualified with "/x86_64"), but Setup seems to
insist that all three be valid, and, of course, the last one with
"/index.html /" wouldn't have ever worked.  But, also, he really only
wanted the one for 2014/04/06/120212.

A solution for Mike was to edit setup.rc and remove all three URLs and
then re-add only the correct one.  But, that was not obvious and
required the use of an external, native editor.

But this brings up a preceived deficiency in how Setup handles mirror
URLs and failures.  Specifically, that multiple URLs with the same host
& protocol are treated as one virtual entity.  I can understand the
intent to allow a mirror to be split up between multiple paths, but the
problem is that that knowledge is being hidden from the user.

It also appears that Setup lacks an effective way to edit the list of
mirrors in a find-grained manner.

Historically, mirrors supply only one URL for a repository, and I
realize that the Cygwin Time Machine necessarily deviates from that
norm.  But, there really is no other choice in how to present
individual time slices of Cygwin such that Setup can access them.

I'd like to suggest that Setup needs to perform come kind of URL
validation at the time a user provided URL is added.  Something simple
like getting the head of setup.ini at that time of addition, perhaps.
That might help prevent bad mirrors from being added by the user.  I
realize that that doesn't prevent the mirror path from being removed
later, but, that leads to my next point:

Better detail and management.

Setup hides the full URL from the user when it displays URLs in most any
way, and I feel this is a disservice.  It really should display the full
URL, particularly when managing the list of mirrors, but also when
choosing a mirror to use.  I'm not sure where the requirement to group
multiple URLs with the same host, together, came from (haven't checked
the email archives, yet), but I can see both pros and cons for this
functionality.  However it really should be under the users control and
not assumed.

As it is now, the failure Mike observed didn't give him any real clue as
to what or why, which is why he email'd me.  Having users refer to the
setup.log to do diagnostics isn't very user-friendly, and there's no
obvious logic or notice that multiple URLs will be used, so it was a bit
baffling to me to see that too.

Now, of course, the quick fix is to "not enter mutliple/bad URLs", but
that belies the user experience with Setup, which doesn't do a whole lot
of validation of user values until they are tried to be accessed much
later on, which is much too late.

Anyway, just wanted to bring this to light and see what people think.

I haven't had time to look at the code for Setup but I'm hoping that the
good people who are most familiar with it can give a quick evaluation if
some of this is easy to implement or change.

Yes, I know, "PTC"  :)

-- 
--=> Peter A. Castro
Email: doctor at fruitbat dot org / Peter dot Castro at oracle dot com
	"Cats are just autistic Dogs" -- Dr. Tony Attwood