Improve check on downloaded tarballs to workaround stupid proxies ?

Thomas Petazzoni
Mon Jul 25 09:48:00 GMT 2011


Last week, I had to use Crosstool-NG in an environment where HTTP and
FTP servers are accessed through a proxy. Unfortunately, the stupid
behavior of the proxy combined with how Crosstool-NG fetches tarball
didn't play well.

Basically, the proxy is stupid enough that when you do "wget
ftp://someserver/something", when "something" doesn't exist on
"someserver", instead of returning an error, the proxy generates a
short HTML page saying "The file does not exist". So wget returns with
a successful error code, but the downloaded file is a HTML page and not
the expected tarball.

Unfortunately, Crosstool-NG has a logic that consists in trying
multiple locations for tarballs before finding the right one. Combined
with the above mentioned proxy-misfeature, Crosstool-NG ends up
thinking that the tarball was downloaded while in fact only a few
hundreds bytes of HTML has been downloaded. Crosstool-NG continues its
execution, until the extraction step, which of course, fails.

This problem occured with the latest stable version of ct-ng,
specifically on the gcc and cloog-ppl tarballs.

I think ct-ng should do a few more checks on the downloaded files to
ensure that they at least remotely look like valid tarballs. For
example, check with "file --mime-type" that the file looks valid, or
some other similar check. If the file isn't a valid tarball, then the
next URL is tried. This would allow to workaround stupid proxies, and
make ct-ng even easier to use in so-called corporate environments.


Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.

For unsubscribe information see

More information about the crossgcc mailing list