[RFC] FIPS compliance and other crypt(3) improvements

[Alexandre Oliva]

We behave in somewhat unexpected ways when presented with semi-standard
arguments for crypt.  Namely, there are various specified extensions to
the salt parameter that we do not implement.  That's fine.  The problem
is that it is not entirely clear that we do not implement them, because
we accept and use salts such as "\0/" or "$", and use them in ways that
not only fail to match documented extensions such as for "$2$", but that
also deviate from the POSIX-specified behavior, in that we perturb the
DES encryption using out-of-spec characters, that don't match
[0-9A-Za-f/.], and that might even be NUL.

This is what the first attached patch fixes: if we're about to fallback
to DES, we check that the salt starts with two characters (and not less
than that) that belong to the specified alphabet, and return NULL if it
doesn't, rather than encrypting the password with DES using out-of-spec
salt.

One of the consequences of our being sloppy with the checking is that we
may access the salt string past its end, which may cause unpredictable
encryption and out-of-spec return values when salt is an empty string.
We may even segfault if the salt happens to be an empty string right
before an unmapped page.  As a matter of good practice, we shouldn't
access the second character of a string before checking that the first
is not NUL.  This is also fixed by the first patch, as the added
testcase shows.


The second building block is support for telling whether the system is
running in a FIPS-compliant mode.  I'll be the first to confess I'm not
familiar with FIPS security specifications, or how that affects the
overall system behavior, but I'm told both the kernel and some userland
behavior is to be affected by the choice of wehther or not to operate in
FIPS mode.  E.g., the kernel can be told at boot time to operate in this
mode, and applications that ought to be affected currently have to check
/proc/sys/crypto/fips_enabled.

It turns out that GNU/Linux-specific sysconf is already perfectly
capable of opening a file in /proc and parsing a number in there, so I
figured I might as well expose the FIPS setting through sysconf.  This
is what the second patch does.


The third patch, that depends on the other 2, introduces FIPS-specific
logic in crypt(3), disabling DES and MD5 algorithms that, per FIPS, must
not be used because they're too weak.  I'm not convinced this is the
best path to follow, for we'd be explicitly deviating from POSIX.

Programs that relied on e.g. the DES algorithm implementation in
crypt(3), for purposes other than checking login passwords (e.g. for
cracking them, or, who knows, reducing other algorithms to DES- or
MD5-based crypt) would break, even though no security issue is at hand.
I'm of the opinion that a POSIX- and FIPS-compliant implementation ought
to audit crypt() callers, assess which of them are actually dealing with
login passwords, and disable those.  However, for completeness, I'm
supplying the third patch, that implements this change IN GLIBC, so
that, if others think this is the way to go, the patch is written and
tested.


I'd appreciate patch reviews for these patches; if there are objections
to any of them, please let me know!  Thanks in advance,


-------------- next part --------------
A non-text attachment was scrubbed...
Name: crypt-badsalt-bz811753.patch
Type: text/x-diff
Size: 5930 bytes
Desc: not available
URL: <http://sourceware.org/pipermail/libc-alpha/attachments/20120515/99e64755/attachment.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: sysconf-fips-bz811753.patch
Type: text/x-diff
Size: 1360 bytes
Desc: not available
URL: <http://sourceware.org/pipermail/libc-alpha/attachments/20120515/99e64755/attachment-0001.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: crypt-fipsalgo-bz811753.patch
Type: text/x-diff
Size: 2817 bytes
Desc: not available
URL: <http://sourceware.org/pipermail/libc-alpha/attachments/20120515/99e64755/attachment-0002.bin>
-------------- next part --------------

-- 
Alexandre Oliva, freedom fighter    http://FSFLA.org/~lxoliva/
You must be the change you wish to see in the world. -- Gandhi
Be Free! -- http://FSFLA.org/   FSF Latin America board member
Free Software Evangelist      Red Hat Brazil Compiler Engineer