Re: bzip2 1.0.7 released

[Mark Wielaard <mark@klomp.org>]

Hi,

A bit more analysis before I go to sleep.

On Thu, 2019-06-27 at 22:58 +0200, Mark Wielaard wrote:
> On Thu, Jun 27, 2019 at 08:54:08PM +0200, Mark Wielaard wrote:
> > * Make sure nSelectors is not out of range (CVE-2019-12900)
> 
> Well, that was quick... There is already a regression report about
> this fix. See 
> https://bugs.launchpad.net/ubuntu/+source/bzip2/+bug/1834494
> 
> The fix itself is certainly correct:
> 
> diff --git a/decompress.c b/decompress.c
> index ab6a624..f3db91d 100644
> --- a/decompress.c
> +++ b/decompress.c
> @@ -280,21 +280,21 @@ Int32 BZ2_decompress ( DState* s )
>                 if (uc == 1) s->inUse[i * 16 + j] = True;
>              }
>        makeMaps_d ( s );
>        if (s->nInUse == 0) RETURN(BZ_DATA_ERROR);
>        alphaSize = s->nInUse+2;
>  
>        /*--- Now the selectors ---*/
>        GET_BITS(BZ_X_SELECTOR_1, nGroups, 3);
>        if (nGroups < 2 || nGroups > 6) RETURN(BZ_DATA_ERROR);
>        GET_BITS(BZ_X_SELECTOR_2, nSelectors, 15);
> -      if (nSelectors < 1) RETURN(BZ_DATA_ERROR);
> +      if (nSelectors < 1 || nSelectors > BZ_MAX_SELECTORS)
> RETURN(BZ_DATA_ERROR);
>        for (i = 0; i < nSelectors; i++) {
>           j = 0;
>           while (True) {
>              GET_BIT(BZ_X_SELECTOR_3, uc);
>              if (uc == 0) break;
>              j++;
>              if (j >= nGroups) RETURN(BZ_DATA_ERROR);
>           }
>           s->selectorMtf[i] = j;
>        }
> 
> Because if nSelectors would be > BZ_MAX_SELECTORS it would write over
> memory after the selectorMtf array.
> 
> The problem with the file in the report is that it does contain some
> nSelectors that are slightly larger than BZ_MAX_SELECTORS.
> 
> The test file can be found here:
> 
https://developer.nvidia.com/embedded/dlc/l4t-jetson-xavier-driver-package-31-1-0
> 
> The fix is simple:
> 
> diff --git a/bzlib_private.h b/bzlib_private.h
> index 7975552..ef870d9 100644
> --- a/bzlib_private.h
> +++ b/bzlib_private.h
> @@ -122,7 +122,7 @@ extern void bz_internal_error ( int errcode );
>  #define BZ_G_SIZE   50
>  #define BZ_N_ITERS  4
>  
> -#define BZ_MAX_SELECTORS (2 + (900000 / BZ_G_SIZE))
> +#define BZ_MAX_SELECTORS (7 + (900000 / BZ_G_SIZE))
>  
>  
>  
> But of course I cannot tell why increasing the max with 5 is correct.
> It might well be that the file is invalid. Before the fix bunzip2
> would overwrite some memory after the selectorMtf array. So it might
> be the file decompressed by accident in the past.
> 
> I'll look a but deeper, but if people have a clue what exactly is
> going on that would be appreciated.

So it looks like some implementations might add more selectors than
necessary. For example lbzip2 seems to use a max of 18000 + 1 + 7.
Which might explain why our 18002 = 2 + (900000 / 50) isn't enough, and
why my random increase of 5 seemed to work for the given file.

In general the nSelector field can be up to 15 bits, so 32768. So we
definitely do want to check the input doesn't overflow (or make
BZ_MAX_SELECTORS 32768, but that seems excessive).

Cheers,

Mark