ubsan: integer overflow in tc-i386.c:offset_in_range

[H.J. Lu]

On Thu, May 22, 2025 at 6:31 AM Alan Modra <amodra@gmail.com> wrote:
>
> On Thu, May 22, 2025 at 06:19:11AM +0800, H.J. Lu wrote:
> > On Thu, May 22, 2025 at 5:55 AM Alan Modra <amodra@gmail.com> wrote:
> > >
> > > or $9223372036854775808,%eax
> > > runtime error: negation of -9223372036854775808 cannot be represented
> > > in type 'offsetT' (aka 'long'); cast to an unsigned type to negate
> > > this value to itself
> > >
> > > Nothing in this function needs a signed type, despite the value under
> >                                                      Did you mean "a
> > unsigned type"?
>
> No.  offsetT is signed, addressT unsigned.  The code is all bit
> twiddling, no expressions are signed comparisons or similar, so I made
> the parameter unsigned rather than writing -(addressT) val.

This function is called on both imm and offset operands which both are
signed, not unsigned.  So the warning comes from (-val & ~mask).
Can it be changed to (-((addressT) val) -mask) instead?

> > > test being signed.
> > >
> > >         * config/tc-i386.c (offset_in_range): Make "val" unsigned.
> > >
> > > diff --git a/gas/config/tc-i386.c b/gas/config/tc-i386.c
> > > index 162e28e0325..e1f30d4b133 100644
> > > --- a/gas/config/tc-i386.c
> > > +++ b/gas/config/tc-i386.c
> > > @@ -2975,7 +2975,7 @@ smallest_imm_type (offsetT num)
> > >  }
> > >
> > >  static offsetT
> > > -offset_in_range (offsetT val, int size)
> > > +offset_in_range (addressT val, int size)
> > >  {
> > >    addressT mask;
> > >
> > >
> > > --
> > > Alan Modra
> >
> >
> > --
> > H.J.
>
> --
> Alan Modra



-- 
H.J.