pointer comparison breaks Solaris' qsort

Alexandre Oliva aoliva@redhat.com
Tue Jun 8 19:46:00 GMT 2004


On Jun  8, 2004, Jakub Jelinek <jakub@redhat.com> wrote:

> On Tue, Jun 08, 2004 at 06:53:31AM -0300, Alexandre Oliva wrote:
>> This patch fixes a linker crash caused by inconsistent results being
>> returned by the function passed to qsort().  In certain cases, when it
>> returns the same value when comparing both a with b and b with a, the
>> qsort function may end up being called with a pointer that's not
>> within the bounds of the array passed to qsort().

> That is a Solaris bug then.

Nope.  It's not the difference between the arguments passed to the
compare function that isn't a multiple of the pointer size.  We pass
qsort() an array of pointers, and it's the pointers we get from the
array (not pointers into the array) that we use in a pointer
subtraction, even though the objects they point to are not part of the
same array.

So we invoke undefined behavior twice: by computing the difference
between pointers to different objects, and then by returning
inconsistent values from the compare function because the differences
were miscomputed.  As soon as you invoke undefined behavior, all bets
are off.

-- 
Alexandre Oliva             http://www.ic.unicamp.br/~oliva/
Red Hat Compiler Engineer   aoliva@{redhat.com, gcc.gnu.org}
Free Software Evangelist  oliva@{lsd.ic.unicamp.br, gnu.org}



More information about the Binutils mailing list