Re: [PATCH] Fix p{readv,writev}{64} consolidation implementation

[Adhemerval Zanella <adhemerval dot zanella at linaro dot org>]

On 16/06/2016 12:25, Chris Metcalf wrote:
> On 6/15/2016 4:17 PM, Adhemerval Zanella wrote:
>> On 15/06/2016 15:21, Chris Metcalf wrote:
>>> On 6/15/2016 9:45 AM, Adhemerval Zanella wrote:
>>>> On 15/06/2016 02:37, Mike Frysinger wrote:
>>>>> On 14 Jun 2016 18:54, Adhemerval Zanella wrote:
>>>>>> This patch fixes the p{readv,writev}{64} consolidation implementation
>>>>>> from commits 4e77815 and af5fdf5.  Different from pread/pwrite
>>>>>> implementation, preadv/pwritev implementation does not require
>>>>>> __ALIGNMENT_ARG because kernel syscall prototypes define
>>>>>> the high and low part of the off_t, if it is the case, directly
>>>>>> (different from pread/pwrite where the architecture ABI for passing
>>>>>> 64-bit values must be in consideration for passsing the arguments).
>>>>> i had looked at that specifically but thought it ok because the old code
>>>>> was using the alignment arg.  was the old code broken too ?
>>>>>
>>>>> this is what the preadv code looked like:
>>>>> -ssize_t
>>>>> -__libc_preadv (int fd, const struct iovec *vector, int count, off_t offset)
>>>>> -{
>>>>> -  assert (sizeof (offset) == 4);
>>>>> -  return SYSCALL_CANCEL (preadv, fd,
>>>>> -                         vector, count, __ALIGNMENT_ARG
>>>>> -                         __LONG_LONG_PAIR (offset >> 31, offset));
>>>>> -}
>>>>>
>>>>> although i guess this isn't too surprising as this code was in the
>>>>> generic sysdeps dir which currently doesn't have as many users as
>>>>> we wish it did :).
>>>>>
>>>> The idea is not really to align the argument to zero pass, but rather to split
>>>> the possible 64-bits argument in high and low as required (as the default
>>>> implementation was doing [2]). On tile, it is working because the preadv.c
>>>> offset is 32-bits and thus the high word is indeed zero, but it is passing
>>>> one superfluous argument.
>>> No, what happens is that instead of passing r3=pos_low=lo, r4=pos_high=0, we
>>> are passing r3=pos_low=0, r4=pos_high=lo, r5=0.  This means that we get a crazy
>>> high offset and either fail or (for a sufficiently large file) get the wrong data.
>>> Filed as bug 20261.
>> I was referring to *old* behaviour (pre-consolidation) implementation
>> (the sysdeps/unix/sysv/linux/generic/wordsize-32/preadv.c), which did:
>>
>> -- 
>>    return SYSCALL_CANCEL (preadv, fd,
>>                           vector, count, __ALIGNMENT_ARG
>>                           __LONG_LONG_PAIR (offset >> 31, offset));
>> -- 
>>
>> It has 2 issue:
>>
>>   1. It passed one superfluous argument to preadv. On tilepro build
>>      I noted that is using internal_syscall6, which means it is passing
>>      { fd, vector, cound, 0, offset, offset>>31 }.  It is not wrong,
>>      since for this code off_t will be the old 32-bit value, but the
>>      semantic is wrong.
> 
> No, it's definitely wrong :-)
> 
> We pass "offset" as the high part, and "0" as the low part. Accordingly, what
> the kernel sees is a request for "offset << 32" rather than "offset".  The bug
> report I filed contains a repro that does fail on tilegx32.

I am not following because I am referring to the old code that was *suppose*
to work on tile (either on 32-bits kernel or 64-bits kernel with compat
enabled). I rechecked the compilation against GLIBC 2.23 (which does not 
contain this consolidation) and I am seeing tilepro passing:

"R00" (fd), "R01" (vector), "R02" (count), "R03" (0), "R04" (offset), "R05" (offset >> 31)

Which indeed seems wrong with the issue you pointed out.  So how was this
suppose to work on previous GLIBC?

The new consolidation implementation indeed contains a bug and that's what
my v2 [1] is trying to fix.  With the v2 I am not seeing:

"R00" (fd), "R01" (vector), "R02" (count), "R03" ((long) (offset)), "R04" ((long) (((uint64_t) (offset)) >> 32))

[1] https://sourceware.org/ml/libc-alpha/2016-06/msg00607.html

> 
>>   2. __LONG_LONG_PAIR is not correct for big-endian.
> 
> Yes, this needs to be fixed on the kernel side for consistency. Many other
> syscalls pass their arguments with __LONG_LONG_PAIR.  Handling it this way
> in the kernel seems pretty standard at this point, e.g. regs_to_64 in
> arch/arm64/include/asm/assembler.h, or merge_64 in
> arch/mips/kernel/linux32.c.  Even the fixed-endian platforms do it
> consistently, e.g. "hi, lo" on parisc/s390/sparc and "lo, hi" on metag/x86.
> 
> It's possible little-endian compat powerpc shares this bug since it seems to
> take fixed "hi, lo" arguments, e.g. in compat_sys_readahead().  I'll
> forward this email on to the powerpc kernel maintainers to make sure
> they have a chance to think about it.
> 
> See https://lkml.kernel.org/g/1466019219-10462-1-git-send-email-cmetcalf@mellanox.com

I *really* do not think this is correct approach mainly because it break an
already defined kernel ABI and old BE preadv call on GLIBC was not done
by using __LONG_LONG_PAIR.  Old GLIBC will just fail in this new kernel
interface.  Unfortunately this is something we will need to carry on IMHO.

That's why I just removed the wrong assumption in old
sysdeps/unix/sysv/linux/generic/wordsize-32/preadv{64} about __LONG_LONG_PAIR.
 

>> Now for BZ#20261 I do not think it applicable since this is a fix for
>> consolidation done in development phase, it does not appear in any
>> released version.
> 
> It breaks on all versions, as far as I can tell.  I tested it on our RHEL 6 platform
> (based on backported glibc 2.12) and it fails there.

What exactly breaks in all version? What I am referring is that not the 
bug is 'invalid', but rather it does not apply to a *user visible* one
because the consolidation code is only in 2.24 master branch (it is not
in any *released* version).

Now, if it fails on older releases it is not due this change, but rather
due old sysdeps/unix/sysv/linux/generic/wordsize-32/preadv{64}.  Now, if
it is the case it is indeed a user visible bug.

> 
>> Now, your analysis is correct for *current* code and it is contains the
>> __LONG_LONG_PAIR issue due the SYSCALL_LL usage.  I will change it
>> and post a second version.
> 
> I think you should revert that part of the change and stick with __LONG_LONG_PAIR.
>