[PATCH] Fix p{readv,writev}{64} consolidation implementation

[Chris Metcalf]

(+Chung-Lin Tang for possible nios2 bug.)

On 6/16/2016 11:49 AM, Adhemerval Zanella wrote:
> On 16/06/2016 12:25, Chris Metcalf wrote:
>> On 6/15/2016 4:17 PM, Adhemerval Zanella wrote:
>>> On 15/06/2016 15:21, Chris Metcalf wrote:
>>>> On 6/15/2016 9:45 AM, Adhemerval Zanella wrote:
>>>>> On 15/06/2016 02:37, Mike Frysinger wrote:
>>>>>> On 14 Jun 2016 18:54, Adhemerval Zanella wrote:
>>>>>>> This patch fixes the p{readv,writev}{64} consolidation implementation
>>>>>>> from commits 4e77815 and af5fdf5.  Different from pread/pwrite
>>>>>>> implementation, preadv/pwritev implementation does not require
>>>>>>> __ALIGNMENT_ARG because kernel syscall prototypes define
>>>>>>> the high and low part of the off_t, if it is the case, directly
>>>>>>> (different from pread/pwrite where the architecture ABI for passing
>>>>>>> 64-bit values must be in consideration for passsing the arguments).
>>>>>> i had looked at that specifically but thought it ok because the old code
>>>>>> was using the alignment arg.  was the old code broken too ?
>>>>>>
>>>>>> this is what the preadv code looked like:
>>>>>> -ssize_t
>>>>>> -__libc_preadv (int fd, const struct iovec *vector, int count, off_t offset)
>>>>>> -{
>>>>>> -  assert (sizeof (offset) == 4);
>>>>>> -  return SYSCALL_CANCEL (preadv, fd,
>>>>>> -                         vector, count, __ALIGNMENT_ARG
>>>>>> -                         __LONG_LONG_PAIR (offset >> 31, offset));
>>>>>> -}
>>>>>>
>>>>>> although i guess this isn't too surprising as this code was in the
>>>>>> generic sysdeps dir which currently doesn't have as many users as
>>>>>> we wish it did :).
>>>>>>
>>>>> The idea is not really to align the argument to zero pass, but rather to split
>>>>> the possible 64-bits argument in high and low as required (as the default
>>>>> implementation was doing [2]). On tile, it is working because the preadv.c
>>>>> offset is 32-bits and thus the high word is indeed zero, but it is passing
>>>>> one superfluous argument.
>>>> No, what happens is that instead of passing r3=pos_low=lo, r4=pos_high=0, we
>>>> are passing r3=pos_low=0, r4=pos_high=lo, r5=0.  This means that we get a crazy
>>>> high offset and either fail or (for a sufficiently large file) get the wrong data.
>>>> Filed as bug 20261.
>>> I was referring to *old* behaviour (pre-consolidation) implementation
>>> (the sysdeps/unix/sysv/linux/generic/wordsize-32/preadv.c), which did:
>>>
>>> -- 
>>>     return SYSCALL_CANCEL (preadv, fd,
>>>                            vector, count, __ALIGNMENT_ARG
>>>                            __LONG_LONG_PAIR (offset >> 31, offset));
>>> -- 
>>>
>>> It has 2 issue:
>>>
>>>    1. It passed one superfluous argument to preadv. On tilepro build
>>>       I noted that is using internal_syscall6, which means it is passing
>>>       { fd, vector, cound, 0, offset, offset>>31 }.  It is not wrong,
>>>       since for this code off_t will be the old 32-bit value, but the
>>>       semantic is wrong.
>> No, it's definitely wrong :-)
>>
>> We pass "offset" as the high part, and "0" as the low part. Accordingly, what
>> the kernel sees is a request for "offset << 32" rather than "offset".  The bug
>> report I filed contains a repro that does fail on tilegx32.
> I am not following because I am referring to the old code that was *suppose*
> to work on tile (either on 32-bits kernel or 64-bits kernel with compat
> enabled). I rechecked the compilation against GLIBC 2.23 (which does not
> contain this consolidation) and I am seeing tilepro passing:
>
> "R00" (fd), "R01" (vector), "R02" (count), "R03" (0), "R04" (offset), "R05" (offset >> 31)

This is wrong, and this was indeed a bug prior to your consolidation patch.
This is a generic/wordsize-32 specific bug which apparently was not caught
during testing of tilepro or tilegx32.  In addition to the erroneous __ALIGNMENT_ARG,
we should be using LO_HI_LONG() here rather than __LONG_LONG_PAIR.
I suspect this is also a bug in the nios2 glibc.

> Which indeed seems wrong with the issue you pointed out.  So how was this
> suppose to work on previous GLIBC?

It doesn't.  That's why I filed the bug.

> The new consolidation implementation indeed contains a bug and that's what
> my v2 [1] is trying to fix.  With the v2 I am not seeing:
>
> "R00" (fd), "R01" (vector), "R02" (count), "R03" ((long) (offset)), "R04" ((long) (((uint64_t) (offset)) >> 32))
>
> [1]https://sourceware.org/ml/libc-alpha/2016-06/msg00607.html

When you say "not seeing", I'm assuming you mean "now seeing".

In any case, "vector, count, lo, hi" is what we should see here.

>>>    2. __LONG_LONG_PAIR is not correct for big-endian.
>> Yes, this needs to be fixed on the kernel side for consistency. Many other
>> syscalls pass their arguments with __LONG_LONG_PAIR.  Handling it this way
>> in the kernel seems pretty standard at this point, e.g. regs_to_64 in
>> arch/arm64/include/asm/assembler.h, or merge_64 in
>> arch/mips/kernel/linux32.c.  Even the fixed-endian platforms do it
>> consistently, e.g. "hi, lo" on parisc/s390/sparc and "lo, hi" on metag/x86.
>>
>> It's possible little-endian compat powerpc shares this bug since it seems to
>> take fixed "hi, lo" arguments, e.g. in compat_sys_readahead().  I'll
>> forward this email on to the powerpc kernel maintainers to make sure
>> they have a chance to think about it.
>>
>> Seehttps://lkml.kernel.org/g/1466019219-10462-1-git-send-email-cmetcalf@mellanox.com
> I *really* do not think this is correct approach mainly because it break an
> already defined kernel ABI and old BE preadv call on GLIBC was not done
> by using __LONG_LONG_PAIR.  Old GLIBC will just fail in this new kernel
> interface.  Unfortunately this is something we will need to carry on IMHO.
>
> That's why I just removed the wrong assumption in old
> sysdeps/unix/sysv/linux/generic/wordsize-32/preadv{64} about __LONG_LONG_PAIR.

Yes, there are two issues here that I am conflating - sorry about that.

The preadv/pwritev stuff should be using LO_HI_LONG.  I think your
v2 patch is correct.  No kernel change is required here.

When I went and reviewed this stuff for tile, it led me to the tile kernel ABI
code that was not properly handling __LONG_LONG_PAIR in the kernel
for APIs that expect 64-bit values like truncate/ftruncate, pread64/pwrite64,
sync_file_range, and fallocate.

I think our installed base of "tilepro/tilegx32 big-endian" is so small as to
be effectively zero.  We had one or two customers interested in big-endian 64-bit,
and a couple of customers interested in tilegx compat 32-bit, but I don't think we
ever had customers interested in the cross product of those two features.

Generally speaking I would otherwise agree and say we should provide
tile-specific variants of all those broken syscalls in glibc to support the existing ABI.

>>> Now for BZ#20261 I do not think it applicable since this is a fix for
>>> consolidation done in development phase, it does not appear in any
>>> released version.
>> It breaks on all versions, as far as I can tell.  I tested it on our RHEL 6 platform
>> (based on backported glibc 2.12) and it fails there.
> What exactly breaks in all version? What I am referring is that not the
> bug is 'invalid', but rather it does not apply to a *user visible* one
> because the consolidation code is only in 2.24 master branch (it is not
> in any *released* version).
>
> Now, if it fails on older releases it is not due this change, but rather
> due old sysdeps/unix/sysv/linux/generic/wordsize-32/preadv{64}.  Now, if
> it is the case it is indeed a user visible bug.

Yes, that's exactly the case for preadv/pwritev, due to the bogus _ALIGNMENT_ARG;
it breaks on all 32-bit tile platforms with all public releases.

-- 
Chris Metcalf, Mellanox Technologies
http://www.mellanox.com