s390x help needed - kernel read faults

[David Smith <dsmith at redhat dot com>]

I've been looking at BZ639338
(<https://bugzilla.redhat.com/show_bug.cgi?id=639338>).  Basically the
testsuite is getting lots of kernel read faults on RHEL6 (2.6.32) that
it doesn't get on RHEL5 (2.6.18).

This reminded me of PR12341 "deref()/store_deref() variations between
arches" (<http://sourceware.org/bugzilla/show_bug.cgi?id=12341>).  I
hadn't found the time to see if the s390x could use the kernel's own
get_user()/put_user() macros instead of having custom s390x asm code in
loc2c-runtime.h.

However, replacing systemtap's custom s390x asm code didn't fix the
kernel read faults - it fixed some but others popped up.  That work did
find PR13289 "%m/%M printf formatting operators access memory", so it
wasn't wasted at all.

I've learned a lot about the s390x platform that I didn't know before.
The one most relevant here is that the s390x has 3 different address
spaces - "home", "primary", and "secondary" with some instructions that
only operate on certain address spaces.

In RHEL5 (2.6.18) kernels, replacing our custom asm code with the stock
__get_user()/__put_user() showed similar results as RHEL6.  After
looking at it a bit, I realized the main difference between the stock
RHEL5 __get_user() and our version is that we use the 'mvc' instruction
and the stock __get_user() code uses the 'mvcp' instruction.  The 'mvcp'
instruction only works on addresses in the primary address space (where
user data is stored on 2.6.18 kernels).  Systemtap's deref() macro is
used to read both user data and kernel data, but the stock RHEL5
__get_user() function will only read user data.

In RHEL6 (2.6.32) kernels, things are more complicated.  The stock
__get_user()/__put_user() macros can end up using one of 3 sets of code
(uaccess_std.c, uaccess_pt.c, uaccess_mvcos.c), based on the system's
hardware and the size of the data request.  In addition, which address
space will be used for kernel data vs user data appears to also differ
based on the system's hardware.  But, once again the stock
__get_user()/__put_user() code only works on true user data, not kernel
data.

At this point my s390x knowledge is exhausted.  I need help from people
who actually understand this platform to be able to fix
deref()/store_deref() to properly work on kernel data and user data in
newer kernels.  Hopefully, we can use as much stock kernel code as
possible, to avoid this problem popping up again in the future.

Thanks.

-- 
David Smith
dsmith@redhat.com
Red Hat
http://www.redhat.com
256.217.0141 (direct)
256.837.0057 (fax)