Re: exercising current aarch64 kprobe support with systemtap

[William Cohen <wcohen at redhat dot com>]

On 06/09/2016 12:17 PM, William Cohen wrote:
> I have been exercising the current kprobes and uprobe patches for
> arm64 that are in the test_upstream_arm64_devel branch of
> https://github.com/pratyushanand/linux with systemtap.  There are a
> two issues that I have seen on this kernel with systemtap.  There are
> some cases where kprobes fail to register at places that appear to be
> reasonable places for a kprobe.  The other issue is that kernel starts
> having soft lockups when the hw_watch_addr.stp tests runs.  To get
> systemtap with the newer kernels need the attached hack because of
> changes in the aarch64 macro args.
> 
> EINVAL for seemingly valid kprobe registration
> 
> Below shows the bz1027459.stp failing because of the some of the kprobes not registering.
> 
> # make installcheck RUNTESTFLAGS="--debug systemtap.base/bz1027459.exp"
> ...
> 
> 
> spawn stap /root/systemtap_write/systemtap/testsuite/systemtap.base/bz1027459.stp
> WARNING: probe kernel.function("SyS_set_tid_address@kernel/fork.c:1236").call (address 0xfffffc00080c9578) registration error (rc -22)
> WARNING: probe kernel.function("SyS_sched_setaffinity@kernel/sched/core.c:4690").call (address 0xfffffc0008104d58) registration error (rc -22)
> WARNING: probe kernel.function("SyS_sched_get_priority_min@kernel/sched/core.c:5013").call (address 0xfffffc0008105250) registration error (rc -22)
> WARNING: probe kernel.function("SyS_sched_get_priority_max@kernel/sched/core.c:4986").call (address 0xfffffc00081051e8) registration error (rc -22)
> hi
> FAIL: bz1027459 -p5 (0)
> 
> area around  Sys_set_tid_address
> 
> fffffc00080c956c:	d503201f 	nop
> fffffc00080c9570:	08dc4c80 	.word	0x08dc4c80
> fffffc00080c9574:	fffffc00 	.word	0xfffffc00
> 
> fffffc00080c9578 <SyS_set_tid_address>:
> fffffc00080c9578:	a9be7bfd 	stp	x29, x30, [sp,#-32]!
> fffffc00080c957c:	910003fd 	mov	x29, sp
> 
> area around SyS_sched_setaffiniity
> 
> fffffc0008104d4c:	17ffff73 	b	fffffc0008104b18 <sched_setaffinity+0x438>
> fffffc0008104d50:	08dd9d80 	.word	0x08dd9d80
> fffffc0008104d54:	fffffc00 	.word	0xfffffc00
> 
> fffffc0008104d58 <SyS_sched_setaffinity>:
> fffffc0008104d58:	a9bb7bfd 	stp	x29, x30, [sp,#-80]!
> fffffc0008104d5c:	910003fd 	mov	x29, sp
> 
> area around SyS_sched_get_priority_min
> 
> fffffc0008105244:	f9400bf3 	ldr	x19, [sp,#16]
> fffffc0008105248:	a8c27bfd 	ldp	x29, x30, [sp],#32
> fffffc000810524c:	d65f03c0 	ret
> 
> fffffc0008105250 <SyS_sched_get_priority_min>:
> fffffc0008105250:	a9be7bfd 	stp	x29, x30, [sp,#-32]!
> fffffc0008105254:	910003fd 	mov	x29, sp
> 
> 
> area around SyS_sched_get_priority_max
> 
> 
> fffffc00081051dc:	17ffffe8 	b	fffffc000810517c <sys_sched_yield+0x34>
> fffffc00081051e0:	08dd9d80 	.word	0x08dd9d80
> fffffc00081051e4:	fffffc00 	.word	0xfffffc00
> 
> fffffc00081051e8 <SyS_sched_get_priority_max>:
> fffffc00081051e8:	a9be7bfd 	stp	x29, x30, [sp,#-32]!
> fffffc00081051ec:	910003fd 	mov	x29, sp
> 
> 
> The stp (store pair) instructions at the beginning of these functions
> should be fine to instrument.  One thing that I could think of causing
> a problem is the test to make sure that the instruction is not inside
> a load exclusive/store exclusive region.  The test might be mistaking
> some of the data before the start of the function as load exclusive
> instructions.


I verified that the cause of kprobes not being registered is the scan
backward for load exclusive instructions.  For one example have:

...
fffffc00080c98cc:	d503201f 	nop
fffffc00080c98d0:	08dc4c80 	.word	0x08dc4c80
fffffc00080c98d4:	fffffc00 	.word	0xfffffc00

fffffc00080c98d8 <SyS_set_tid_address>:
fffffc00080c98d8:	a9be7bfd 	stp	x29, x30, [sp,#-32]!
fffffc00080c98dc:	910003fd 	mov	x29, sp

The previous function has 0xfffffc0008dc4c80 as data at the end of the
function. The scan backwards from the beginning of the current
function Sys_set_tid_address stumbles into that data and interprets
the 0x08dc4c80 as load exclusive instructions.  This causes the kprobe
registration to fail.

Disabled the is_probed_address_atomic() scan for atomic instructions
allows the test to work:

make installcheck RUNTESTFLAGS="--debug systemtap.base/bz1027459.exp"
...
Running target unix
Running /root/systemtap_write/systemtap/testsuite/systemtap.base/bz1027459.exp ...
PASS: bz1027459 -p5

		=== systemtap Summary ===

# of expected passes		1


Somehow the is_probed_address_atomic and arm_kprobe_decode_insn
functions need to avoid scanning past the beginning of a function.

-Will