in stable 4.4.131 kprobes seems to be broken
Victor Kamensky
kamensky@cisco.com
Tue Jul 10 07:34:00 GMT 2018
On Tue, 10 Jul 2018, Greg Kroah-Hartman wrote:
> [Adding the stable list]
>
> On Mon, Jul 09, 2018 at 11:53:37PM -0700, Victor Kamensky wrote:
>> Hi Masami,
>>
>> I am not sure maybe you are aware about this issue already, but
>> I am writing in case if you are not.
>>
>> After move to 4.4.131 stable very basic SystemTap script that involves
>> kprobes stopped working - it crashes kernel.
>>
>> The SystemTap script like this:
>>
>> probe kernel.function("meminfo_proc_show") {
>> println("meminfo_proc_show called")
>> }
>>
>> which effectively will translate into kprobe entry in meminfo_proc_show
>> function, crashes like this.
>>
>> BUG: unable to handle kernel paging request at ffffffffc063f002
>> IP: [<ffffffffa3c3558b>] resume_execution+0xe5/0x151
>> PGD 24613067 ca
>> PUD 24615067 PMD 1523e9067 PTE 3370d061
>> Oops: 0003 [#1] SMP Modules linked in: <snip>
>> CPU: 3 PID: 20078 Comm: <snip> Tainted: G O 4.4.131 #1
>> task: ffff880031ead780 ti: ffff880032238000 task.ti: ffff880032238000
>> t /RIP: 0010:[<ffffffffa3c3558b>] [<ffffffffa3c3558b>] resume_execution+0xe5/0x151
>> RSP: 0018:ffff88017fccaec8 EFLAGS: 00010006
>> RAX: ffffffffa3dad154 RBX: ffffffffa3dad152 RCX: 000000003f9c0ff9
>> RDX: ffffffffc063f002 RSI: ffff88017fccaf58 RDI: 0000000000000041
>> RBP: ffff8801549f83c0 R08: 0000000000000001 R09: 0000000000000001
>> R10: ffff88003223bdd8 R11: 0000000000000246 R12: ffff88003223bdd8
>> R13: ffff88017fccaf58 R14: ffffffffc063f000 R15: ffff88017fccee40
>> FS: 00007f9f34520240(0000) GS:ffff88017fcc0000(0000) knlGS:0000000000000000
>> CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>> CR2: ffffffffc063f002 CR3: 0000000152b46000 CR4: 0000000000100670
>> Stack:
>> ffff88017fccee40 ffff88017fccaf58 ffff8801549f83c0 0000000000000000
>> ffff880033cde500 0000000000000001 ffffffffa3c35633 ffff88017fccaf58
>> ffff880031ead780 0000000000000000 ffffffffa3c04e9d 0000000000004000
>> Call Trace:
>> <#DB>
>> [<ffffffffa3c35633>] ? kprobe_debug_handler+0x3c/0xc8
>> [<ffffffffa3c04e9d>] ? do_debug+0x7a/0x182
>> [<ffffffffa4163afe>] ? debug+0x3e/0x70
>> [<ffffffffa3dad14d>] ? meminfo_proc_open+0x1c/0x1c
>> [<ffffffffa3d73111>] ? seq_buf_alloc+0x23/0x3c
>> <<EOE>>
>> [<ffffffffa3d735a8>] ? seq_read+0x1a6/0x40d
>> [<ffffffffa3da5700>] ? proc_reg_read+0x47/0x65
>> [<ffffffffa3da56b9>] ? proc_reg_write+0x65/0x65
>> [<ffffffffa3d55ebf>] ? __vfs_read+0x34/0xea
>> [<ffffffffa3d564f9>] ? rw_verify_area+0x7a/0xc9
>> [<ffffffffa3d565dc>] ? vfs_read+0x94/0xf9
>> [<ffffffffa3d570a4>] ? SyS_read+0x61/0xb4
>> [<ffffffffa4161b8a>] ? entry_SYSCALL_64_fastpath+0x1e/0x8e
>> Code: 7d 70 00 75 3e 49 8b 95 80 00 00 00 49 39 d6 73 2b 48 89 d0 4c 29 f0
>> 48 8d 48 05 48 83 f9 0e 77 1b b9 fb ff ff ff 48 01 d8 29 d1 <c6> 02 e9 01 c8
>> 89 42 01 c7 45 70 01 00 00 00 eb 07 c7 45 70 ff RIP [<ffffffffa3c3558b>]
>> resume_execution+0xe5/0x151
>> RSP <ffff88017fccaec8>
>> CR2: ffffffffc063f002
>>
>> Decoding crash:
>>
>> (gdb) p &resume_execution
>> $1 = (void (*)(struct kprobe *, struct pt_regs *,
>> struct kprobe_ctlblk *)) 0xffffffff810354a6 <resume_execution>
>> (gdb) b *(0xffffffff810354a6 + 0xe5)
>> Breakpoint 1 at 0xffffffff8103558b: file <snip>/kernel-source/arch/x86/kernel/kprobes/core.c, line 126.
>>
>> corresponds to:
>>
>> static nokprobe_inline void
>> __synthesize_relative_insn(void *from, void *to, u8 op)
>> {
>> struct __arch_relative_insn {
>> u8 op;
>> s32 raddr;
>> } __packed *insn;
>>
>> insn = (struct __arch_relative_insn *)from;
>> insn->raddr = (s32)((long)(to) - ((long)(from) + 5));
>> insn->op = op; <----------------------------------------
>> }
>>
>>
>> (gdb) p /x &kprobe_debug_handler
>> $1 = 0xffffffff810355f7
>> (gdb) b *(0xffffffff810355f7 + 0x3c)
>> Breakpoint 1 at 0xffffffff81035633: file <snip>/kernel-source/arch/x86/kernel/kprobes/core.c, line 932.
>>
>> corresponds to
>>
>> static void resume_execution(struct kprobe *p, struct pt_regs *regs,
>> struct kprobe_ctlblk *kcb)
>> {
>> <snip>
>> if (p->ainsn.boostable == 0) {
>> if ((regs->ip > copy_ip) &&
>> (regs->ip - copy_ip) + 5 < MAX_INSN_SIZE) {
>> /*
>> * These instructions can be executed directly if it
>> * jumps back to correct address.
>> */
>> synthesize_reljump((void *)regs->ip,
>> (void *)orig_ip + (regs->ip - copy_ip)); <-------
>> p->ainsn.boostable = 1;
>> } else {
>> p->ainsn.boostable = -1;
>> }
>> }
>>
>> Further digging points to the following couple commits that 4.4.y stable
>> picked up:
>>
>> 1dd26ec7 kprobes/x86: Fix to set RWX bits correctly before releasing trampoline
>> 76bee4 kprobes/x86: Set kprobes pages read-only
>>
>> Effectively the way I read: the problem is that linux-4.4.y branch is
>> missing your following commit
>>
>> commit 804dec5bda9b4fcdab5f67fe61db4a0498af5221
>> Author: Masami Hiramatsu <mhiramat@kernel.org>
>> Date: Wed Mar 29 14:00:25 2017 +0900
>>
>> kprobes/x86: Do not modify singlestep buffer while resuming
>>
>> Do not modify singlestep execution buffer (kprobe.ainsn.insn)
>> while resuming from single-stepping, instead, modifies
>> the buffer to add a jump back instruction at preparing
>> buffer.
>>
>> Signed-off-by: Masami Hiramatsu <mhiramat@kernel.org>
>> Cc: Ananth N Mavinakayanahalli <ananth@linux.vnet.ibm.com>
>>
>> and as result synthesize_reljump function call still exists from
>> resume_execution function but it does not have proper page permission
>> manipulation code around it, so making kprobes pages as
>> read-only has averse effect: I.e when called from resume_execution
>> function it hits read-only page and crashes.
>>
>> Please advise how to proceed besides obvious backing out above mentioned
>> two commits.
>
> So if you apply the above commit, all works properly for you?
That commit seems to be part of some x86 kprobes restructuring, I
hesitated to try it. After your email tried to cherry-pick to 4.4.y
and it does not go in smoothly at all, quite a bit of conflicts.
I may not phrase it correctly: My point was that in 4.4 there
are more places where synthesize_reljump function is called, which
latter got removed. And this function modifies kprobes page, so
lifting up read-only protection needs to be done in more places
compared to latter kernels. Whether it can be done correctly and
proper thing to do in do_debug/resume_execution function
exception processing context I cannot tell.
Thanks,
Victor
> thanks,
>
> greg k-h
>
More information about the Systemtap
mailing list