I have a module that simply puts a kprobe on sys_getuid() and another program that calls getuid() in a tight loop. Removing the module while the test program is running will usually result in an immediate kernel crash on my dual-processor 64-bit Xeon system running RHEL4U2 (2.6.9-15.ELsmp). On FC4-smp kernels I have tried, removing the module with the kprobe crashes the test program, but not the kernel. x86_64 up kernels and i686 kernels (both up and smp) seem OK. However I don't have a real i686 smp system, only a hyperthreaded cpu.
Created attachment 612 [details] Test module and load program Unpack the tar file. Type "make". as root, run "./please_crash_me"
Does the equivalent systemtap script have the same effect? Systemtap's lifecycle control logic is intended to prevent concurrent kprobe execution and removal. global n probe kernel.function("sys_getuid") { n++ }
FWIW, I tried the C tarball modules on a 2proc ppc64 lpar without issues.
(In reply to comment #2) > Does the equivalent systemtap script have the same effect? Systemtap's > lifecycle control logic is intended to prevent concurrent kprobe execution and > removal. I verified it also crashes. I have never understood your argument that you can prevent concurrent kprobe execution and removal from outside kprobes. The generated code only prevents the core logic from being executed. There is still a window of vulnerability after the kprobe is triggered and before it checks the atomic session state and determines it should not be executing.
I ran the test on IA64 where I had RHEL4 U2 2.6.9-16.EL and I did not see any problem. The test FINISHED OK.
Crashes very promptly on a dual-CPU AMD-64 (elm3b30) running RHEL4 U2. Doesn't crash on my Pentium M uniprocessor.
Created attachment 625 [details] Patch to fix #1234 on x86_64 Here's a patch that appears to fix the problem on x86_64. With this patch, the please_crash_me script runs to completion, producing the expected output in /var/log/messages: Aug 27 02:14:57 elm3b30 kernel: kprobe registered Aug 27 02:14:59 elm3b30 kernel: sys_getuid() called 2207424 times. Aug 27 02:14:59 elm3b30 kernel: kprobe registered Aug 27 02:15:01 elm3b30 kernel: sys_getuid() called 2156331 times. ... Aug 27 02:18:18 elm3b30 kernel: kprobe registered Aug 27 02:18:20 elm3b30 kernel: sys_getuid() called 2171258 times. This patch is intended for vanilla v2.6.13-rc5-mm1, but should apply (perhaps with a bit of an offset) to RHEL4 U2 as well. When a kprobe gets unregistered between when we hit the probepoint and when we go looking for the associated kprobe object, we need to let the CPU continue as if the probepoint hadn't been hit. We were trying to do that, but neglecting to set the IP back to the beginning of the probed instruction. We haven't been able to reproduce this bug on the other architectures. Theoretically, though, I think every architecture's version of kprobe_handler() needs to be fixed in this same way. Being in the middle of a 4-day weekend, I don't have time to do those patches and the associated testing. Perhaps Ananth, Prasanna, Kevin, and/or Anil would like to take a crack at it. If not, I'm in the phone book. Given very bad luck, I think you could see this bug on any multiprocessor. Your chance of hitting it increases with the frequency of the probe hits. This bug crashed elm3b90 when it was running RHEL4 U2, but when it ran v2.6.13-rc5-mm1, the bug consistently just caused an oops or two and killed the itest process.
Changing resolution to FIXED.
Added Anil, Prasanna and myself to the cc list. Anil, Prasanna, Please review Jim's fix and test it on ia64 and ia32 respectively. I'll give it a spin on ppc64 on Monday. Thanks, Ananth
Adding Hien to the cc list so he can followup with the fix when he is back from vacation.
(In reply to comment #9) > Added Anil, Prasanna and myself to the cc list. > Anil, Prasanna, > Please review Jim's fix and test it on ia64 and ia32 respectively. I'll give it > a spin on ppc64 on Monday. The above patch looks fine to me and the same is need for ia32 also. IA64 does not need the above fix since IP does not gets incremented when break instruction is encountered and hence IA64 need not have to correct the IP. As mentioned earlier, IA64 has no probelems and the test runs fine. Not sure whether ppc64 needs the above fix?
I just verified, as with IA64, we don't need this on PPC64 either. We don't advance the instruction pointer to the next instruction on PPC64 too.
I verified the patch fixes the problem for me. (RHEL4U2 dual processor x86_64)
Jim, I verified the patch on i386 4-way SMP box, it fixes the problem. Thanks prasanna
Created attachment 639 [details] Patch against RHEL4 U2 to fix #1234 on i386 and x86_64 As noted above, the problem doesn't exist for ia64 or ppc64. Dave Miller confirmed that it's not a problem for sparc64. That leaves i386 and x86_64, for which I've created and tested patches for RHEL4 U2 (attached) and v2.6.13. The v2.6.13 patch has been accepted into the -mm tree.
Created attachment 1760 [details] spam