Re: Segmentation Fault: Task died at uprobe probepoint

[Gaurav Dhiman <dimanuec at gmail dot com>]

On Tue, Jun 15, 2010 at 2:23 AM, Mark Wielaard <mjw@redhat.com> wrote:
> On Mon, 2010-06-14 at 20:52 -0700, Gaurav Dhiman wrote:
>> On Mon, Jun 14, 2010 at 6:33 PM, Frank Ch. Eigler <fche@redhat.com> wrote:
>> > Gaurav Dhiman <dimanuec@gmail.com> writes:
>> >> [...]
>> >> Following are the logs from the Pass 5 of the run with stap -vvvt
>> >> -DDEBUG_UPROBES:
>> >>
>> >> Pass 5: starting run.
>> >> [...]
>> >> stapio:stp_main_loop:571 probe_start() returned 0
>> >> stap_uprobe_change_plus:67: +uprobe spec 0 idx 0 process uprobe-test[25763] addr 0000000000400517 pp process("/home/gdhiman/tests/uprobe-test").function("test_func@/home/gdhiman/tests/uprobe-test.c:4")
>> >> test begin!!
>> >> test_func called!
>> >> i=0x5
>> >> stap_uprobe_change_minus:220: -uprobe spec 0 idx 0 process uprobe-test[25763] reloc 0000000000400517 pp process("/home/gdhiman/tests/uprobe-test").function("test_func@/home/gdhiman/tests/uprobe-test.c:4")
>> >>
>> >> The program receives a SIGSEGV signal at this point.
>> >
>> > OK, something is wrong here. ?See that -uprobe line? ?It means stap
>> > has for some reason determined that it's time to remove the uprobe,
>> > but that should not happen until either the script or the target
>> > program exits.
>> >
>> > This will make things more verbose, but could you try adding
>> > -DDEBUG_TASK_FINDER and -DDEBUG_TASK_FINDER_VMA to the stap
>> > command line?
>> >
>> You are right, that looks suspicious; it is actually removing the
>> probe, when it should not be doing it. Here is the more verbose
>> output:
>>
>> test_func called!
>> i=0x5
>> stap_uprobe_process_found:246: -proc pid 27237 stf ffffffffa001dce0
>> ffffffffa001dce0 path /home/gdhiman/tests/uprobe-test
>> stap_uprobe_change_minus:220: -uprobe spec 0 idx 0 process
>> uprobe-test[27237] reloc 0000000000400517 pp
>> process("/home/gdhiman/tests/uprobe-test").function("test_func@/home/gdhiman/tests/uprobe-test.c:4")
>> _stp_tf_exec_cb:29: tsk 27237:27237 , register_p: 0, process_p: 1
>
> Unless I am reading the logs wrong, I do think this is expected. The
> process gets a SIGSEGV and so uprobes is notified that it needs to clean
> up (-proc pid 27237 means process is gone).
>
>> > I meant the uprobe-test binary, just to see what instructions were
>> > in the vicinity of the test_func function. ?Just that area.
>> >
>>
>> Here it is:
>>
>> 000000000040050c <test_func>:
>> ? 40050c: ? ? ? 55 ? ? ? ? ? ? ? ? ? ? ?push ? %rbp
>> ? 40050d: ? ? ? 48 89 e5 ? ? ? ? ? ? ? ?mov ? ?%rsp,%rbp
>> ? 400510: ? ? ? 48 83 ec 10 ? ? ? ? ? ? sub ? ?$0x10,%rsp
>> ? 400514: ? ? ? 89 7d fc ? ? ? ? ? ? ? ?mov ? ?%edi,-0x4(%rbp)
>> ? 400517: ? ? ? 8b 75 fc ? ? ? ? ? ? ? ?mov ? ?-0x4(%rbp),%esi
>> ? 40051a: ? ? ? bf 3c 06 40 00 ? ? ? ? ?mov ? ?$0x40063c,%edi
>> ? 40051f: ? ? ? b8 00 00 00 00 ? ? ? ? ?mov ? ?$0x0,%eax
>> ? 400524: ? ? ? e8 cf fe ff ff ? ? ? ? ?callq ?4003f8 <printf@plt>
>> ? 400529: ? ? ? 8b 45 fc ? ? ? ? ? ? ? ?mov ? ?-0x4(%rbp),%eax
>> ? 40052c: ? ? ? c9 ? ? ? ? ? ? ? ? ? ? ?leaveq
>> ? 40052d: ? ? ? c3 ? ? ? ? ? ? ? ? ? ? ?retq
>
> That does look OK, even though it looks different from my local build:
>
> Dump of assembler code for function test_func:
> ? 0x080483d0 <+0>: ? ? push ? %ebp
> ? 0x080483d1 <+1>: ? ? mov ? ?%esp,%ebp
> ? 0x080483d3 <+3>: ? ? push ? %ebx
> ? 0x080483d4 <+4>: ? ? sub ? ?$0x14,%esp
> ? 0x080483d7 <+7>: ? ? mov ? ?0x8(%ebp),%ebx
> ? 0x080483da <+10>: ? ?movl ? $0x80484f4,(%esp)
> ? 0x080483e1 <+17>: ? ?mov ? ?%ebx,0x4(%esp)
> ? 0x080483e5 <+21>: ? ?call ? 0x80482f4 <printf@plt>
> ? 0x080483ea <+26>: ? ?mov ? ?%ebx,%eax
> ? 0x080483ec <+28>: ? ?add ? ?$0x14,%esp
> ? 0x080483ef <+31>: ? ?pop ? ?%ebx
> ? 0x080483f0 <+32>: ? ?pop ? ?%ebp
> ? 0x080483f1 <+33>: ? ?ret
> End of assembler dump.
>
> What compiler and flags did you use? Here gcc version 4.4.4 20100503
> (Red Hat 4.4.4-2) with -g -O2.
>

I am using gcc 4.4.3 and using -g -O2 as well.

> If possible could you set ulimit -c unlimited and catch the core file of
> the program and give the disassembly from test_func in the core? Maybe
> the breakpoint instruction in there was misplaced?

I will try to send this tomorrow.

Thanks,
-Gaurav