Bug 11140 - SystemTap userspace marker in shared libraries cause probed program crash
Summary: SystemTap userspace marker in shared libraries cause probed program crash
Status: RESOLVED FIXED
Alias: None
Product: systemtap
Classification: Unclassified
Component: runtime (show other bugs)
Version: unspecified
: P2 critical
Target Milestone: ---
Assignee: Unassigned
URL:
Keywords:
Depends on:
Blocks: blockers-1.1
  Show dependency treegraph
 
Reported: 2010-01-05 21:44 UTC by William Cohen
Modified: 2010-01-14 16:07 UTC (History)
1 user (show)

See Also:
Host:
Target:
Build:
Last reconfirmed:


Attachments
A short systemtap script to trigger the problem (184 bytes, text/plain)
2010-01-05 21:46 UTC, William Cohen
Details
Script to find the address of the sigbus faults (394 bytes, text/plain)
2010-01-05 22:42 UTC, William Cohen
Details
Output of script with -DDEBUG_TASK_FINDER_VMA (996 bytes, text/plain)
2010-01-06 16:15 UTC, William Cohen
Details
scox's proposed patch to address this issue (3.18 KB, patch)
2010-01-07 18:43 UTC, William Cohen
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description William Cohen 2010-01-05 21:44:40 UTC
The use of userspace markers in a shared library causes the program
using the shared library to crash on F-12. This can be observed with a
current git checkout of systemtap:

Observed this when attempting to use markers in python, which are in a
shared library.  The instrumented python srpm is at

http://people.redhat.com/wcohen/fudcon2009/python-2.6.2-5.fc12.src.rpm

When a script using the markers in the python shared library (problem
triggered with attached pytrace.stp script), applications using the
shared library do fail.


1. In one window start the systemtap script with:

 stap  /home/wcohen/research/profiling/pytrace.stp

2. Attempt to run python with:

$ python
Bus error (core dumped)

The expected result would be the systemtap script printing out
information as python runs.

A systemtap RPM was built from the systemtap git repository and
installed on the system.

The kernel was kernel-2.6.31.9-174.fc12.x86_64

The problem appears to be related to the following git checking:

http://sources.redhat.com/git/gitweb.cgi?p=systemtap.git;a=commit;h=93c8419165a7c651989cc344364b86695e1f142c
Comment 1 William Cohen 2010-01-05 21:46:21 UTC
Created attachment 4495 [details]
A short systemtap script to trigger the problem

This script is just suppose to print out information and should not interfere
with the operation of a python program running in userspace.
Comment 2 William Cohen 2010-01-05 21:50:53 UTC
When attempting the run /usr/bin/python under gdb get the following:

$ gdb /usr/bin/python
GNU gdb (GDB) Fedora (7.0.1-19.fc12)
Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /usr/bin/python...Reading symbols from
/usr/lib/debug/usr/bin/python.debug...done.
done.
(gdb) run
Starting program: /usr/bin/python 
During startup program terminated with signal SIGBUS, Bus error.
(gdb) 
Comment 3 William Cohen 2010-01-05 22:42:45 UTC
Created attachment 4496 [details]
Script to find the address of the sigbus faults

To allow comparison between runs address randomization was turned off with:

sysctl -w kernel.randomize_va_space=0

The attached sigbus.stp script was run:


$ stap /tmp/sigbus.stp |grep sig
2735979:32542:0x00007ffff7fc965c:r:sigbus:2
2736032:32542:0x00007ffff7fc965e:r:sigbus:2

Got a pmap of a python process with the problem script not running.

00007ffff7c22000   1464K r-x--	/usr/lib64/libpython2.6.so.1.0
00007ffff7d90000   2044K -----	/usr/lib64/libpython2.6.so.1.0
00007ffff7f8f000    236K rw---	/usr/lib64/libpython2.6.so.1.0
00007ffff7fca000     56K rw---	  [ anon ]

Those addresses appear to be close to the end of the rw region of
/usr/lib64/libpython2.6.so.1.0

$ nm /usr/lib/debug//usr/lib64/libpython2.6.so.1.0.debug |grep sema
00000000003a765c B function__entry_semaphore
00000000003a765e B function__return_semaphore

The problem appears to be the code writing to
 function__entry_semaphore and function__return_semaphore is causing
 the sigbus error.  The number of faults is equal to the number of
 probes on a point.  

From the sigbus.stp script output it looks like a get_user() macro is causing
the sigbus.
Comment 4 William Cohen 2010-01-06 16:15:23 UTC
Created attachment 4500 [details]
Output of script with -DDEBUG_TASK_FINDER_VMA

Ran the script with following command line to get idea what is going on in
taskfinder:

/home/wcohen/research/profiling/pytrace.stp -DDEBUG_TASK_FINDER_VMA -c python
>& /tmp/x
Comment 5 William Cohen 2010-01-07 18:43:33 UTC
Created attachment 4502 [details]
scox's proposed patch to address this issue

This is the proposed patch that scox developed to address the problem of the
get_user() occurring at the wrong time. The userspace markers in python works
with this patch.
Comment 6 William Cohen 2010-01-12 01:16:38 UTC
The old implementation of userspace probing used __access_process_vm() in the
runtime access_process_vm.h. This function used copy_to_user_page().

On x86 machine cache consistency is hardware enforced. The resulting
copy_to_user_page() ends up be just a memcpy().

However, On the ia64 (and powerpc) the code needs to take some additional steps
to make sure the cache and memory are consistent. This results in the ia64 (and
powerpc) copy_to_user_page including a flush_icache_user_range():

#define copy_to_user_page(vma, page, vaddr, dst, src, len) \
do { memcpy(dst, src, len); \
     flush_icache_user_range(vma, page, vaddr, len); \
} while (0)

There is some discussion about the copy_to_user_page() and cache coherence at:

http://rhkernel.org/RHEL5+2.6.18-8.el5/Documentation/cachetlb.txt#L352

Comment 7 Frank Ch. Eigler 2010-01-12 13:12:42 UTC
(In reply to comment #6)
> However, On the ia64 (and powerpc) the code needs to take some additional steps
> to make sure the cache and memory are consistent. 

But note that this should not apply to our data-space semaphore accesses.

We could fork another version of __access_process_vm that uses memcpy instead
of copy_to_user_page().

Comment 8 Stan Cox 2010-01-14 16:07:59 UTC
Use __access_process_vm_noflush for static user semaphore decrement.
commit: 3c5b8e2b99