The use of userspace markers in a shared library causes the program using the shared library to crash on F-12. This can be observed with a current git checkout of systemtap: Observed this when attempting to use markers in python, which are in a shared library. The instrumented python srpm is at http://people.redhat.com/wcohen/fudcon2009/python-2.6.2-5.fc12.src.rpm When a script using the markers in the python shared library (problem triggered with attached pytrace.stp script), applications using the shared library do fail. 1. In one window start the systemtap script with: stap /home/wcohen/research/profiling/pytrace.stp 2. Attempt to run python with: $ python Bus error (core dumped) The expected result would be the systemtap script printing out information as python runs. A systemtap RPM was built from the systemtap git repository and installed on the system. The kernel was kernel-2.6.31.9-174.fc12.x86_64 The problem appears to be related to the following git checking: http://sources.redhat.com/git/gitweb.cgi?p=systemtap.git;a=commit;h=93c8419165a7c651989cc344364b86695e1f142c
Created attachment 4495 [details] A short systemtap script to trigger the problem This script is just suppose to print out information and should not interfere with the operation of a python program running in userspace.
When attempting the run /usr/bin/python under gdb get the following: $ gdb /usr/bin/python GNU gdb (GDB) Fedora (7.0.1-19.fc12) Copyright (C) 2009 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu". For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>... Reading symbols from /usr/bin/python...Reading symbols from /usr/lib/debug/usr/bin/python.debug...done. done. (gdb) run Starting program: /usr/bin/python During startup program terminated with signal SIGBUS, Bus error. (gdb)
Created attachment 4496 [details] Script to find the address of the sigbus faults To allow comparison between runs address randomization was turned off with: sysctl -w kernel.randomize_va_space=0 The attached sigbus.stp script was run: $ stap /tmp/sigbus.stp |grep sig 2735979:32542:0x00007ffff7fc965c:r:sigbus:2 2736032:32542:0x00007ffff7fc965e:r:sigbus:2 Got a pmap of a python process with the problem script not running. 00007ffff7c22000 1464K r-x-- /usr/lib64/libpython2.6.so.1.0 00007ffff7d90000 2044K ----- /usr/lib64/libpython2.6.so.1.0 00007ffff7f8f000 236K rw--- /usr/lib64/libpython2.6.so.1.0 00007ffff7fca000 56K rw--- [ anon ] Those addresses appear to be close to the end of the rw region of /usr/lib64/libpython2.6.so.1.0 $ nm /usr/lib/debug//usr/lib64/libpython2.6.so.1.0.debug |grep sema 00000000003a765c B function__entry_semaphore 00000000003a765e B function__return_semaphore The problem appears to be the code writing to function__entry_semaphore and function__return_semaphore is causing the sigbus error. The number of faults is equal to the number of probes on a point. From the sigbus.stp script output it looks like a get_user() macro is causing the sigbus.
Created attachment 4500 [details] Output of script with -DDEBUG_TASK_FINDER_VMA Ran the script with following command line to get idea what is going on in taskfinder: /home/wcohen/research/profiling/pytrace.stp -DDEBUG_TASK_FINDER_VMA -c python >& /tmp/x
Created attachment 4502 [details] scox's proposed patch to address this issue This is the proposed patch that scox developed to address the problem of the get_user() occurring at the wrong time. The userspace markers in python works with this patch.
The old implementation of userspace probing used __access_process_vm() in the runtime access_process_vm.h. This function used copy_to_user_page(). On x86 machine cache consistency is hardware enforced. The resulting copy_to_user_page() ends up be just a memcpy(). However, On the ia64 (and powerpc) the code needs to take some additional steps to make sure the cache and memory are consistent. This results in the ia64 (and powerpc) copy_to_user_page including a flush_icache_user_range(): #define copy_to_user_page(vma, page, vaddr, dst, src, len) \ do { memcpy(dst, src, len); \ flush_icache_user_range(vma, page, vaddr, len); \ } while (0) There is some discussion about the copy_to_user_page() and cache coherence at: http://rhkernel.org/RHEL5+2.6.18-8.el5/Documentation/cachetlb.txt#L352
(In reply to comment #6) > However, On the ia64 (and powerpc) the code needs to take some additional steps > to make sure the cache and memory are consistent. But note that this should not apply to our data-space semaphore accesses. We could fork another version of __access_process_vm that uses memcpy instead of copy_to_user_page().
Use __access_process_vm_noflush for static user semaphore decrement. commit: 3c5b8e2b99