This is the mail archive of the
systemtap@sourceware.org
mailing list for the systemtap project.
Re: Problem with vfs probe on Proxmox kernel
On 26/01/17 20:05, David Smith wrote:
> On 01/26/2017 12:42 PM, Adam Guderski wrote:
>> On 26/01/17 18:37, David Smith wrote:
>>> On 01/25/2017 04:44 PM, Adam Guderski wrote:
>>>> Hello,
>>>>
>>>> I've got problem running sample traceio.stp script (obtained from
>>>> https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html-single/SystemTap_Beginners_Guide/#traceio2)
>>>> on Proxmox VE 4.4 system. Proxmox is a Debian-based Linux distro which
>>>> uses a custom Ubuntu-based kernel. The Proxmox repository does not
>>>> provide a kernel with debug symbols, so I had to compile it from
>>>> provided sources - version 4.4.35-2-pve, to be specific. Debian repos
>>>> (jessie) provide systemtap in version 2.9, which from what I read does
>>>> not support 4.4 kernels, so I compiled systemtap from sources (from
>>>> master branch, version 3.1/0.159).
>>>>
>>>> With such setup, I'm able to run sample "hello world" stap script and
>>>> couple of others, but all scripts with vfs probes fail. I attach full
>>>> verbose (-vvv) output here: http://pastebin.com/DiWMsrtr, but I think
>>>> the main error can be explained below. What can I do about it? I
>>>> searched for couple of hours but did not find anything that would help.
>>>
>>> As the following states:
>>>
>>>> # stap -v traceio2.stp 0xfb00
>>>> Pass 1: parsed user script and 119 library scripts using
>>>> 71608virt/43716res/4464shr/39904data kb, in 140usr/0sys/132real ms.
>>>> semantic error: while resolving probe point: identifier 'kernel' at
>>>> /usr/local/share/systemtap/tapset/linux/vfs.stp:987:19
>>>> source: probe vfs.write = kernel.function("vfs_write")
>>>> ^
>>>>
>>>> semantic error: missing x86_64 kernel/module debuginfo [man
>>>> warning::debuginfo] under '/lib/modules/4.4.35-2-pve/build'
>>>
>>> Note that a systemtap "hello world" program doesn't need kernel
>>> debuginfo, but a script using vfs probes does need kernel debuginfo.
>>> Does any script that needs debuginfo work, or is it just vfs probes? To
>>> test, try 'stap -l 'kernel.function("*_write")'. You should see items
>>> like 'sys_write' and 'vfs_write' in the output.
>>
>> Sadly, the command "stap -l 'kernel.function("*_write")'" returns
>> nothing at all.
>
> That's actually good. This definitely means that stap can't find any
> debuginfo (vs. the problem being stap found the debuginfo but didn't
> find vfs_write() in the debuginfo).
>
>>> systemtap is expecting to find the kernel's debuginfo at
>>> /lib/modules/4.4.35-2-pve/build. Is it there? If not, you'll either need
>>> to move it there or use the 'SYSTEMTAP_DEBUGINFO_PATH' environment
>>> variable to tell systemtap where you've put the debuginfo.
>>
>> Could you elaborate how debuginfo files look like? This is what I've got
>> in the directory:
>
> For unstripped userspace exes, the debuginfo is present in the exe. If
> you probe an unstripped userspace exe, systemtap shouldn't have any
> trouble finding the debuginfo. To make the exes smaller, the debuginfo
> can be extracted and put in another file, typically in
> /usr/lib/debug/EXEPATH.debug. So, if you install the debuginfo for
> /bin/ls, you'll find its debuginfo in /usr/lib/debug/bin/ls.debug.
>
> For kernels, things are a bit more tricky. You typically boot off a
> compressed stripped kernel image, most likely called
> /boot/vmlinuz-KERNEL_VER (where 'KERNEL_VER' is the kernel version). The
> debuginfo is typically stored in an unstripped uncompressed kernel file
> down in /usr/lib/debug. Here's on a Fedora 25 system I see:
>
> ====
> # ls -l /boot/vmlinuz-4.9.5-200.fc25.x86_64
> /usr/lib/debug/lib/modules/4.9.5-200.fc25.x86_64/vmlinux
> -rwxr-xr-x. 1 root root 6891144 Jan 20 07:43
> /boot/vmlinuz-4.9.5-200.fc25.x86_64
> -rwxr-xr-x. 1 root root 541141280 Jan 20 07:48
> /usr/lib/debug/lib/modules/4.9.5-200.fc25.x86_64/vmlinux
> # file /boot/vmlinuz-4.9.5-200.fc25.x86_64
> /usr/lib/debug/lib/modules/4.9.5-200.fc25.x86_64/vmlinux
> /boot/vmlinuz-4.9.5-200.fc25.x86_64: Linux kernel
> x86 boot executable bzImage, version 4.9.5-200.fc25.x86_64
> (mockbuild@bkernel02.phx2.fedoraproject.o, RO-rootFS, swap_dev 0x6,
> Normal VGA
> /usr/lib/debug/lib/modules/4.9.5-200.fc25.x86_64/vmlinux: ELF 64-bit LSB
> executable, x86-64, version 1 (SYSV), statically linked,
> BuildID[sha1]=05af643a7230786a9f0fe4fc1c05db0be64ef44c, not stripped
> ====
>
> As you can see the vmlinux file is much bigger than the vmlinuz file,
> since it is unstripped and uncompressed.
>
> Now those paths are all for distro files, and you've got a self-built
> kernel. But, the principle is still the same.
>
>> # ls -la /lib/modules/4.4.35-2-pve/build/
>> total 1564
>> drwxr-xr-x 26 root root 4096 Jan 22 02:09 .
>> drwxr-xr-x 3 root root 4096 Jan 18 00:52 ..
>> drwxr-xr-x 33 root root 4096 Jan 22 02:09 arch
>> drwxr-xr-x 3 root root 4096 Jan 22 02:09 block
>> drwxr-xr-x 2 root root 4096 Jan 22 02:09 certs
>> -rw-r--r-- 1 root root 189740 Jan 22 01:29 .config
>> drwxr-xr-x 4 root root 4096 Jan 22 02:09 crypto
>> drwxr-xr-x 127 root root 4096 Jan 22 02:09 drivers
>> drwxr-xr-x 2 root root 4096 Jan 22 02:09 firmware
>> drwxr-xr-x 74 root root 4096 Jan 22 02:09 fs
>> drwxr-xr-x 30 root root 4096 Jan 22 02:09 include
>> drwxr-xr-x 2 root root 4096 Jan 22 02:09 init
>> drwxr-xr-x 2 root root 4096 Jan 22 02:09 ipc
>> -rw-r--r-- 1 root root 2622 Dec 22 08:43 Kbuild
>> -rw-r--r-- 1 root root 252 Dec 22 08:43 Kconfig
>> drwxr-xr-x 15 root root 4096 Jan 22 02:09 kernel
>> drwxr-xr-x 12 root root 4096 Jan 22 02:09 lib
>> -rw-r--r-- 1 root root 56164 Jan 22 00:46 Makefile
>> drwxr-xr-x 3 root root 4096 Jan 22 02:09 mm
>> -rw-r--r-- 1 root root 1218952 Jan 22 01:29 Module.symvers
>> drwxr-xr-x 60 root root 4096 Jan 22 02:09 net
>> drwxr-xr-x 16 root root 4096 Jan 22 02:09 samples
>> drwxr-xr-x 13 root root 20480 Jan 22 02:09 scripts
>> drwxr-xr-x 9 root root 4096 Jan 22 02:09 security
>> drwxr-xr-x 23 root root 4096 Jan 22 02:09 sound
>> drwxr-xr-x 10 root root 4096 Jan 22 02:09 spl
>> drwxr-xr-x 21 root root 4096 Jan 22 02:09 tools
>> drwxr-xr-x 8 root root 4096 Jan 22 02:09 ubuntu
>> drwxr-xr-x 2 root root 4096 Jan 22 02:09 usr
>> drwxr-xr-x 4 root root 4096 Jan 22 02:09 virt
>> drwxr-xr-x 13 root root 4096 Jan 22 02:09 zfs
>>
>> Am I missing something? Are debuginfo files output somewhere else during
>> compilation from source?
>
> Based on the paths stored when compiling your kernel, systemtap is
> looking for the vmlinux file in '/lib/modules/4.4.35-2-pve/build/'.
> Obviously it isn't there. Can you search around and see if you can find
> it and link it to its proper place?
I can't find it (find / -type f -name '*vmlinux*'). When I compiled from
source, I noticed this lines in Makefile for Proxmox VE kernel:
# strip debug info
find tmp/lib/modules -name \*.ko -print | while read f ; do strip
--strip-debug "$$f"; done
I thought that when I comment this out my kernel will have debug info
compiled into it. Interestingly, the config file for my kernel says that
CONFIG_DEBUG_INFO is enabled:
# grep DEBUG_INFO /boot/config-4.4.35-2-pve
CONFIG_DEBUG_INFO=y
# CONFIG_DEBUG_INFO_REDUCED is not set
# CONFIG_DEBUG_INFO_SPLIT is not set
CONFIG_DEBUG_INFO_DWARF4=y