Systemtap cannot probe user programs in docker

[lifeibiren@gmail.com]

     Thanks for your ripley.

     I just looked into it today.And it turns out that docker's storage 
driver might be the problem.
     In my test case, programs put in aufs or overlay2 filesystem can't 
be probed with uprobe, even if I run it from host instead of docker 
container. I modify systemtap's runtime to print inode number and it's 
correct. But if I mount a host's directory to docker containers, such 
problems are gone. So I guess it has nothing to do with systemtap.

     Then I change docker's driver and test again and again, finding 
that it only works well with vfs driver. (only tested aufs, overlay2 and 
vfs.)


On 12/23/2017 08:33 PM, Arkady wrote:
> Containers run in different namespaces. For example, inside of a
> Docker container PIDs start from 1.
> I would suggest to hook the Docker kernel module.
> 
> 
> On Sat, Dec 23, 2017 at 12:48 PM, lifeibiren@gmail.com
> <lifeibiren@gmail.com> wrote:
>> HI,
>>
>>      I try to probe user program in docker but it doesn't return anything at
>> exit. But it works in the host which runs the docker container. When I probe
>> kernel in docker, it works fine.
>>
>>      I google for half a day and got nothing. There are also some other guys
>> who meet the same problems and can't resolve it. Can anybody help? Thanks.
>>
>>
>> ----------------------------Execution----------------------------------
>> Run docker by: 'docker run -it --privileged fa57ca962f13 /bin/bash'
>>
>> $ stap para-callgraph.stp 'process("test2").function("*")' -c ./test2
>> WARNING: function _start return probe is blacklisted: keyword at
>> para-callgraph.stp:24:1
>>   source: probe $1.return { trace(-1, $$return) }
>>
>>
>> While in host:
>> $ stap para-callgraph.stp 'process("test2").function("*")' -c ./test2
>> WARNING: function _start return probe is blacklisted: keyword at
>> para-callgraph.stp:24:1
>>   source: probe $1.return { trace(-1, $$return) }
>>           ^
>>       0 test2(22954):->_start
>>      15 test2(22954): ->__libc_csu_init
>>      19 test2(22954):  ->_init
>>      22 test2(22954):  <-_init
>>      24 test2(22954):  ->frame_dummy
>>      28 test2(22954):   ->register_tm_clones
>>      31 test2(22954):   <-register_tm_clones
>>      32 test2(22954):  <-frame_dummy
>>      34 test2(22954): <-__libc_csu_init
>>      36 test2(22954): ->main
>>     102 test2(22954):  ->dummy
>>     105 test2(22954):  <-dummy
>>     107 test2(22954): <-main
>>     111 test2(22954): ->__do_global_dtors_aux
>>     114 test2(22954):  ->deregister_tm_clones
>>     117 test2(22954):  <-deregister_tm_clones
>>     119 test2(22954): <-__do_global_dtors_aux
>>     121 test2(22954): ->_fini
>>     123 test2(22954): <-_fini
>>
>> -----------------------------Source-------------------------------
>> $ cat para-callgraph.stp
>> #! /usr/bin/env stap
>> function trace(entry_p, extra) {
>>    %( $# > 1 %? if (tid() in trace) %)
>>    printf("%s%s%s %s\n",
>>           thread_indent (entry_p),
>>           (entry_p>0?"->":"<-"),
>>           ppfunc (),
>>           extra)
>> }
>> %( $# > 1 %?
>> global trace
>> probe $2.call {
>>    trace[tid()] = 1
>> }
>> probe $2.return {
>>    delete trace[tid()]
>> }
>> %)
>> probe $1.call   { trace(1, $$parms) }
>> probe $1.return { trace(-1, $$return) }
>>
>>
>> ---------------------------Enviroment-------------------------
>> Host OS: Ubuntu 16.04 LTS
>>
>> $ uname -r
>> 4.4.0-98-generic
>>
>> $ stap -V
>> Systemtap translator/driver (version 2.9/0.165, Debian version 2.9-2ubuntu2
>> (xenial))
>> Copyright (C) 2005-2015 Red Hat, Inc. and others
>> This is free software; see the source for copying conditions.
>> enabled features: AVAHI LIBSQLITE3 NLS NSS TR1_UNORDERED_MAP
>>
>> $ docker version
>> Client:
>>   Version:      17.09.0-ce
>>   API version:  1.32
>>   Go version:   go1.8.3
>>   Git commit:   afdb6d4
>>   Built:        Tue Sep 26 22:42:18 2017
>>   OS/Arch:      linux/amd64
>>
>> Server:
>>   Version:      17.09.0-ce
>>   API version:  1.32 (minimum version 1.12)
>>   Go version:   go1.8.3
>>   Git commit:   afdb6d4
>>   Built:        Tue Sep 26 22:40:56 2017
>>   OS/Arch:      linux/amd64
>>   Experimental: false