Re: sendmail function probe registration error with systemtap

[Xiaokui Shu <subbyte at gmail dot com>]

Hi Frank,

I have two things to add.

1) I use -DSTP_NO_OVERLOAD=1 to make sure the script does not quit. I
forget to write it in the previous email.
2) I find the following scenario about sendmail:
If I run stap to trace sendmail, no matter how many emails I send to
the monitored server, I can get only two processes for sendmail on the
server.

root      5323  0.0  0.0  14304  2532 ?        Ss   Jan08   0:06
sendmail: accepting connections
smmsp     5338  0.0  0.0  13852  2048 ?        Ss   Jan08   0:00
sendmail: Queue runner@01:00:00 for /var/spool/clientmqueue

When I stop stap, I get more processes:

root      5323  0.0  0.0  14304  2532 ?        Ss   Jan08   0:06
sendmail: accepting connections
smmsp     5338  0.0  0.0  13852  2048 ?        Ss   Jan08   0:00
sendmail: Queue runner@01:00:00 for /var/spool/clientmqueue
root     14757  0.0  0.0  14300  3328 ?        S    18:03   0:00
sendmail: ./s0AN3uP7014749 from queue
root     14759  0.0  0.0  14300  3328 ?        S    18:03   0:00
sendmail: ./s0AN3uis014755 from queue
...

Then I can receive emails in my mailbox. It seems that sendmail queued
requests and the tracing block sendmail from forking subprocesses to
handle the requests in the queue.

Best,
Xiaokui

On Fri, Jan 10, 2014 at 4:43 PM, Xiaokui Shu <subbyte@gmail.com> wrote:
> Dear Frank,
>
> I am very glad that you can help. I am not familiar with perf probe,
> and thank you for your kindness to help us.
>
> I am running Fedora 19 with the latest kernel in the repository. I
> installed sendmail from the official repository as well as debug-info
> for glibc and coreutils. sendmail is tested working properly without
> systemtap tracing scripts. We are doing experiments tracing function
> calls of sendmail using trace.stp I provide in my first email.
>
> # yum list installed | grep sendmail
> sendmail.i686                         8.14.7-1.fc19                     @fedora
> sendmail-cf.noarch                    8.14.7-1.fc19                     @fedora
> sendmail-debuginfo.i686               8.14.7-1.fc19
>  @fedora-debuginfo
> sendmail-devel.i686                   8.14.7-1.fc19                     @fedora
> sendmail-milter.i686                  8.14.7-1.fc19                     @fedora
>
> # yum list installed | grep glibc
> glibc.i686                            2.17-20.fc19                      @updates
> glibc-common.i686                     2.17-20.fc19                      @updates
> glibc-debuginfo.i686                  2.17-20.fc19
>  @updates-debuginfo
> glibc-debuginfo-common.i686           2.17-20.fc19
>  @updates-debuginfo
> glibc-devel.i686                      2.17-20.fc19                      @updates
> glibc-headers.i686                    2.17-20.fc19                      @updates
>
> # yum list installed | grep coreutils
> coreutils.i686                        8.21-11.fc19                      @fedora
> coreutils-debuginfo.i686              8.21-11.fc19
>  @fedora-debuginfo
> policycoreutils.i686                  2.1.14-46.6.fc19                  @updates
>
>
> Best,
> Xiaokui
>
> On Fri, Jan 10, 2014 at 4:15 PM, Frank Ch. Eigler <fche@redhat.com> wrote:
>> On Fri, Jan 10, 2014 at 03:26:09PM -0500, Xiaokui Shu wrote:
>>> I find nothing changed with -DSTP_ALIBI. It does not stop sendmail
>>> from not receiving emails. Do you suggest this is a uprobe problem
>>> instead of systemtap?
>>
>> Yup, high likelihood.
>>
>>> The command I run stap is:
>>> #stap -DSTP_ALIBI -o sendmail.trace -v trace.stp sendmail
>>
>>> == stap -V ==
>>> Systemtap translator/driver (version 2.4/0.156, rpm 2.4-1.fc19)
>>> [...]
>>> == dmesg | fgrep 'Linux version' ==
>>> [    0.000000] Linux version 3.12.6-200.fc19.i686.PAE
>>> (mockbuild@bkernel02) (gcc version 4.8.2 20131212 (Red Hat 4.8.2-7)
>>> (GCC) ) #1 SMP Mon Dec 23 16:58:40 UTC 2013
>>> [...]
>>
>> Thanks a lot.  So it's a nice recent kernel & systemtap combination.
>>
>> One way to proceed further is to try to reproduce the problem with
>> "perf probe" rather than systemtap.  For that, one might run "stap -k
>> ..." to preserve the C intermediate code to fish out the low-level
>> probe addresses, arrange to feed the same set to perf-probe, then
>> activate them on a copy of sendmail, and see if it flies.  In case
>> it doesn't, it's time to file a kernel bugzilla entry.
>>
>> We can help with this part, but would need e.g. the generated .c file
>> and the exact rpm version of your sendmail.
>>
>>
>> - FChE