Systemtap do_filp_open failure on a few linux packages

[Henrik /KaarPoSoft]

Hi David,

Thank you very much for going through my input and providing an 
excellent answer.
Your input nailed it for me (although with a twist; see below)

On 06/17/2013 08:16 PM, David Smith wrote:
> On 06/15/2013 04:52 PM, Henrik /KaarPoSoft wrote:
>> Dear all,
>>
>> I have experienced a very strange issue related to systemtap.
>> Any insights or help you might be able to provide to help me debug this
>> further would be most appreciated.
>>
>> I am developing a linux distribution called KaarPux:
>> http://kaarpux.kaarposoft.dk/
>>
>> Using a few scripts, some 600+ linux packages are build and installed.
>> Generally, this works like a charm.
>>
>> In order to automatically collect package dependencies, I have created
>> a small systemtap script to show files opened for reading:
>> http://sourceforge.net/p/kaarpux/code/ci/be342bf5667253421f562b7bc29bab8e0a2560aa/tree/master/chroot_scripts/kx_open.stp
>
> The script looks reasonable. One small note, you shouldn't need the
> '@defined($return)' check since $return should always be defined in
> 'kernel.function("do_filp_open").return'.
>

OK, thanks, will remove it later...
Kept it for now: better safe than sorry.

>
>> For most of the 600+ packages, building is successfull, and the probe
>> returns what seems to be reasonable results.
>>
>> However, for a few packages, building fails:
>> - firefox
>> - thunderbird
>> - libreoffice
>> - ghc-binary
>> - ghc
>
> Here's my thought. I don't think it is the probe itself that is causing
> the problem, my guess would be that it is our staprun loader (a setuid
> executable). When the module gets loaded, we unset several environment
> variables, like 'IFS', 'CDPATH', 'ENV', 'BASH_ENV' for security reasons.
>
> I'd guess those failing packages depend on something set in the environment.
>
> If this is the case, to work around this problem you might be able to
> modify the script you use to run staprun (linux_functions.shinc) to add
> the environment variable back in. Here's that line from
> linux_functions.shinc (split up a bit):
>
> ====
>   staprun /lib/modules/$(uname -r)/systemtap/kx_open.ko \
>    -c "./scripts/${PASS}/${PKG}_${STEP}.sh" \
>    -o "${PIPE}" > ./log/${PASS}/${PKG}_${STEP}.log 2>&1
> ====
>
> You could change that to
>
> ====
>    staprun /lib/modules/$(uname -r)/systemtap/kx_open.ko \
>     -c "env 'FOO=BAR' ./scripts/${PASS}/${PKG}_${STEP}.sh" \
>     -o "${PIPE}" > ./log/${PASS}/${PKG}_${STEP}.log 2>&1
> ====
>
> Your other option would be to put the missing environment variable in
> scripts/${PASS}/${PKG}_${STEP}.sh.
>
> This might not be the problem, but it is certainly worth investigating.
>

My problem turned out to be environment variables indeed!
Thank you for pointing me in this direction.

However, the culprit was that LD_LIBRARY_PATH got killed.

Simply adding env LD_LIBRARY_PATH=... to the staprun solved the problem:
http://sourceforge.net/p/kaarpux/code/ci/master/tree/master/shinc/linux_functions.shinc

So, again:
THANKS A MILLION FOR THE HINT !!!

/Henrik