I was trying to use SystemTap on RHEL4.5, but I could not. It looks like the
runtime/transport/symbols.c file requires the linux/sort.h kernel header, but it
was not backported to kernel-2.6.9-22.EL. I tried downgrading SystemTap but it
did not work as well (probably another bug). Upgrading the kernel may not be an
option in some big companies. Is it possible to make SystemTap more "portable"
by avoiding using kernel headers only for the later kernel versions?

[root@bl20pg2-1 ~]# stap -vve 'probe begin { printf("test") exit()}'
SystemTap translator/driver (version 0.5.12 built 2007-01-23)
(Using Red Hat elfutils 0.125 libraries.)
[...]
  gcc -Wp,-MD,/tmp/stapfg6jFT/.stap_bd4545d46b7c585956ce09d04b7dc0af_243.o.d
-nostdinc -iwithprefix include -D__KERNEL__ -Iinclude  -Wall -Wstrict-prototypes
-Wno-trigraphs -fno-strict-aliasing -fno-common -Os -fomit-frame-pointer -g
-Wdeclaration-after-statement -pipe -msoft-float -m32 -fno-builtin-sprintf
-fno-builtin-log2 -fno-builtin-puts  -mpreferred-stack-boundary=2
-fno-unit-at-a-time -march=i686 -mregparm=3 -Iinclude/asm-i386/mach-generic
-Iinclude/asm-i386/mach-default -freorder-blocks -Wno-unused -Werror
-I"/usr/share/systemtap/runtime"   -DMODULE
-DKBUILD_BASENAME=stap_bd4545d46b7c585956ce09d04b7dc0af_243
-DKBUILD_MODNAME=stap_bd4545d46b7c585956ce09d04b7dc0af_243 -c -o
/tmp/stapfg6jFT/.tmp_stap_bd4545d46b7c585956ce09d04b7dc0af_243.o
/tmp/stapfg6jFT/stap_bd4545d46b7c585956ce09d04b7dc0af_243.c
In file included from /usr/share/systemtap/runtime/transport/transport.c:20,
                 from /usr/share/systemtap/runtime/io.c:14,
                 from /usr/share/systemtap/runtime/print.c:15,
                 from /usr/share/systemtap/runtime/runtime.h:76,
                 from
/tmp/stapfg6jFT/stap_bd4545d46b7c585956ce09d04b7dc0af_243.c:34:
/usr/share/systemtap/runtime/transport/symbols.c:15:24: linux/sort.h: No such
file or directory
In file included from /usr/share/systemtap/runtime/transport/transport.c:20,
                 from /usr/share/systemtap/runtime/io.c:14,
                 from /usr/share/systemtap/runtime/print.c:15,
                 from /usr/share/systemtap/runtime/runtime.h:76,
                 from
/tmp/stapfg6jFT/stap_bd4545d46b7c585956ce09d04b7dc0af_243.c:34:
/usr/share/systemtap/runtime/transport/symbols.c: In function
`_stp_load_module_symbols':
/usr/share/systemtap/runtime/transport/symbols.c:281: warning: implicit
declaration of function `sort'
make[1]: *** [/tmp/stapfg6jFT/stap_bd4545d46b7c585956ce09d04b7dc0af_243.o] Error 1
make: *** [_module_/tmp/stapfg6jFT] Error 2
make: Leaving directory `/usr/src/kernels/2.6.9-22.0.2.EL-smp-i686'
[...]

[root@bl20pg2-1 ~]# cat /etc/redhat-release 
Red Hat Enterprise Linux AS release 4 (Nahant Update 5)
[root@bl20pg2-1 ~]# uname -a
Linux bl20pg2-1.somewhere.redhat.com 2.6.9-22.0.2.ELsmp #1 SMP Thu Jan 5
17:13:01 EST 2006 i686 i686 i386 GNU/Linux

[root@bl20pg2-1 ~]# rpm -qa | grep -e kernel -e systemtap -e glibc | sort
glibc-2.3.4-2.36
glibc-common-2.3.4-2.36
glibc-devel-2.3.4-2.36
glibc-headers-2.3.4-2.36
glibc-kernheaders-2.4-9.1.100.EL
kernel-2.6.9-22.EL
kernel-debuginfo-2.6.9-22.0.2.EL
kernel-smp-2.6.9-22.0.2.EL
kernel-smp-2.6.9-22.EL
kernel-smp-devel-2.6.9-22.0.2.EL
kernel-utils-2.4-13.1.99
systemtap-0.5.12-1
systemtap-runtime-0.5.12-1

[root@bl20pg2-1 ~]# ls
/usr/src/redhat/BUILD/kernel-2.6.9/linux-2.6.9/include/linux/sort.h
ls: /usr/src/redhat/BUILD/kernel-2.6.9/linux-2.6.9/include/linux/sort.h: No such
file or directory
[root@bl20pg2-1 ~]# ls /usr/include/linux/sort.h
ls: /usr/include/linux/sort.h: No such file or directory
[root@bl20pg2-1 ~]# ls
/usr/src/kernels/2.6.9-22.0.2.EL-smp-i686/include/linux/sort.h
ls: /usr/src/kernels/2.6.9-22.0.2.EL-smp-i686/include/linux/sort.h: No such file
or directory

------- Additional Comment #1 From Eugene Teo 2007-06-19 06:20 -------

Created an attachment (id=1897)
Here's my proposed patch.

------- Additional Comment #2 From Eugene Teo 2007-06-19 06:34 -------

Created an attachment (id=1898)
Here's my proposed patch for systemtap-0.5.12.

I have uploaded two patches, one for systemtap-0.5.12, and one for
systemtap-cvs, and both are tested. Thanks.

------- Additional Comment #3 From Josh Stone 2007-08-13 22:28 -------

It looks like Martin committed this in runtime/transport/symbols.c r1.13

------- Additional Comment #4 From Eugene Teo 2007-08-13 23:36 -------

Yes, and it wasn't clear when it was committed since my Changelog entry was left
out.