Seeing TCP socket options on apps you don't have source code for
Problem
I (Mike MacCana) needed to identify whether an app we did not have source code for was using TCP_NODELAY on its sockets. Originally I thought netstat or lsof may do this per other OSs, but reading the lsof FAQ:
3.14.1 Why doesn't lsof report socket options, socket states, and TCP flags and values for my dialect? Linux No socket options and values, socket states, or TCP flags and values are reported. The support for "-Tf" could not be added to Linux, because socket options, socket states, and TCP flags and values are not available via the /proc file system.
Damn, that sucks. However the SystemTap Networking Tapset provides a tcp.setsockopt breakpoint which I can use. See http://www.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/pdf/SystemTap_Tapset_Reference.pdf (search for tcp.setsockopt ).
Scripts
So here's the .stap file.
# Show sockets setting options # Return enabled or disabled based on value of optval function getstatus(optval) { if ( optval == 1 ) return "enabling" else return "disabling" } probe begin { print ("\nChecking for apps setting socket options\n") } # Set a socket option probe tcp.setsockopt { status = getstatus(user_int($optval)) printf (" App '%s' (PID %d) is %s socket option %s... ", execname(), pid(), status, optstr) } # Check setting the socket option worked probe tcp.setsockopt.return { if ( ret == 0 ) printf ("success") else printf ("failed") printf ("\n") } probe end { print ("\nClosing down\n") }
Let's make a quick app to test the tap works. The app below makes a socket with NO_DELAY and hangs around for 10 seconds before cleaning up:
# -*- python -*- # -*- coding: utf-8 -*- '''Simple example to create a socket with NODELAY. ''' host = 'localhost' port = 60000 import socket import time def settcpoption(socket,option,enabled): '''Set a TCP option to be True or False on a socket''' # /usr/src/redhat/BUILD/linux-2.6.18/include/linux/socket.h socketlevels = { 'ip':0, 'tcp':6, 'udp':17, 'ipv6':41, 'icmpv6':58, 'sctp':132, 'raw':255, 'ipx':256, 'ax25':257, 'atalk':258, 'netrom':259, 'rose':260, 'decnet':261, 'x25':262, 'packet':263, 'atm':264, 'aal':265, 'irda':266, 'netbeui':267, 'llc':268, 'dccp':269, 'netlink':270, 'tipc':271, } # /usr/src/redhat/BUILD/linux-2.6.18/include/linux/tcp.h tcpoptions = { 'nodelay':1, 'maxseg':2, 'cork':3, 'keepidle':4, 'keepintvl':5, 'keepcnt':6, 'syncnt':7, 'linger2':8, 'defer_accept':9, 'window_clamp':10, 'info':11, 'quickack':12, 'congestion':13, 'timestamps':1, 'sack':2, 'wscale':4, 'ecn':8, } socket.setsockopt(socketlevels['ip'], tcpoptions[option], int(enabled)) def makeclient(): '''Create a client socket''' #create an INET, STREAMing socket clientsocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM) return clientsocket if __name__ == '__main__': clientsocket = makeclient() # Set TCP_NODELAY before sending short messages that should be sent immediately. settcpoption(clientsocket,'nodelay',True) # Set TCP_CORK before sending a series of data # that should be considered as a single message # settcpoption(clientsocket,'cork',True) time.sleep(10000)
We'll run the demo app first, then run stap in another tab:
Output
Checking for apps setting socket options App 'python2.4' (PID 14415) is enabling socket option TCP_NODELAY... success App 'python2.4' (PID 14420) is disabling socket option TCP_NODELAY... success
Sweet, it works with our demo app. We can now run the tap while our proprietary application runs, and see whether it's using NODELAY or not.
Lessons
We can use SystemTap to get info out of the kernel even if the developers haven't exposed it via /proc or /sys.