Seeing TCP socket options on apps you don't have source code for

Problem

I (Mike MacCana) needed to identify whether an app we did not have source code for was using TCP_NODELAY on its sockets. Originally I thought netstat or lsof may do this per other OSs, but reading the lsof FAQ:

3.14.1  Why doesn't lsof report socket options, socket states, and TCP
        flags and values for my dialect?

Linux
                No socket options and values, socket states, or TCP
                flags and values are reported.  The support for "-Tf"
                could not be added to Linux, because socket options,
                socket states, and TCP flags and values are not
                available via the /proc file system.

Damn, that sucks. However the SystemTap Networking Tapset provides a tcp.setsockopt breakpoint which I can use. See http://www.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/pdf/SystemTap_Tapset_Reference.pdf (search for tcp.setsockopt ).

Scripts

So here's the .stap file.

# Show sockets setting options

# Return enabled or disabled based on value of optval
function getstatus(optval)
{
    if ( optval == 1 )
        return "enabling"
    else
        return "disabling"
}

probe begin
{
        print ("\nChecking for apps setting socket options\n")
}

# Set a socket option 
probe tcp.setsockopt
{
    status = getstatus(user_int($optval))
        printf ("  App '%s' (PID %d) is %s socket option %s... ", execname(), pid(), status, optstr)
}

# Check setting the socket option worked
probe tcp.setsockopt.return
{
    if ( ret == 0 )
        printf ("success")
    else
        printf ("failed")
    printf ("\n")    
}


probe end
{
        print ("\nClosing down\n")
}

Let's make a quick app to test the tap works. The app below makes a socket with NO_DELAY and hangs around for 10 seconds before cleaning up:

# -*- python -*-
# -*- coding: utf-8 -*-
'''Simple example to create a socket with NODELAY. '''

host = 'localhost'
port = 60000

import socket
import time




def settcpoption(socket,option,enabled):
    '''Set a TCP option to be True or False on a socket'''

    # /usr/src/redhat/BUILD/linux-2.6.18/include/linux/socket.h
    socketlevels = {
        'ip':0,
        'tcp':6,
        'udp':17,
        'ipv6':41,
        'icmpv6':58,
        'sctp':132,
        'raw':255,
        'ipx':256,
        'ax25':257,
        'atalk':258,
        'netrom':259,
        'rose':260,
        'decnet':261,
        'x25':262,
        'packet':263,
        'atm':264,
        'aal':265,
        'irda':266,
        'netbeui':267,
        'llc':268,
        'dccp':269,
        'netlink':270,
        'tipc':271,
        }
    # /usr/src/redhat/BUILD/linux-2.6.18/include/linux/tcp.h    
    tcpoptions = {
        'nodelay':1,      
        'maxseg':2,      
        'cork':3,      
        'keepidle':4,      
        'keepintvl':5,      
        'keepcnt':6,      
        'syncnt':7,      
        'linger2':8,      
        'defer_accept':9,      
        'window_clamp':10,     
        'info':11,     
        'quickack':12,     
        'congestion':13,     
        'timestamps':1,
        'sack':2,
        'wscale':4,
        'ecn':8,
        }    
        
    socket.setsockopt(socketlevels['ip'], tcpoptions[option], int(enabled))

def makeclient():
    '''Create a client socket'''
    #create an INET, STREAMing socket
    clientsocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    return clientsocket
    

if __name__ == '__main__':
    clientsocket = makeclient()
    # Set TCP_NODELAY before sending short messages that should be sent immediately.
    settcpoption(clientsocket,'nodelay',True)
    
    # Set TCP_CORK before sending a series of data 
    # that should be considered as a single message
    # settcpoption(clientsocket,'cork',True)    
    time.sleep(10000)

We'll run the demo app first, then run stap in another tab:

Output

Checking for apps setting socket options
  App 'python2.4' (PID 14415) is enabling socket option TCP_NODELAY... success
  App 'python2.4' (PID 14420) is disabling socket option TCP_NODELAY... success

Sweet, it works with our demo app. We can now run the tap while our proprietary application runs, and see whether it's using NODELAY or not.

Lessons

We can use SystemTap to get info out of the kernel even if the developers haven't exposed it via /proc or /sys.


WarStories

None: WSTCPSocketOptions (last edited 2010-03-26 10:19:49 by ln-bas00)