• WSunixSockets

Unix socket traffic tracing

Problem

When two programs talk across UNIX sockets, what do they say to each other? Maybe it's an exchange of secrets - or of sundry rumours. Do we really want to know?

Well, if you do, and you don't feel like imposing systemcall level tracing on the involved processes, systemtap can help you. Hi ho, silver.

Scripts

unix_sockets.stp

probe kernel.function("unix_stream_sendmsg") {
  if ($sock->sk != $1) next # comment out this line if you want systemwide rather than per-socket tracing

  printf("%d %s %p ", pid(), execname(), $sock->sk)

  len = 0
  for (i = 0; i < $msg->msg_iovlen; i++)
    len += $msg->msg_iov[i]->iov_len

  printf("%d [", len)

  for (i = 0; i < $msg->msg_iovlen; i++)
    printf("%s", user_buffer_printable($msg->msg_iov[i]->iov_base, $msg->msg_iov[i]->iov_len))

  printf("] [")

  for (i = 0; i < $msg->msg_iovlen; i++)
    printf("%s", user_buffer_hexdump($msg->msg_iov[i]->iov_base, $msg->msg_iov[i]->iov_len))

  printf("]\n")
}

/* The following stuff will be copied into the standard tapset, at which point 
 * it can be removed from this script, and you will not need "-g" (guru mode) to run it.
 */

# function user_buffer_raw:string (addr:long, size:long) { return _user_buffer (addr, size, 0) }
function user_buffer_printable:string (addr:long, size:long) { return _user_buffer (addr, size, 1) }
function user_buffer_hexdump:string (addr:long, size:long) { return _user_buffer (addr, size, 2) }

function _user_buffer:string (addr:long, size:long, mode:long) %{ /* pure */
  size_t i;
  long rc = 0;
  char byte;
  char *output = THIS->__retvalue;
  size_t limit = (THIS->mode == 2 ? MAXSTRINGLEN/2 : MAXSTRINGLEN) - 1;
  for (i = 0; i < THIS->size && i < limit && rc >= 0; i++) {
    rc = _stp_strncpy_from_user (& byte, (const char __user*) (uintptr_t)(THIS->addr+i), 1);
    if (rc < 0) break;
    if (THIS->mode == 0) *output++ = byte;
    else if (THIS->mode == 1) *output++ = ((byte >= 32 && byte < 127) ? byte : '.');
    else if (THIS->mode == 2) { *output++ = "0123456789ABCDEF"[(byte >> 4) & 0x0f];
                                *output++ = "0123456789ABCDEF"[byte & 0x0f]; }
  }
  *output = '\0';
%}

Output

# lsof -p PID | grep unix
mysqld  24970 mysql   12u  unix 0xffff81012c2c4500           4335583 /var/lib/mysql/mysql.sock
mysqld  24970 mysql   25u  unix 0xffff81009df84000          10989751 /var/lib/mysql/mysql.sock
# stap -g unix_sockets.stp 0xffff81009df84000
24970 mysqld 0xffff81009df84000 111 [..... ....def....DATABASE()...."..............def....USER()....M........................fche@localhost.........] [010000010220000002036465660000000A44415441424153452829000C080022000000FD00001F00001C0000030364656600000006555345522829000C0800]
24970 mysqld 0xffff81009df84000 253 [.....,....def....@@character_set_client..............0....def....@@character_set_connection..............,....def....@@characte] [01000001042C000002036465660000001640406368617261637465725F7365745F636C69656E74000C080006000000FD01001F000030000003036465660000]
24970 mysqld 0xffff81009df84000 140 [....Uptime: 2340205  Threads: 1  Questions: 780  Slow queries: 0  Opens: 91  Flush tables: 1  Open tables: 27  Queries per seco] [88000001557074696D653A20323334303230352020546872656164733A203120205175657374696F6E733A203738302020536C6F7720717565726965733A20]

Note that it may be necessary to redefine MAXSTRINGLEN to a larger value to avoid truncating the output if the iovec buffer does not fit, for example by calling stap with -DMAXSTRINGLEN=4096.

Lessons

With the right probe point found by some source grokking, the basic tracing probe is very easy. One could trace the receiving side as well as the sending side. The functions for hex dumping user/kernel buffers belongs into the standard tapset.

WarStories