script(s) for tracking execution paths

[Tom Zanussi]

Hi,

I've been having some fun playing around with maps keyed on stack
traces (very cool to be able to do this) and have checked into
systemtap.samples a systemtap script (kmalloc-stacks.stp) and a perl
script (kmalloc-top) that in combination continuously track the
different execution paths to kmalloc().  It keeps counts of the
different execution paths in a systemtap hash, and periodically dumps
the hash contents (and clears the hash table) to the perl script,
which rehashes them in user space.  At the end of the run (Ctrl-C), it
sorts the accumulated hash table and filters out any stacks that
contain any of the user-supplied regexps (-e option), which helps cut
down on noise e.g.

# ./kmalloc-top -e sys_select,alloc_skb

Here's an example of the output (gathered over a kernel compile):

----

Will exclude stacks containing: sys_select|alloc_skb
Will print stacks with counts >= 1.

This path seen 79948 times:
 0xc015c961 : __kmalloc+0x8/0x80
 0xe010334a : ext3_htree_store_dirent+0x2e/0x12e [ext3]
 0xe0109511 : htree_dirblock_to_tree+0xaf/0xd1 [ext3]
 0xe0109598 : ext3_htree_fill_tree+0x65/0x1d3 [ext3]
 0xe01035fc : ext3_dx_readdir+0xe4/0x191 [ext3]
 0xc0193fe6 : filldir64+0x0/0xc5
 0xc0193fe6 : filldir64+0x0/0xc5
 0xe01030e6 : ext3_readdir+0x358/0x4bb [ext3]
 0xc0193fe6 : filldir64+0x0/0xc5
 0xc0193fe6 : filldir64+0x0/0xc5
 0xc0193d84 : vfs_readdir+0x98/0xad
 0xc019410e : sys_getdents64+0x63/0xa6
 0xc0103a61 : syscall_call+0x7/0xb

This path seen 59041 times:
 0xc015c961 : __kmalloc+0x8/0x80
 0xe010334a : ext3_htree_store_dirent+0x2e/0x12e [ext3]
 0xe0109511 : htree_dirblock_to_tree+0xaf/0xd1 [ext3]
 0xe0109598 : ext3_htree_fill_tree+0x65/0x1d3 [ext3]
 0xe01035fc : ext3_dx_readdir+0xe4/0x191 [ext3]
 0xc0193fe6 : filldir64+0x0/0xc5
 0xc0193fe6 : filldir64+0x0/0xc5
 0xe01030e6 : ext3_readdir+0x358/0x4bb [ext3]
 0xc02141b6 : copy_to_user+0x42/0x60
 0xc0193fe6 : filldir64+0x0/0xc5
 0xc0193fe6 : filldir64+0x0/0xc5
 0xc0193d84 : vfs_readdir+0x98/0xad
 0xc019410e : sys_getdents64+0x63/0xa6
 0xc0103a61 : syscall_call+0x7/0xb

This path seen 50562 times:
 0xc015c961 : __kmalloc+0x8/0x80
 0xc01b5185 : load_elf_interp+0x78/0x397
 0xc011904c : do_page_fault+0x0/0x59f
 0xc0103c7b : error_code+0x4f/0x54
 0xc0213f1f : clear_user+0x48/0x56
 0xc01b61e6 : load_elf_binary+0xb54/0xd94
 0xc018bcc6 : search_binary_handler+0x62/0x1be
 0xc018c00a : do_execve+0x1e8/0x217
 0xc0101a89 : sys_execve+0x35/0x1ee
 0xc0108446 : do_syscall_trace+0xef/0x123
 0xc0103a61 : syscall_call+0x7/0xb

.
.
.

This path seen 1 times:
 0xc015c961 : __kmalloc+0x8/0x80
 0xe003ea57 : do_get_write_access+0x243/0x691 [jbd]
 0al>
<40xc017fece : __getblk+0x2c/0x52
 0xc0181cfd : ll_rw_block+0x23/0x87
 0xe003eebd : journal_get_write_access+0x18/0x26 [jbd]
 0xe010880a : ext3_reserve_inode_write+0x4f/0x8c [ext3]
 0xe010886b : ext3_mark_inode_dirty+0x24/0x3a [ext3]
 0xe01088ea : ext3_dirty_inode+0x69/0x82 [ext3]
 0xc0103c7b : error_code+0x4f/0x54
 0xc01ac428 : __mark_inode_dirty+0x28/0x2e8
 0xc0128604 : current_fs_time+0x4e/0x69
 0xc01a0a24 : update_atime+0x45/0x72
 0xc01535f8 : do_generic_mapping_read+0x376/0x4b0
 0xc01538be : __generic_file_aio_read+0xb2/0x1fe
 0xc

This path seen 1 times:
 0xc015c961 : __kmalloc+0x8/0x80
 0xc02df3c0 : hid_input_field+0x48/0x2ac
 0xc02df6c2 : hid_input_report+0x9e/0xb8
 0xc02df780 : hid_irq_in+0xa4/0xa6
 0xe01f8298 : uhci_destroy_urb_priv+0xb5/0x10a [uhci_hcd]
 0xc02d3cac : usb_hcd_giveback_urb+0x24/0x66
 0xe01f9903 : uhci_finish_urb+0x47/0x163 [uhci_hcd]
 0xe01f939c : uhci_transfer_result+0x18b/0x271 [uhci_hcd]
 0xe01f9a63 : uhci_finish_completion+0x44/0x56 [uhci_hcd]
 0xe01f9b56 : uhci_scan_schedule+0xaa/0x13a [uhci_hcd]
 0xc01405b3 : __wake_up_bit+0x2e/0x33
 0xe01fa01d : uhci_irq+0x71/0x241 [uhci_hcd]
 0xc015653c : mempool_free+0x67/0x1ca
 0xc02d3d10 : usb_hcd_irq+0x22/0x52
 0xc

This path seen 1 times:
 0xc015c961 : __kmalloc+0x8/0x80
 0xe003ea57 : do_get_write_access+0x243/0x691 [jbd]
 0xc017fece : __getblk+0x2c/0x52
 0xc015c961 : __kmalloc+0x8/0x80
 0xe02fa386 : _stp_map_key_str+0x36/0x6b [stap_17853]
 0xe003eebd : journal_get_write_access+0x18/0x26 [jbd]
 0xe010880a : ext3_reserve_inode_write+0x4f/0x8c [ext3]
 0xe010886b : ext3_mark_inode_dirty+0x24/0x3a [ext3]
 0xe01088ea : ext3_dirty_inode+0x69/0x82 [ext3]
 0xc0103c7b : error_code+0x4f/0x54
 0xc01ac428 : __mark_inode_dirty+0x28/0x2e8
 0xc0128604 : current_fs_time+0x4e/0x69
 0xc01a0a24 : update_atime+0x45/0x72
 0xc01535f8 : do_generic_mapping_read+0x376/0x4b0
 0xc

Total kmallocs (before filtering): 558745
Num stacks before filtering: 6237
Num stacks after filtering: 1410


Tom