[SCRIPT] NUMA page fault accounting.

[Jose R. Santos]

Hi folks,

My teams has had the need to do analysis of page faults that happen on a 
per NUMA node basis and I've come up with this simple SystemTap script 
that does this for me.  I only have access to a PPC64 NUMA box but I 
took care to use generic arch independed NUMA code and I think it should 
work for any kernel that has NUMA enable.  Here a short output  of 
running the stream benchmark with a 4.5GB array on a 8GB system.

Starting pagefault counters
Printing counters:
            Execname         PID        Read Faults     Write Faults
====================    ========        ===========     ============
           stream       22786                 33           668186    
Node[0]=521176  Node[1]=147043  
                stpd       22803                  1                3    
Node[0]=3       Node[1]=1       

and the script:

#! stap -g

global execnames, page_faults, node_faults

function addr_to_node:long(addr:long)
%{
        int nid;
        int pfn = __pa(THIS->addr) >> PAGE_SHIFT;
        for_each_online_node(nid)
                if ( node_start_pfn(nid) <= pfn &&
                        pfn < (node_start_pfn(nid) +
                        NODE_DATA(nid)->node_spanned_pages) )
                {
                        THIS->__retvalue = nid;                
                        break;
                }

%}

probe kernel.function("__handle_mm_fault") {
        execnames[pid()] = execname()
        page_faults [pid(), $write_access ? 1 : 0] ++
        node_faults [pid(), addr_to_node($address)] ++

}

function print_pf () {
        print ("            Execname\t     PID\tRead Faults\tWrite 
Faults\n")
        print 
("====================\t========\t===========\t============\n")
        foreach (pid in execnames) {
                printf ("%20s\t%8d\t%11d\t%12d\t", execnames[pid], pid,
                        page_faults[pid,0], page_faults[pid,1])

                foreach ([pid2,node+] in node_faults) {
                        if (pid2 == pid)
                                printf ("Node[%d]=%d\t", node,
                                        node_faults[pid2, node])
                }
                print ("\n")

        }
}

probe begin {
  print ("Starting pagefault counters \n")
}

probe end {
  print ("Printing counters: \n")
  print_pf ()
  print ("Done\n")
}


Comments welcome - Enjoy

-JRS