Poor scaling

[Avi Kivity]

Running a process with 12 active threads, each bound to its own core, 
and touching a systemtap array fairly frequently, I see:


   72.91%  scylla           [kernel.kallsyms]     [k] _raw_spin_lock
    3.49%  scylla           [kernel.kallsyms]     [k] 
start_callback.isra.62
    1.77%  scylla           scylla                [.] 
_ZSt13__adjust_heapIN9__gnu_cxx17__normal_iteratorIPPN8logalloc11region_implESt6vectorIS4_SaIS4_EEEElS4_NS0_5__ops15_Iter_comp_iterIZNS2_7tracker4impl15compact_on_idleERKSt8functionIFbvEEEUlS4_S4_E_EEEvT_T0_SM_T1_T2_.isra.437.constprop.582 

    1.54%  scylla           [kernel.kallsyms]     [k] _raw_spin_unlock
    1.39%  scylla           scylla                [.] 
_ZN7reactor10smp_pollfn4pollEv
    1.29%  scylla           libc-2.19.so          [.] epoll_pwait
    1.21%  swapper          [kernel.kallsyms]     [k] poll_idle
    1.06%  scylla           [kernel.kallsyms]     [k] 
_raw_read_lock_irqsave
    1.01%  scylla           scylla                [.] 
_ZSt11__push_heapIN9__gnu_cxx17__normal_iteratorIPPN8logalloc11region_implESt6vectorIS4_SaIS4_EEEElS4_NS0_5__ops14_Iter_comp_valIZNS2_7tracker4impl15compact_on_idleERKSt8functionIFbvEEEUlS4_S4_E_EEEvT_T0_SM_T1_T2_.isra.436.constprop.583 

    0.80%  scylla           [kernel.kallsyms]     [k] 
_raw_read_unlock_irqrestore
    0.78%  scylla           scylla                [.] 
_ZN17smp_message_queue20flush_response_batchEv
    0.75%  scylla           [kernel.kallsyms]     [k] 
utrace_report_syscall_entry
    0.63%  scylla           [kernel.kallsyms]     [k] task_utrace_struct
    0.50%  scylla           [kernel.kallsyms]     [k] native_read_tsc
    0.37%  scylla           [kernel.kallsyms]     [k] 
utrace_report_syscall_exit
    0.32%  scylla           [kernel.kallsyms]     [k] delay_tsc
    0.28%  scylla           scylla                [.] 
_ZN17smp_message_queue13process_queueILm2EZNS_16process_incomingEvEUlPNS_9work_itemEE_EEmRNS_8lf_queueET0_.isra.2371 

    0.26%  scylla           scylla                [.] 
_ZN17smp_message_queue13process_queueILm4EZNS_19process_completionsEvEUlPNS_9work_itemEE_EEmRNS_8lf_queueET0_.isra.2278 

    0.20%  scylla           [kernel.kallsyms]     [k] _raw_spin_lock_irq
    0.18%  scylla           scylla                [.] 
_ZN6memory9cpu_pages4freeEPv
    0.18%  scylla           liblz4.so.1.0.0       [.] LZ4_compress
    0.17%  scylla           scylla                [.] 
_ZN7reactor9run_tasksER15circular_bufferISt10unique_ptrI4taskSt14default_deleteIS2_EESaIS5_EE 



Now this is not something I can run to monitor a production load.


Is there any way I can improve scaling?