On RHEL7 (3.10.0-123.el7.s390x), when I run the following command: ==== # stap -v ../src/testsuite/systemtap.examples/profiling/functioncallcount.stp "*@mm/*.c" -c "sleep 1" Pass 1: parsed user script and 103 library script(s) using 143240virt/28188res/3092shr/25540data kb, in 130usr/20sys/442real ms. WARNING: function mem_init is in blacklisted section: keyword at ../src/testsuite/systemtap.examples/profiling/functioncallcount.stp:7:1 source: probe kernel.function(@1).call { # probe functions listed on commandline ^ WARNING: function paging_init is in blacklisted section: keyword at :7:1 source: probe kernel.function(@1).call { # probe functions listed on commandline ^ WARNING: function free_initrd_mem is in blacklisted section: keyword at :7:1 source: probe kernel.function(@1).call { # probe functions listed on commandline ^ WARNING: function nopfault is in blacklisted section: keyword at :7:1 source: probe kernel.function(@1).call { # probe functions listed on commandline ^ WARNING: function pfault_irq_init is in blacklisted section: keyword at :7:1 source: probe kernel.function(@1).call { # probe functions listed on commandline ^ WARNING: kprobes function do_protection_exception is blacklisted: keyword at :7:1 source: probe kernel.function(@1).call { # probe functions listed on commandline ^ WARNING: kprobes function do_dat_exception is blacklisted: keyword at :7:1 source: probe kernel.function(@1).call { # probe functions listed on commandline ^ WARNING: function fault_init is in blacklisted section: keyword at :7:1 source: probe kernel.function(@1).call { # probe functions listed on commandline ^ WARNING: function vmem_pte_alloc is blacklisted: keyword at :7:1 source: probe kernel.function(@1).call { # probe functions listed on commandline ^ WARNING: function vmem_map_init is in blacklisted section: keyword at :7:1 source: probe kernel.function(@1).call { # probe functions listed on commandline ^ WARNING: function vmem_convert_memory_chunk is in blacklisted section: keyword at :7:1 source: probe kernel.function(@1).call { # probe functions listed on commandline ^ WARNING: function cmma_init is in blacklisted section: keyword at :7:1 source: probe kernel.function(@1).call { # probe functions listed on commandline ^ WARNING: function cmma is in blacklisted section: keyword at :7:1 source: probe kernel.function(@1).call { # probe functions listed on commandline ^ WARNING: function set_huge_pte_at is blacklisted: keyword at :7:1 source: probe kernel.function(@1).call { # probe functions listed on commandline ^ WARNING: function huge_pte_offset is blacklisted: keyword at :7:1 source: probe kernel.function(@1).call { # probe functions listed on commandline ^ WARNING: function huge_pte_alloc is blacklisted: keyword at :7:1 source: probe kernel.function(@1).call { # probe functions listed on commandline ^ WARNING: function absent_pages_in_range is in blacklisted section: keyword at :7:1 source: probe kernel.function(@1).call { # probe functions listed on commandline ^ WARNING: function setup_per_cpu_pageset is in blacklisted section: keyword at :7:1 source: probe kernel.function(@1).call { # probe functions listed on commandline ^ WARNING: function free_area_init is in blacklisted section: keyword at :7:1 source: probe kernel.function(@1).call { # probe functions listed on commandline ^ WARNING: function free_area_init_nodes is in blacklisted section: keyword at :7:1 source: probe kernel.function(@1).call { # probe functions listed on commandline ^ WARNING: function free_bootmem_with_active_regions is in blacklisted section: keyword at :7:1 source: probe kernel.function(@1).call { # probe functions listed on commandline ^ WARNING: function cmdline_parse_movablecore is in blacklisted section: keyword at :7:1 source: probe kernel.function(@1).call { # probe functions listed on commandline ^ WARNING: function page_alloc_init is in blacklisted section: keyword at :7:1 source: probe kernel.function(@1).call { # probe functions listed on commandline ^ WARNING: function set_pageblock_order is in blacklisted section: keyword at :7:1 source: probe kernel.function(@1).call { # probe functions listed on commandline ^ WARNING: function cmdline_parse_kernelcore is in blacklisted section: keyword at :7:1 source: probe kernel.function(@1).call { # probe functions listed on commandline ^ WARNING: function find_min_pfn_for_node is in blacklisted section: keyword at :7:1 source: probe kernel.function(@1).call { # probe functions listed on commandline ^ WARNING: function alloc_large_system_hash is in blacklisted section: keyword at :7:1 source: probe kernel.function(@1).call { # probe functions listed on commandline ^ WARNING: function node_map_pfn_alignment is in blacklisted section: keyword at :7:1 source: probe kernel.function(@1).call { # probe functions listed on commandline ^ WARNING: function set_dma_reserve is in blacklisted section: keyword at :7:1 source: probe kernel.function(@1).call { # probe functions listed on commandline ^ WARNING: function sparse_memory_present_with_active_regions is in blacklisted section: keyword at :7:1 source: probe kernel.function(@1).call { # probe functions listed on commandline ^ WARNING: function find_min_pfn_with_active_regions is in blacklisted section: keyword at :7:1 source: probe kernel.function(@1).call { # probe functions listed on commandline ^ WARNING: function page_writeback_init is in blacklisted section: keyword at :7:1 source: probe kernel.function(@1).call { # probe functions listed on commandline ^ WARNING: function swap_setup is in blacklisted section: keyword at :7:1 source: probe kernel.function(@1).call { # probe functions listed on commandline ^ WARNING: function kswapd_init is in blacklisted section: keyword at :7:1 source: probe kernel.function(@1).call { # probe functions listed on commandline ^ WARNING: function shmem_init is in blacklisted section: keyword at :7:1 source: probe kernel.function(@1).call { # probe functions listed on commandline ^ WARNING: function setup_vmstat is in blacklisted section: keyword at :7:1 source: probe kernel.function(@1).call { # probe functions listed on commandline ^ WARNING: function extfrag_debug_init is in blacklisted section: keyword at :7:1 source: probe kernel.function(@1).call { # probe functions listed on commandline ^ WARNING: function bdi_class_init is in blacklisted section: keyword at :7:1 source: probe kernel.function(@1).call { # probe functions listed on commandline ^ WARNING: function default_bdi_init is in blacklisted section: keyword at :7:1 source: probe kernel.function(@1).call { # probe functions listed on commandline ^ WARNING: function set_mminit_loglevel is in blacklisted section: keyword at :7:1 source: probe kernel.function(@1).call { # probe functions listed on commandline ^ WARNING: function mminit_verify_pageflags_layout is in blacklisted section: keyword at :7:1 source: probe kernel.function(@1).call { # probe functions listed on commandline ^ WARNING: function mm_sysfs_init is in blacklisted section: keyword at :7:1 source: probe kernel.function(@1).call { # probe functions listed on commandline ^ WARNING: function pcpu_embed_first_chunk is in blacklisted section: keyword at :7:1 source: probe kernel.function(@1).call { # probe functions listed on commandline ^ WARNING: function pcpu_free_alloc_info is in blacklisted section: keyword at :7:1 source: probe kernel.function(@1).call { # probe functions listed on commandline ^ WARNING: function pcpu_setup_first_chunk is in blacklisted section: keyword at :7:1 source: probe kernel.function(@1).call { # probe functions listed on commandline ^ WARNING: function percpu_init_late is in blacklisted section: keyword at :7:1 source: probe kernel.function(@1).call { # probe functions listed on commandline ^ WARNING: function pcpu_dfl_fc_free is in blacklisted section: keyword at :7:1 source: probe kernel.function(@1).call { # probe functions listed on commandline ^ WARNING: function pcpu_alloc_alloc_info is in blacklisted section: keyword at :7:1 source: probe kernel.function(@1).call { # probe functions listed on commandline ^ WARNING: function percpu_alloc_setup is in blacklisted section: keyword at :7:1 source: probe kernel.function(@1).call { # probe functions listed on commandline ^ WARNING: function pcpu_dfl_fc_alloc is in blacklisted section: keyword at :7:1 source: probe kernel.function(@1).call { # probe functions listed on commandline ^ WARNING: function setup_per_cpu_areas is in blacklisted section: keyword at :7:1 source: probe kernel.function(@1).call { # probe functions listed on commandline ^ WARNING: function create_boot_cache is in blacklisted section: keyword at :7:1 source: probe kernel.function(@1).call { # probe functions listed on commandline ^ WARNING: function slab_proc_init is in blacklisted section: keyword at :7:1 source: probe kernel.function(@1).call { # probe functions listed on commandline ^ WARNING: function create_kmalloc_caches is in blacklisted section: keyword at :7:1 source: probe kernel.function(@1).call { # probe functions listed on commandline ^ WARNING: function create_kmalloc_cache is in blacklisted section: keyword at :7:1 source: probe kernel.function(@1).call { # probe functions listed on commandline ^ WARNING: function workingset_init is in blacklisted section: keyword at :7:1 source: probe kernel.function(@1).call { # probe functions listed on commandline ^ WARNING: function init_zero_pfn is in blacklisted section: keyword at :7:1 source: probe kernel.function(@1).call { # probe functions listed on commandline ^ WARNING: function copy_pte_range is blacklisted: keyword at :7:1 source: probe kernel.function(@1).call { # probe functions listed on commandline ^ WARNING: function __pte_alloc is blacklisted: keyword at :7:1 source: probe kernel.function(@1).call { # probe functions listed on commandline ^ WARNING: function __pte_alloc_kernel is blacklisted: keyword at :7:1 source: probe kernel.function(@1).call { # probe functions listed on commandline ^ WARNING: function disable_randmaps is in blacklisted section: keyword at :7:1 source: probe kernel.function(@1).call { # probe functions listed on commandline ^ WARNING: function mmap_init is in blacklisted section: keyword at :7:1 source: probe kernel.function(@1).call { # probe functions listed on commandline ^ WARNING: function special_mapping_fault is blacklisted: keyword at :7:1 source: probe kernel.function(@1).call { # probe functions listed on commandline ^ WARNING: function special_mapping_close is blacklisted: keyword at :7:1 source: probe kernel.function(@1).call { # probe functions listed on commandline ^ WARNING: function anon_vma_init is in blacklisted section: keyword at :7:1 source: probe kernel.function(@1).call { # probe functions listed on commandline ^ WARNING: function vmalloc_init is in blacklisted section: keyword at :7:1 source: probe kernel.function(@1).call { # probe functions listed on commandline ^ WARNING: function proc_vmalloc_init is in blacklisted section: keyword at :7:1 source: probe kernel.function(@1).call { # probe functions listed on commandline ^ WARNING: function vm_area_add_early is in blacklisted section: keyword at :7:1 source: probe kernel.function(@1).call { # probe functions listed on commandline ^ WARNING: function vm_area_register_early is in blacklisted section: keyword at :7:1 source: probe kernel.function(@1).call { # probe functions listed on commandline ^ WARNING: function __alloc_bootmem_nopanic is in blacklisted section: keyword at :7:1 source: probe kernel.function(@1).call { # probe functions listed on commandline ^ WARNING: function ___alloc_bootmem_node is in blacklisted section: keyword at :7:1 source: probe kernel.function(@1).call { # probe functions listed on commandline ^ WARNING: function __alloc_bootmem_low is in blacklisted section: keyword at :7:1 source: probe kernel.function(@1).call { # probe functions listed on commandline ^ WARNING: function reserve_bootmem is in blacklisted section: keyword at :7:1 source: probe kernel.function(@1).call { # probe functions listed on commandline ^ WARNING: function init_bootmem is in blacklisted section: keyword at :7:1 source: probe kernel.function(@1).call { # probe functions listed on commandline ^ WARNING: function bootmem_bootmap_pages is in blacklisted section: keyword at :7:1 source: probe kernel.function(@1).call { # probe functions listed on commandline ^ WARNING: function init_bootmem_node is in blacklisted section: keyword at :7:1 source: probe kernel.function(@1).call { # probe functions listed on commandline ^ WARNING: function __alloc_bootmem_node_high is in blacklisted section: keyword at :7:1 source: probe kernel.function(@1).call { # probe functions listed on commandline ^ WARNING: function ___alloc_bootmem_node_nopanic is in blacklisted section: keyword at :7:1 source: probe kernel.function(@1).call { # probe functions listed on commandline ^ WARNING: function __alloc_bootmem_low_node is in blacklisted section: keyword at :7:1 source: probe kernel.function(@1).call { # probe functions listed on commandline ^ WARNING: function __free is in blacklisted section: keyword at :7:1 source: probe kernel.function(@1).call { # probe functions listed on commandline ^ WARNING: function bootmem_debug_setup is in blacklisted section: keyword at :7:1 source: probe kernel.function(@1).call { # probe functions listed on commandline ^ WARNING: function free_all_bootmem_node is in blacklisted section: keyword at :7:1 source: probe kernel.function(@1).call { # probe functions listed on commandline ^ WARNING: function free_bootmem_late is in blacklisted section: keyword at :7:1 source: probe kernel.function(@1).call { # probe functions listed on commandline ^ WARNING: function __alloc_bootmem is in blacklisted section: keyword at :7:1 source: probe kernel.function(@1).call { # probe functions listed on commandline ^ WARNING: function alloc_bootmem_bdata is in blacklisted section: keyword at :7:1 source: probe kernel.function(@1).call { # probe functions listed on commandline ^ WARNING: function __alloc_bootmem_node_nopanic is in blacklisted section: keyword at :7:1 source: probe kernel.function(@1).call { # probe functions listed on commandline ^ WARNING: function mark_bootmem is in blacklisted section: keyword at :7:1 source: probe kernel.function(@1).call { # probe functions listed on commandline ^ WARNING: function alloc_bootmem_core is in blacklisted section: keyword at :7:1 source: probe kernel.function(@1).call { # probe functions listed on commandline ^ WARNING: function free_all_bootmem_core is in blacklisted section: keyword at :7:1 source: probe kernel.function(@1).call { # probe functions listed on commandline ^ WARNING: function reserve_bootmem_node is in blacklisted section: keyword at :7:1 source: probe kernel.function(@1).call { # probe functions listed on commandline ^ WARNING: function init_bootmem_core is in blacklisted section: keyword at :7:1 source: probe kernel.function(@1).call { # probe functions listed on commandline ^ WARNING: function free_bootmem_node is in blacklisted section: keyword at :7:1 source: probe kernel.function(@1).call { # probe functions listed on commandline ^ WARNING: function free_all_bootmem is in blacklisted section: keyword at :7:1 source: probe kernel.function(@1).call { # probe functions listed on commandline ^ WARNING: function mark_bootmem_node is in blacklisted section: keyword at :7:1 source: probe kernel.function(@1).call { # probe functions listed on commandline ^ WARNING: function ___alloc_bootmem_nopanic is in blacklisted section: keyword at :7:1 source: probe kernel.function(@1).call { # probe functions listed on commandline ^ WARNING: function __alloc_bootmem_node is in blacklisted section: keyword at :7:1 source: probe kernel.function(@1).call { # probe functions listed on commandline ^ WARNING: function __reserve is in blacklisted section: keyword at :7:1 source: probe kernel.function(@1).call { # probe functions listed on commandline ^ WARNING: function __alloc_bootmem_low_nopanic is in blacklisted section: keyword at :7:1 source: probe kernel.function(@1).call { # probe functions listed on commandline ^ WARNING: function free_bootmem is in blacklisted section: keyword at :7:1 source: probe kernel.function(@1).call { # probe functions listed on commandline ^ WARNING: function memblock_allow_resize is in blacklisted section: keyword at :7:1 source: probe kernel.function(@1).call { # probe functions listed on commandline ^ WARNING: function memblock_alloc_nid is in blacklisted section: keyword at :7:1 source: probe kernel.function(@1).call { # probe functions listed on commandline ^ WARNING: function memblock_alloc_base is in blacklisted section: keyword at :7:1 source: probe kernel.function(@1).call { # probe functions listed on commandline ^ WARNING: function memblock_phys_mem_size is in blacklisted section: keyword at :7:1 source: probe kernel.function(@1).call { # probe functions listed on commandline ^ WARNING: function __memblock_alloc_base is in blacklisted section: keyword at :7:1 source: probe kernel.function(@1).call { # probe functions listed on commandline ^ WARNING: function early_memblock is in blacklisted section: keyword at :7:1 source: probe kernel.function(@1).call { # probe functions listed on commandline ^ WARNING: function memblock_alloc is in blacklisted section: keyword at :7:1 source: probe kernel.function(@1).call { # probe functions listed on commandline ^ WARNING: function memblock_mem_size is in blacklisted section: keyword at :7:1 source: probe kernel.function(@1).call { # probe functions listed on commandline ^ WARNING: function memblock_is_reserved is in blacklisted section: keyword at :7:1 source: probe kernel.function(@1).call { # probe functions listed on commandline ^ WARNING: function memblock_enforce_memory_limit is in blacklisted section: keyword at :7:1 source: probe kernel.function(@1).call { # probe functions listed on commandline ^ WARNING: function memblock_alloc_base_nid is in blacklisted section: keyword at :7:1 source: probe kernel.function(@1).call { # probe functions listed on commandline ^ WARNING: function memblock_alloc_try_nid is in blacklisted section: keyword at :7:1 source: probe kernel.function(@1).call { # probe functions listed on commandline ^ WARNING: function procswaps_init is in blacklisted section: keyword at :7:1 source: probe kernel.function(@1).call { # probe functions listed on commandline ^ WARNING: function init_frontswap is in blacklisted section: keyword at :7:1 source: probe kernel.function(@1).call { # probe functions listed on commandline ^ WARNING: function hugetlb_exit is in blacklisted section: keyword at :7:1 source: probe kernel.function(@1).call { # probe functions listed on commandline ^ WARNING: function hugetlb_nrpages_setup is in blacklisted section: keyword at :7:1 source: probe kernel.function(@1).call { # probe functions listed on commandline ^ WARNING: function hugetlb_add_hstate is in blacklisted section: keyword at :7:1 source: probe kernel.function(@1).call { # probe functions listed on commandline ^ WARNING: function hugetlb_default_setup is in blacklisted section: keyword at :7:1 source: probe kernel.function(@1).call { # probe functions listed on commandline ^ WARNING: function alloc_bootmem_huge_page is in blacklisted section: keyword at :7:1 source: probe kernel.function(@1).call { # probe functions listed on commandline ^ WARNING: function hugetlb_init is in blacklisted section: keyword at :7:1 source: probe kernel.function(@1).call { # probe functions listed on commandline ^ WARNING: function hugetlb_hstate_alloc_pages is in blacklisted section: keyword at :7:1 source: probe kernel.function(@1).call { # probe functions listed on commandline ^ WARNING: function sparse_early_usemaps_alloc_node is in blacklisted section: keyword at :7:1 source: probe kernel.function(@1).call { # probe functions listed on commandline ^ WARNING: function memory_present is in blacklisted section: keyword at :7:1 source: probe kernel.function(@1).call { # probe functions listed on commandline ^ WARNING: function sparse_init is in blacklisted section: keyword at :7:1 source: probe kernel.function(@1).call { # probe functions listed on commandline ^ WARNING: function node_memmap_size_bytes is in blacklisted section: keyword at :7:1 source: probe kernel.function(@1).call { # probe functions listed on commandline ^ WARNING: function vmemmap_pte_populate is blacklisted: keyword at :7:1 source: probe kernel.function(@1).call { # probe functions listed on commandline ^ WARNING: function sparse_mem_maps_populate_node is in blacklisted section: keyword at :7:1 source: probe kernel.function(@1).call { # probe functions listed on commandline ^ WARNING: function ksm_init is in blacklisted section: keyword at :7:1 source: probe kernel.function(@1).call { # probe functions listed on commandline ^ WARNING: function slab_sysfs_init is in blacklisted section: keyword at :7:1 source: probe kernel.function(@1).call { # probe functions listed on commandline ^ WARNING: function bootstrap is in blacklisted section: keyword at :7:1 source: probe kernel.function(@1).call { # probe functions listed on commandline ^ WARNING: function setup_slub_debug is in blacklisted section: keyword at :7:1 source: probe kernel.function(@1).call { # probe functions listed on commandline ^ WARNING: function setup_slub_min_order is in blacklisted section: keyword at :7:1 source: probe kernel.function(@1).call { # probe functions listed on commandline ^ WARNING: function kmem_cache_init_late is in blacklisted section: keyword at :7:1 source: probe kernel.function(@1).call { # probe functions listed on commandline ^ WARNING: function setup_slub_min_objects is in blacklisted section: keyword at :7:1 source: probe kernel.function(@1).call { # probe functions listed on commandline ^ WARNING: function setup_slub_max_order is in blacklisted section: keyword at :7:1 source: probe kernel.function(@1).call { # probe functions listed on commandline ^ WARNING: function setup_slub_nomerge is in blacklisted section: keyword at :7:1 source: probe kernel.function(@1).call { # probe functions listed on commandline ^ WARNING: function kmem_cache_init is in blacklisted section: keyword at :7:1 source: probe kernel.function(@1).call { # probe functions listed on commandline ^ WARNING: function setup_transparent_hugepage is in blacklisted section: keyword at :7:1 source: probe kernel.function(@1).call { # probe functions listed on commandline ^ WARNING: function hugepage_init is in blacklisted section: keyword at :7:1 source: probe kernel.function(@1).call { # probe functions listed on commandline ^ WARNING: function mem_cgroup_move_charge_pte_range is blacklisted: keyword at :7:1 source: probe kernel.function(@1).call { # probe functions listed on commandline ^ WARNING: function mem_cgroup_count_precharge_pte_range is blacklisted: keyword at :7:1 source: probe kernel.function(@1).call { # probe functions listed on commandline ^ WARNING: function mem_cgroup_init is in blacklisted section: keyword at :7:1 source: probe kernel.function(@1).call { # probe functions listed on commandline ^ WARNING: function enable_swap_account is in blacklisted section: keyword at :7:1 source: probe kernel.function(@1).call { # probe functions listed on commandline ^ WARNING: function page_cgroup_init is in blacklisted section: keyword at :7:1 source: probe kernel.function(@1).call { # probe functions listed on commandline ^ WARNING: function init_cleancache is in blacklisted section: keyword at :7:1 source: probe kernel.function(@1).call { # probe functions listed on commandline ^ Pass 2: analyzed script: 1536 probe(s), 7 function(s), 4 embed(s), 1 global(s) using 179368virt/64488res/4284shr/61668data kb, in 710usr/340sys/6198real ms. Pass 3: translated to C into "/tmp/stapfAIHgx/stap_7f45c75080212845924364cc12433973_290793_src.c" using 179368virt/64836res/4608shr/61668data kb, in 150usr/40sys/271real ms. Pass 4: compiled C into "stap_7f45c75080212845924364cc12433973_290793.ko" in 5700usr/790sys/7925real ms. Pass 5: starting run. ==== I get the following kernel panic: ==== [ 129.520018] Unable to handle kernel pointer dereference at virtual kernel add ress 000000f421d05000 [ 129.520063] Oops: 003b [#1] SMP [ 129.520066] Modules linked in: stap_7f45c75080212845924364cc12433973_2_2903(O F) sg qeth_l2 vmur nfsd auth_rpcgss nfs_acl lockd sunrpc xfs libcrc32c dasd_fba_ mod dasd_eckd_mod dasd_mod lcs ctcm fsm qeth qdio ccwgroup dm_mirror dm_region_h ash dm_log dm_mod [ 129.520093] CPU: 0 PID: 730 Comm: systemd-udevd Tainted: GF O------- ------- 3.10.0-123.el7.s390x #1 [ 129.520097] task: 000000000165ecc0 ti: 000000007f040000 task.ti: 000000007f04 0000 [ 129.520100] Krnl PSW : 0704e00180000000 000000000026cf7e (__mem_cgroup_commit _charge+0x4e/0x468) [ 129.520110] R:0 T:1 IO:1 EX:1 Key:0 M:1 W:0 P:0 AS:3 CC:2 PM:0 EA: 3 Krnl GPRS: 0000000000000000 0000000000000001 000000f421d050d8 000000000000f420 [ 129.520119] 000000f421d050d8 0000000000000001 0000000000000000 000 003d101912400 [ 129.520189] 0000000000000001 0000000000000001 000000007ff79000 000 0000000000001 [ 129.520196] 000003d101912400 00000000005cd340 000000000026cf6e 000 000007f043bb0 [ 129.520205] Krnl Code: 000000000026cf6e: e320f0a00024 stg %r2,160( %r15) 000000000026cf74: a7190001 lghi %r1,1 #000000000026cf78: e340f0a00004 lg %r4,160(%r15) >000000000026cf7e: e32040000004 lg %r2,0(%r4) 000000000026cf84: b9040032 lgr %r3,%r2 000000000026cf88: b9810031 ogr %r3,%r1 000000000026cf8c: eb2340000030 csg %r2,%r3,0(%r4) 000000000026cf92: a744fff9 brc 4,26cf84 [ 129.520287] Call Trace: [ 129.520291] ([<000003ffff82f000>] 0x3ffff82f000) [ 129.520299] [<000000000026f56e>] mem_cgroup_charge_common+0x9e/0xe0 [ 129.520306] [<0000000000234420>] do_wp_page+0x178/0xa80 [ 129.520313] [<0000000000237afe>] handle_mm_fault+0x7a6/0xe98 [ 129.520318] [<00000000005b3588>] 01: HCPGSP2629I The virtual machine is plac ed in CP mode due to a SIGP stop from CPU 01. do_protection_exception+0x198/0x3d0 [ 129.520367] [<00000000005b1de6>] pgm_check_handler+0x17a/0x17e [ 129.520369] [<000003fffd5b0900>] 0x3fffd5b0900 [ 129.520371] Last Breaking-Event-Address: [ 129.520373] [<0000000000271f6a>] lookup_page_cgroup+0x42/0x48 [ 129.520400] [ 129.520403] Kernel panic - not syncing: Fatal exception: panic_on_oops 00: HCPGIR450W CP entered; disabled wait PSW 00020001 80000000 00000000 0010DEEE ====
OK, I've finally narrowed this one down a bit more. There are 2 problems here. There are 2 functions that crash the kernel when a kprobe is placed on them, without systemtap involved. I verified this using the scripts down in src/scripts/kprobes_test. They are: set_pageblock_flags_group() lookup_page_cgroup() I've filed bugzilla bugs on each of those: BZ1123425 - kprobe on set_pageblock_flags_group() causes kernel panic on s390x BZ1123429 - kprobe on lookup_page_cgroup() causes kernel panic on s390x We will probably need to add those functions to the blacklist. With those 2 functions removed from the list produced by: stap -l 'kernel.function("*@mm/*.c").call' I still see a crash. So, I modified the scripts in src/scripts/kprobes_test to build systemtap modules instead of straight kernel modules. After running that, it appears that the following function is the culprit: free_pages() Here's the crash you get when probing free_pages(): ==== [ 6071.705497] Kernel BUG at 00000000002118b6 [verbose debug info unavailable] [ 6071.705535] specification exception: 0006 [#1] SMP [ 6071.705537] Modules linked in: probe_module(OF) tun ext4 mbcache jbd2 loop sg qeth_l2 vmur nfsd auth_rpcgss nfs_acl lockd sunrpc xfs libcrc32c dasd_fba_mod l cs ctcm fsm dasd_eckd_mod qeth qdio dasd_mod ccwgroup dm_mirror dm_region_hash d m_log dm_mod [last unloaded: probe_module] [ 6071.705564] CPU: 0 PID: 34156 Comm: basename Tainted: GF O---------- ---- 3.10.0-123.el7.s390x #1 [ 6071.705568] task: 000000007c87daa0 ti: 0000000068d50000 task.ti: 0000000068d5 0000 [ 6071.705571] Krnl PSW : 0704e00180000000 00000000002118b6 (__free_pages+0x36/0 x90) [ 6071.705580] R:0 T:1 IO:1 EX:1 Key:0 M:1 W:0 P:0 AS:3 CC:2 PM:0 EA: 3 Krnl GPRS: 0000000000000001 0000000000000001 000000000acfcecc 000000000acfcecd [ 6071.705588] 000003ff7fffffff 0000000000000000 0000000080000000 000 000003fda0008 [ 6071.705592] 0000000068d53e00 00000000ae64dfff 00000000ae64e000 000 000001d1e9738 [ 6071.706153] ------------[ cut here ]------------ [ 6071.706154] Kernel BUG at 00000000002118b6 [verbose debug info unavailable] [ 6071.708530] 0000000000000002 0000000000747a01 0000000068d53c30 000 0000068d53c08 [ 6071.708543] Krnl Code: 00000000002118ac: d01c18231b21 trtr 2083(29, %r1),2849(%r1) #00000000002118b2: ba32d01c cs %r3,%r2,28(%r13) >00000000002118b6: a744fffc brc 4,2118ae 00000000002118ba: ec260010007e cij %r2,0,6,2118da 00000000002118c0: b904002d lgr %r2,%r13 00000000002118c4: ecc80012007c cgij %r12,0,8,2118e8 00000000002118ca: b904003c lgr %r3,%r12 00000000002118ce: c0e5ffffeef5 brasl %r14,20f6b8 [ 6071.708595] Call Trace: [ 6071.708599] ([<00000000ae64dfff>] 0xae64dfff) [ 6071.708606] [<000000000023316a>] free_pgd_range+0x40a/0x480 [ 6071.708613] [<00000000002332ce>] free_pgtables+0xee/0x148 [ 6071.708619] [<000000000023e84c>] 01: HCPGSP2629I The virtual machine is plac ed in CP mode due to a SIGP stop from CPU 01. exit_mmap+0x12c/0x1c8 [ 6071.708656] [<000000000012d8ae>] mmput+0x7e/0x138 [ 6071.708659] [<000000000013723e>] do_exit+0x2be/0xa88 [ 6071.708663] [<0000000000137abe>] do_group_exit+0x4e/0xe0 [ 6071.708679] [<0000000000137b7a>] SyS_exit_group+0x2a/0x30 [ 6071.708682] [<00000000005b1c1c>] sysc_tracego+0x14/0x1a [ 6071.708687] [<000003fffd624694>] 0x3fffd624694 [ 6071.708690] Last Breaking-Event-Address: [ 6071.708692] [<0000000000211920>] free_pages.part.49+0x10/0x18 [ 6071.708696] [ 6071.708698] Kernel panic - not syncing: Fatal exception: panic_on_oops [ 6071.708701] specification exception: 0006 [#2] SMP [ 6071.708706] Modules linked in:00: HCPGIR450W CP entered; disabled wait PSW 00 020001 80000000 00000000 0010DEEE ==== Here's the source to free_pages(): ==== void free_pages(unsigned long addr, unsigned int order) { if (addr != 0) { VM_BUG_ON(!virt_addr_valid((void *)addr)); __free_pages(virt_to_page((void *)addr), order); } } ==== So, I'm guessing we're hitting that VM_BUG_ON().
Here's a small update. This BUG doesn't happen on the debug kernel (which is odd). The BUG still happens with -DSTP_ALIBI.
Here's additional information. The following works fine (using 'kprobe.function'): # stap -DSTP_ALIBI -ve 'probe kprobe.function("free_pages").call { printf("here\n") }' Bug the corresponding 'kernel.function' version crashes almost immediately: # stap -DSTP_ALIBI -ve 'probe kernel.function("free_pages").call { printf("here\n") }' Looking at the translator's C output, 'kernel.function("free_pages").call' puts probes at 2 addresses: { .address=(unsigned long)0x111928ULL, .module="kernel", .section="_stext", .probe=(&stap_probes[0]), }, { .address=(unsigned long)0x111910ULL, .module="kernel", .section="_stext", .probe=(&stap_probes[1]), }, }; Putting a probe at the first address, 0x111928ULL, works fine. Putting a probe at the 2nd address, 0x111910ULL, causes the BUG to appear immediately.
Here's the output of the eu-readelf that relates to free_pages() # eu-readelf --debug-dump=loc /usr/lib/debug/lib/modules/3.10.0-123.el7.s390x/vmlinux ==== [422a8b] 0x0000000000211910 <free_pages.part.49>..0x000000000021191c <free_pages.part.49+0xc> [ 0] reg2 0x000000000021191c <free_pages.part.49+0xc>..0x0000000000211926 [ 0] GNU_entry_value: [ 0] reg2 [ 3] stack_value [422ac4] 0x0000000000211910 <free_pages.part.49>..0x0000000000211925 <free_pages.part.49+0x15> [ 0] reg3 0x0000000000211925 <free_pages.part.49+0x15>..0x0000000000211926 [ 0] GNU_entry_value: [ 0] reg3 [ 3] stack_value [422afd] 0x0000000000211928 <free_pages>..0x0000000000211935 <free_pages+0xd> [ 0] reg2 0x0000000000211935 <free_pages+0xd>..0x0000000000211936 [ 0] GNU_entry_value: [ 0] reg2 [ 3] stack_value [422b36] 0x0000000000211928 <free_pages>..0x0000000000211935 <free_pages+0xd> [ 0] reg3 0x0000000000211935 <free_pages+0xd>..0x0000000000211936 [ 0] GNU_entry_value: [ 0] reg3 [ 3] stack_value ==== So, it looks like the 'free_pages.part' address doesn't work, while the 'free_pages' address does.
Here's disassembly output for 'free_pages' and 'free_pages.part.49': ==== # objdump -d --start-address=0x0000000000211910 --stop-address=0x0000000000211938 /usr/lib/debug/lib/modules/3.10.0-123.el7.s390x/vmlinux /usr/lib/debug/lib/modules/3.10.0-123.el7.s390x/vmlinux: file format elf64-s390 Disassembly of section .text: 0000000000211910 <free_pages.part.49>: 211910: c4 18 00 39 29 34 lgrl %r1,936b78 <vmemmap> 211916: ec 22 06 b9 3a 55 risbg %r2,%r2,6,185,58 21191c: b9 08 00 21 agr %r2,%r1 211920: c0 f4 ff ff ff b0 jg 211880 <__free_pages> 211926: 07 07 nopr %r7 0000000000211928 <free_pages>: 211928: ec 26 00 04 00 7c cgij %r2,0,6,211930 <free_pages+0x8> 21192e: 07 fe br %r14 211930: c0 f4 ff ff ff f0 jg 211910 <free_pages.part.49> 211936: 07 07 nopr %r7 ====
This has been fixed in the upstream kernel and the fix backported to RHEL7. It should be fixed in kernels >= 3.10.0-155.el7.s390x. I just verified it worked in 3.10.0-217.el7.s390x
Looks good using kernel-3.10.0-220.el7.s390x and stap based on commit 772e206. Thank you.