[Bug runtime/26537] New: stapio might get stuck in Dl state forever

[agentzh at gmail dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=26537

            Bug ID: 26537
           Summary: stapio might get stuck in Dl state forever
           Product: systemtap
           Version: unspecified
            Status: UNCONFIRMED
          Severity: normal
          Priority: P2
         Component: runtime
          Assignee: systemtap at sourceware dot org
          Reporter: agentzh at gmail dot com
  Target Milestone: ---

We've noted that sometimes especially under load, the stapio process may get
stuck in the Dl state forever:

```
# ps aux | grep stapio | grep -v grep
root     47763  0.2  0.0  23000  1636 ?        Dl   10:00   1:23 stapio -R -v
-o c-fgraph.bt stap_c_fgraph_X_47763 var1_total_frames=100
var1_sample_frames=100 -F3
```

The kernel backtrace looks interesting:

```
# sudo cat /proc/47763/stack
[<ffffffffc0a7c923>] stp_task_work_exit+0x28/0x4a [stap_c_fgraph_X_47763]
[<ffffffffc0a8fcb5>] utrace_exit+0x20/0x236 [stap_c_fgraph_X_47763]
[<ffffffffc0a8ff36>] stap_stop_task_finder+0x6b/0x7c [stap_c_fgraph_X_47763]
[<ffffffffc0a90a98>] systemtap_module_exit+0x69/0x24b [stap_c_fgraph_X_47763]
[<ffffffffc0a90d66>] _stp_cleanup_and_exit+0xec/0x10b [stap_c_fgraph_X_47763]
[<ffffffffc0a90e5c>] _stp_ctl_write_cmd+0xd7/0x298 [stap_c_fgraph_X_47763]
[<ffffffff9c324f04>] full_proxy_write+0x64/0x9c
[<ffffffff9c2848ba>] __vfs_write+0x3a/0x16d
[<ffffffff9c284b92>] vfs_write+0xb2/0x1a1
[<ffffffff9c284df5>] SyS_write+0x55/0xb9
[<ffffffff9c003949>] do_syscall_64+0x79/0x1b9
[<ffffffff9ca001ad>] entry_SYSCALL_64_after_hwframe+0x169/0x0
[<ffffffffffffffff>] 0xffffffffffffffff
```

And below are all the backtraces for all the OS threads of this stapio process:


```
# sudo cat /proc/47763/task/*/stack
[<ffffffffc0a7c923>] stp_task_work_exit+0x28/0x4a [stap_c_fgraph_X_47763]
[<ffffffffc0a8fcb5>] utrace_exit+0x20/0x236 [stap_c_fgraph_X_47763]
[<ffffffffc0a8ff36>] stap_stop_task_finder+0x6b/0x7c [stap_c_fgraph_X_47763]
[<ffffffffc0a90a98>] systemtap_module_exit+0x69/0x24b [stap_c_fgraph_X_47763]
[<ffffffffc0a90d66>] _stp_cleanup_and_exit+0xec/0x10b [stap_c_fgraph_X_47763]
[<ffffffffc0a90e5c>] _stp_ctl_write_cmd+0xd7/0x298 [stap_c_fgraph_X_47763]
[<ffffffff9c324f04>] full_proxy_write+0x64/0x9c
[<ffffffff9c2848ba>] __vfs_write+0x3a/0x16d
[<ffffffff9c284b92>] vfs_write+0xb2/0x1a1
[<ffffffff9c284df5>] SyS_write+0x55/0xb9
[<ffffffff9c003949>] do_syscall_64+0x79/0x1b9
[<ffffffff9ca001ad>] entry_SYSCALL_64_after_hwframe+0x169/0x0
[<ffffffffffffffff>] 0xffffffffffffffff
[<ffffffff9c0a5687>] do_sigtimedwait+0x1a7/0x241
[<ffffffff9c0a8ef5>] SYSC_rt_sigtimedwait+0x85/0xe1
[<ffffffff9c0a8f6e>] SyS_rt_sigtimedwait+0xe/0x10
[<ffffffff9c003949>] do_syscall_64+0x79/0x1b9
[<ffffffff9ca001ad>] entry_SYSCALL_64_after_hwframe+0x169/0x0
[<ffffffffffffffff>] 0xffffffffffffffff
[<ffffffff9c29b28a>] poll_schedule_timeout+0x4a/0x67
[<ffffffff9c29c648>] do_sys_poll+0x3d8/0x54d
[<ffffffff9c29d0be>] SyS_ppoll+0xce/0x189
[<ffffffff9c003949>] do_syscall_64+0x79/0x1b9
[<ffffffff9ca001ad>] entry_SYSCALL_64_after_hwframe+0x169/0x0
[<ffffffffffffffff>] 0xffffffffffffffff
```

Seems like only the first thread is interesting.

The `stp_task_work_exit` function is very short:

```
static void
stp_task_work_exit(void)
{
        while (atomic_read(&stp_task_work_callbacks))
                schedule_timeout_uninterruptible(1);
        return;
}
```

Seems like it is stuck in the loop here.

-- 
You are receiving this mail because:
You are the assignee for the bug.