Re: Error removing module: Device or resource busy

[Josh Stone <jistone at redhat dot com>]

On 12/22/2011 09:00 PM, Chris Dunlop wrote:
> Hi,
> 
> Linux-3.0.13, systemtap 6.1 and now HEAD(b6a3da4)

Your dump looks like x86_64 - is this on any particular distro?  Are you
building this kernel yourself?

> I've just started using systemtap to help diagnose a problem with zfsonlinux[1].

Cool!

> Whenever I run stap[2], the output ends with:
> 
>   Error removing module 'stap_fcac0085842418e34d8094455dc203e8_1_21605': Device or resource busy.
> 
> (obviously the module name changes) and the module is still loaded:
> 
>   # lsmod | grep stap
>   stap_fcac0085842418e34d8094455dc203e8_1_21605  2896285 1027571582 [permanent]

This is odd.  Reading kernel/module.c, the "[permanent]" should only
come about if the module has no ->exit callback.  And 1027571582 is the
field for the module refcount, which doesn't look plausible.

You must have CONFIG_MODULE_UNLOAD=y, or else the kernel just prints a
dummy " - -" in place of the refcount.  Though I'm not certain how lsmod
translates that, so you might check /proc/modules directly.  But it
seems like your stap modules are being built without this machinery, so
corruption ensues.  Are you sure that /lib/modules/`uname -r`/build
matches the running kernel?

> Using "rmmod -f" on that module results in a kernel NULL pointer dereference[3].

This is bound to crash, even if all else is solved.  If the module exit
routines don't work, then probes may be left installed, and will
certainly die as soon as it tries to call the handler from unloaded
memory.  So don't waste any time trying to get "rmmod -f" working.

Fedora, for instance, does not enable CONFIG_MODULE_FORCE_UNLOAD at all,
because there are very few cases where this would do anything sane.

> Any suggestions about how to avoid the unloadable module?

If you can boot without any of the zfs stuff, then I'd experiment with
that first, just to make sure stap is lined up with all the correct
kernel build infrastructure.  Try something simple, like one of the
syscall examples.

Josh