Developer Setup Tips
Setting up kexec/kdump
The kexec/kdump facility is very useful for diagnosing kernel crashes. On RHEL and Fedora, the kexec-tools package needs to be installed, a clause like crashkernel=64M@64M needs to be added to the kernel's boot command line, and the kdump service needs to be enabled with chkconfig or whatnot. You will need to reboot at least once to activate it all.
To test it, and if you're just feeling cruel to other users, run
# echo c > /proc/sysrq-trigger
If it works, you will be greeted with the boot messages from the kexec-loaded crash kernel. It will save the crashed parent kernel's memory to a big file disk (/var/crash/IPADDRESS-TIMESTAMP/vmcore perhaps). You can affect how much of the memory to save, and whether to compress stuff or not, via /etc/kdump.conf. The author enjoys this configuration option, changing which may require restarting the kdump service.
# grep core_ /etc/kdump.conf core_collector makedumpfile -c -d 19
Finally, what can one do with a crash image? Well, debug it of course. Install the crash tool, which is really a dressed-up copy of gdb. Sic it onto the crash file thusly (on a fedora/RHEL system):
# crash --readnow /usr/lib/debug/lib/modules/`uname -r`/vmlinux /var/crash/DIRECTORY/vmcore crash> bt crash> mod crash> log crash> foreach bt crash> help crash> quit
The log should show any recent systemtap modules. From their base: deadbeef value, one may figure out whether any pointers into the current or former module are at issue.
Setting up a network console on Fedora
When working on a remote system (or a virtual system), sometimes a developer would like to get a console log. One way to do this is by setting up a network console for the machine. We'll call the remote system the client system and the system that will receive the console output the server system.
Client system setup
On the client system, there are several setup steps to go through. Fedora has an init script, /etc/rc.d/init.d/netconsole, that does everything we need. We just need to configure it and turn it on. All these steps will need to be done as root.
1. To configure the init script, edit /etc/sysconfig/netconsole. The item that must be filled in is the SYSLOGADDR field. This can be the IP address of the server system or the name of the server system. For ease of use, just fill in the server system name.
2. (Optional) By default, the listening port of the remote syslog daemon is set to 514. The only reason why you would need to change this is if the server system was logging more than one client system's output. If you need to change this, set SYSLOGPORT to the appropriate value.
3. To configure netconsole to be on when the system boots, run:
# chkconfig netconsole on
4. To turn the netconsole service on now, run:
# service netconsole start
Note that the client system is now setup to start the netconsole service every time the system boots. No more client system configuration is needed.
Server system setup
On the server system, there is just one setup step to go through. You may have to open up the server system's firewall to allow the netconsole traffic through. The GUI way to do this is by selecting the System->Administration->Firewall application. On the "Other Ports" tab, hit "Add" and select the "Port 514 Protocol udp Service syslog" item, then hit "Apply". (If you used a different port on the client system earlier, select that port here.)
Now, when you want to capture the console output from the client system, on the server system run as root:
# nc -dul 514
Alternately, you can configure the existing syslogd daemon on your server to accept these packets. Add "-r" to the syslogd options in /etc/sysconfig/syslog and restart syslogd.
Setting up a serial console for KVM
A simulated serial console for a virtual machine is often more reliable than netconsole. And it sounds easier to say too.
Simply start the qemu/kvm simulator with the additional argument
-serial file:ttyS0.log
and arrange to pass these additional boot options to the guest kernel:
console=ttyS0,57600 console=tty0
Adding
printk.time=1
(or older equivalent) is not a bad idea either.
When using virsh or virt-manager just add the console=ttyS0,57600 console=tty0 to the grub.conf of the guest. Then virsh console <guest-id> will give you the console of the guest on the host. You can get more console output by doing echo 9 | sudo tee /proc/sysrq-trigger on the guest.