To dump useful debugging information that can be used to diagnose the cause of problem.

To try and prevent the kernel from going out of control and causing data corruption, or worse, damage to hardware (although this is very rare).

Use the debugger to collect the evidence surrounding the problem area(s).

Study the code and think about what is going on. hard.

Try to concentrate on thinking about possible causes of the symptoms you are seeing in the debugger. Then think about the causes of the causes all the way down to the real root of the problem. Write a list of the possibilities, placing them in order of perceived likelihood and rule them out in turn, one by one. The process of clarifying thoughts to write them down can be valuable.

Until you have some experience, you may need to use the debugger to try some ideas out on the fly by changing variable values etc.

Study the code before you set to with the debugger; you will be more productive if you have thought about the code first.

Use the debugger to test your assumptions; bugs often come about as a result of incorrect assumptions (have you ever seen those "we should never get here" debugging messages?).

If your assumptions are proved wrong, make it your business to get to the bottom of why you got it wrong and make a mental note for next time.

Discard the debugger as soon as possible; think of Luke Skywalker again here!

Ignore the real cause of problems that come to light. When you find something amiss, don't just shrug your shoulders and fix things up so that everything looks OK. A classic example is adding another section to a switch statement to cover an eventuality you hadn't thought of. Do you understand why that scenario is occurring? Consider the possibility that your approach and/or assumptions are flawed. Don't go "wall-papering over the cracks"; you'll only have to fix it properly later on (or worse, someone else will).

Blindly use the debugger to narrow a problem down; you could well come to the wrong conclusion and you won't learn as much in the process.

Local kernel debugging (done by running gdb with /proc/kcore as the core file), this approach is of very limited use; amongst other limitations, no breakpoints can be set.

User Mode Linux; a way to run Linux inside Linux. This is good for general hacking as no extra hardware is required but not so good for troubleshooting a hardware specific problem. For more information on User Mode Linux, see: the User Mode Linux Homepage

Local kernel debugging using kdb.

Remote kernel debugging using gdb. Here the kernel under test runs on a physically separate machine and communicates with the controlling machine running the debugger, via a serial link. This is the approach described in this document.

First of all, you need a machine to be your testing box. Note that it does not have to have a monitor, keyboard and mouse. My testing machine was cobbled together from spare parts! A keyboard can be useful though, in order to be able to use the magic SysRq key.

Set up ethernet between the two machines. Check you can ping "ritchie" by using it's hostname.

Build two serial cables using the pin out given below. One cable will be used by gdb, the other will be used to give access to the console of testing. This saves having another monitor on the desk and makes capturing Oopsen a lot easier. The ethernet connection is useful for multiple remote logins and quickly transferring kernel images etc. Here is the serial cable pin out: Din 9 pin to Din 9 pin:

![CDATA[
     Solder-side pins:

     \-----------------------/
      \  1   2   3   4   5  /
       \   6   7   8   9   /
        \-----------------/

     Wiring: (use 7 or 10 wire foil screened cable)

     1
     |
     6---------------4

     2---------------3

     3---------------2

     4---------------6
                     |
                     1

     5---------------5

     7---------------8

     8---------------7
]]>

Connect the cable screen to the chassis of one of the connectors, this will help prevent an earth loop between the two machines. This pin out was taken from the text-terminal-howto, I decided to reproduce it here to remove the confusion over which of the many pin outs to use. a standard null-modem cable may work if you don't want to build the cable(s) yourself, however i had to modify mine to the above configuration to make it work. If you go out and buy a null-modem cable especially, I'd advise getting one with connectors that can be disassembled (i.e. not moulded on connectors) to make modification possible.

If you have a Radio Shack store nearby, here are some part numbers:

Thanks to Pragnesh Sampat for providing this information.

connect the two machines together using the two serial cables, com1 to com1, com2 to com2.

Install ssh on "kernighan".

Install sshd on "ritchie" (this is part of the ssh package for some distributions, including Debian).

Check you can log in via ssh to "ritchie" (note: you won't be able to log in as root, but you can su to root once logged in as a user, should you need to).

Give yourself read/write access to dev/ttyS0 and /dev/ttyS1 on both machines.

Install minicom on both machines.

Download and unpack the kernel source to your home directory on "kernighan".

Download the kgdb patch, to the top level kernel source directory from the kgdb home page

Apply the patch: bash-2.03\$ cat kgdb\_2.2.18.diff |patch -p2

make menuconfig (or copy your standard .config file and make oldconfig).

Select the usual configuration options, adding:

o support for console on serial port under character devices
o kernel support for gdb (new) under kernel hacking.

I would advise compiling all additional options directly into the kernel, rather than as modules to start with.

If you have a keyboard directly attached to your testing machine, you may also want to add magic sysrq key support, be sure to read (and maybe print out) Documentation/sysrq.txt.

Proceed with a normal compilation: make dep bzImage.

Copy the image over to "ritchie": scp arch/i386/boot/bzImage ritchie:bzimage-2.2.18-kgdb (note: you will need to have the same user name on ``kernighan'' and ``ritchie'' for this command to work). Alternatively you could setup and use FTP (less secure).

ssh into "ritchie", become root and move the image into /boot.

create a new entry in /etc/lilo.conf (on "ritchie"):

![CDATA[
image=/boot/bzImage-2.2.18-kgdb
	label=kgdb
	root=/dev/hda1
	read-only
	append=" gdb gdbttys=0 console=ttys1"
]]>

The extra command line options tell the gdb debugging stub to listen on /dev/ttyS0, and to tell the kernel to use /dev/ttyS1 as a serial console. You can also control lilo (i.e. choose which image to boot) from the serial console if you add:

serial = 0,9600n8

to the top of your lilo configuration file.

Run lilo (on testing).

If you decided not to make the debugging kernel the default kernel, run lilo -R kgdb to make the new image boot following the next reboot only. (That way if the debugging kernel fails to boot for whatever reason, the machine will boot a working kernel next time to enable you to resolve the problem).

Create a file named .gdbinit in the top level directory of the kernel source tree on "kernighan", containing the following:

define rmt
set remotebaud 38400
target remote /dev/ttyS0
end

Run (as root) minicom -s on "kernighan", go to serial port setup and select these options:

![CDATA[
Serial Device          : /dev/ttyS1
Lockfile Location      : /var/lock
Callin Program         :
Callout Program        :
Bps/Par/Bits           : 9600 8N1
Hardware Flow Control  : No
Software Flow Control  : No
]]>

Go to "Save setup as dfl", and save settings as default before going to Exit, leaving minicom awaiting input.