make is a program that is used to determine which parts of a multi-source-file project need to be rebuilt. Only those source files which have been modified are rebuilt before the final linking step. This is a big plus for the kernel programmer as once the "stock" kernel has been built, recompiles to include changes take seconds instead of minutes. make operates on a Makefile, a file containing the relationships and dependencies between source files along with the commands used to build them.

Just about every directory in the kernel source tree has a makefile, including the top level directory. When you invoke make dep bzImage modules, make operates on the top level makefile which recursively works through the makefiles in each of the subdirectories (depending on which options were included during the config step).

Some programming errors can be caught early on in the development cycle, saving time and effort. You may not have bothered with formal code analysis before, but now would be a good time to start; bugs in kernel code are generally harder to track down and can have more serious implications than their user-space counterparts. Generally speaking, a higher standard of programming is called for; while a user may be prepared to put up with an email client that occasionally core dumps, they certainly won't put up with a kernel that freezes or worse, causes data loss.

lclint is a program that can be used to statically check C code, that is to check it before it is even compiled or executed. Like lint, lclint can be run on unmodified C source code and used to catch classic programming mistakes. However, lclint can do a whole lot more for you, but you must give it "clues" by annotating your source code using comments in the appropriate way.

In short, lclint can give you some of the foresight that an experienced programmer has at their disposal. While it may not be perfectly suited to kernel C code (which has some unusual attributes), it can still be used to prove fragments of code prior to inclusion in the kernel source.

More information on lclint can be found at the lclint home page.

grep is used to find lines matching a given pattern; a useful tool for quickly locating variable or function definitions and use. Example: Suppose you want to find all instances of task_struct:

$grep -r task_struct * | less

This command recursively (-r) scans all files in the source tree and prints lines containing task_struct, those lines are piped to less, which provides a scroll-able output. man grep for more details!

lxr enables you to navigate source code via a web browser, where all variables and functions are links to their respective definitions. Probably the best way to read and understand code.

You can browse the kernel source code online at the Linux Cross Reference Page or at innominate.org, or download lxr and set it up on your box, useful if you don't have a permanent Internet connection. The setup process is quite involved but details are given in the README file that comes with the lxr source code.

cscope is useful for doing many of the things you could use grep for, but is more intelligent and provides a nicer interface to work with. You can search for definitions, uses, strings etc.

Before you can use cscope you need to build an index file. This can be done by issuing the command cscope -b -R -k in the top level source directory; -b to build the index, -R to search recursively through the source tree and -k to indicate kernel use; this ensures the appropriate include files are used when generating the index.

To start a cscope session, type cscope -d. You will then get something that looks like this:

Cscope version 15.3                                    Press the ? key for help













Find this C symbol:
Find this global definition:
Find functions called by this function:
Find functions calling this function:
Find this text string:
Change this text string:
Find this egrep pattern:
Find this file:
Find files #including this file:

The top half of the screen is used to display search results, while the lower half is used to issue commands. Example; suppose you want to find the definition of the file_system_type data structure. Use the arrow keys to move the cursor to the Find this global definition: field and enter the name of the data structure, followed by enter. If all is well, cscope will find just one instance and will open your favourite editor (set by the EDITOR environment variable) to display the appropriate section of the file.

Now suppose you want to find the definition of super_block. Follow the procedure above, which should give you this output:

Global definition: super_block

  File          Line
0 vxfs_extern.h  44 struct super_block;
1 super.c       265 int (*test)(struct super_block *, struct buffer_head *);
2 udfdecl.h      52 struct super_block;
3 fs.h          688 struct super_block 
4 fs.h          936 struct super_block *(*read_super) (struct super_block *,
                    void *, int );
5 udf_fs_sb.h    71 __u32 (*s_partition_func)(struct super_block *, __u32,
                    __u16, __u32);



Find this C symbol:
Find this global definition:
Find functions called by this function:
Find functions calling this function:
Find this text string:
Change this text string:
Find this egrep pattern:
Find this file:
Find files #including this file:

This time, cscope has found multiple possible definitions. You can view each definition in its context by pressing the numbers given next to the list of files, alternatively, you can use Tab to move to the top half of the screen and the arrow keys to select a definition. Simply quit out of the editor to return to cscope. Press Tab again to return to the command area. Note that you can also move up and down using Ctrl-p and Ctrl-n, which saves moving your hands away from a typing position. To exit cscope, press Ctrl-d.

cscope source code and documentation is available from the cscope home page, I would advise using version 15.3 or later as this version includes support for recursive indexing and using kernel headers.

ctags provides similar functionality to cscope, but integrates closely with your editor, allowing look-ups with a few key strokes. Like cscope, ctags builds an index file to speed up searches. The easiest way to generate the index is to type make tags in the top level kernel source directory.

When using vim, moving the cursor to a use of a function/variable/data structure/type and pressing Ctrl-] should take you to the definition. Ctrl-t takes you back to where you were. Note that lookups can be nested, in which case, Ctrl-t takes you up one level.

Details for using ctags with emacs are welcome!

ctags can be obtained from the ctags home page. Debian users will want to install the exuberant-ctags package.

A brief summary of the tools presented:

diff is used to compare two files and output any differences between them. When used in unified mode (-u option) to compare two files (original and modified) a "patch" is produced:

diff -u linux-2.4.14/drivers/char/keyboard.c linux/drivers/char/keyboard.c > my_keyboard_patch

Where the linux-2.4.14 directory holds the original, unmodified source tree and linux directory holds the one you have hacked around with. Distribution of ideas and modifications as patches is a lot more convenient and efficient than distributing complete files or source trees.

The procedure above can be used when just one file has been modified, but what if you need to modify a lot files and produce a patch?

diff -urN linux-2.4.14 linux > my_hefty_kernel_patch

Note that the convention is to generate patches from the directory above the top level kernel source directory i.e. /usr/src if you keep your modified kernel source in /usr/src/linux.

Note that if you are generating a patch to post to the Linux Kernel Mailing List, be sure to follow the instructions given in the FAQ exactly. The FAQ can be found at: http://www.tux.org/lkml/

Of course, use of diff is not restricted to generating patches; it is a useful tool for finding out what has changed between two kernel releases. Uncompressed patches are human readable to some extent; the format is fairly self explanatory. You can even grep a patch if you know what you are looking for.

patch is used to apply patches to a file or a source tree:

cd linux-2.4.13patch -p1 patch-2.4.14

This procedure would update a 2.4.13 tree to 2.4.14. The -p1 option strips the top level directory off all filenames within the patch (as you are patching from inside the top level source directory). Of course, you could apply the patch from the directory above, but you would need to have your directories named in the same way as when the patch was generated. Note that patches are often distributed in compressed form, to save bandwidth. You can save disk space (and typing) by uncompressing patches as you apply them:

bzip2 -dc /usr/src/patch-2.4.14.bz2 | patch -p1

Simply replace bzip2 with gzip if the patch was gzipped instead of bzipped.

A useful script is included in the kernel source tree to semi-automate the process of upgrading a source tree by applying successive patches: linux/scripts/patch-kernel. Read the script to see what it does and how to use it, instructions are given in the comments at the top of the file.

It is often a good idea to do a "dry run" of applying a patch, especially if you are patching a heavily modified tree, or are attempting to apply an old patch against a newer tree. Backing out a partially applied patch can be time consuming and generally is not much fun!

bzip2 -dc /usr/src/patch-2.4.14.bz2 | patch -p1 --dry-run

What if the patch does not apply cleanly? If only a couple of files failed, you could apply the patch anyway and sort things out with a text editor afterwards, alternatively you could manually go through and apply the patch by reading it and making the changes by hand. The manual approach is sometimes necessary to resolve conflicting patches and is a useful technique if you want to understand exactly what a patch changes.

What if you want to remove a previously applied patch? Easy:

bzip2 -dc /usr/src/patch-2.4.14.bz2 | patch -R -p1

This would take your source tree back to 2.4.13.

Any development work is an incremental process, particularly so at the debugging stage. It always pays to keep a record of the process using some form of revision control system, so that a bad change can backed out with minimal effort. A very basic revision control system can be implemented by just keeping a backup of a file before making major changes, however this approach tends to become cumbersome very quickly.

Enter RCS, the little brother of CVS. CVS is great for large projects with many contributors, but is overkill for small personal projects. One of the attractions of RCS is it's simplicity.

First create the RCS directory in the same directory as your source files:

mkdir RCS

Then "check in" a source file:

ci -u some-file.c

You will then be prompted to give a description. By default, RCS deletes the working file upon check in, so you will want the -u option which automatically checks the file out again. Check in all the files you are working on in this way.

Make some changes to one of your source files and check it in again. You'll be prompted for a summary of the changes and the version number will be incremented.

Suppose you have made a mess of the working file and want to revert to a known good version (1.7), check it out using this command:

co -l -r1.7 some-file.c

The -l flag locks the file and gives you write access to it (otherwise you get read access only).

RCS stores the initial file and only differences between versions, saving disk space. For more information on RCS, see the following man pages: rcs, ci, co, rcsintro, rcsdiff, rcsclean, rcsmerge, rlog, rcsfile and ident.

While RCS is good for basic projects involving (changes to) a limited number of files, it quickly becomes unwieldy for larger projects, particularly so with contributions from more than one developer. CVS is particularly good in this area.