Most of the information on the web about configuring the Linux kernel are not targeted to trading applications so I thought a little tutorial might be useful.

Isolcpus is the common name for configuring Linux to exclusively dedicate a CPU core to a certain process, such as a trading system executable. It helps minimize random latency spikes that are the result of context switches from the kernel scheduler multiplexing processes on a core.  It comes from the name of the kernel parameter “isolcpus,” but it typically also refers to the other steps in assigning the core to the process.

To follow along you will need to be running Ubuntu 10.10 on a 2+ core machine.

The Isolcpus tutorial

First run the program “top” by just typing “top” into a terminal, then hit “f” and then “j” and then ENTER, which displays the core assigned to each process in column “P”. See how processes are distributed on both cores, 0 and 1:

We are going to isolate core 1 so the trading system can run on it without interrupts or context switches.

First we need to make the bootloader, GRUB2, display during startup so that we can use it to pass a kernel parameter setting to the kernel before it boots. Comment the two GRUB_HIDDEN_* lines in GRUB’s config file like the following:

sudo gedit /etc/default/grub
...
GRUB_DEFAULT=0
#GRUB_HIDDEN_TIMEOUT=0
#GRUB_HIDDEN_TIMEOUT_QUIET=true
GRUB_TIMEOUT=10
...

Now restart your computer and you will see the bootloader pop up listing your choices for the OS. Based on my configuration, I highlight the first one on the list as the one I’ll be using. When it’s highlighted, hit “e” to edit the kernel parameters. Select to the end of the second to last line and then hit the spacebar and then add “isolcpus=1″

So the line you modified should now look something like (it will be wrapped):

linux    /boot/vmlinuz-2.6.35-25-generic root=UUID=0b3b0a1b-49b1-4d60-9de1-dd7854ee8028 ro   quiet splash isolcpus=1

You are done here now so hit CTRL-x to continue booting. Run “top” again with the same column enabled as before. You should now see something like:

Sort by column P by hitting SHIFT-> a lot until it’s sorting by the rightmost column, and then hitting SHIFT-< once to sort by the second to last column, P, which shows which core each process resides on. You should see very few on core 1. The ones that are there have to do with OS utilities like maintaining the filesystem and take up basically no resources.

To run a program on the isolated core, use the command “taskset”. For example if your executable is named “a.out”, then you would call:

taskset 2 ./a.out

taskset will run your program on core 1. The number corresponds to a bitmast saying which CPU cores the process is allowed to use. 2 => 10 => core 1 but not core 0. 3 => 11 => core 1 or core 0. etc. This is called setting the CPU affinity of the process.

You can have a C program print the cores it has been authorized to use with the following code snippet:

void print_cpuaffinity() {
   unsigned long mask = 2; /* processor 1 (0-indexed) */
   unsigned int len = sizeof(mask);
   if (sched_getaffinity(0, len, &mask) < 0) {
      perror("sched_getaffinity");
   }
   printf("my affinity mask is: %08lx\n", mask);
}

Another way to check that “taskset” worked is to watch in “top” for your program to appear and see if in column P it is assigned to core 1.

Basically isolcpus=1 makes it so that the default affinity for a new process is 1 => 01 instead of 3 => 11.

After doing all this, you should find that your system runs more consistently and never “hangs” unexpectedly.

Some other interesting commands for probing your computer’s internals are:

cat /proc/cpuinfo
ntpq -c rl
gcc -v
uname -a
/lib/libc.so.6
cat /proc/interrupts
service irqbalance status