特别声明：本系列文章LiAnLab.org著作权所有，转载请注明出处。by @宋宝华Barry
Vanilla kernel的问题
Linux kernel在spinlock、irq上下文方面无法抢占，因此高优先级任务被唤醒到得以执行的时间并不能完全确定。同时，Linux kernel本身也不处理优先级反转。RT-Preempt Patch是在Linux社区kernel的基础上，加上相关的补丁，以使得Linux满足硬实时的需求。本文描述了该patch在PC上的实践。我们的测试环境为Ubuntu 10.10，默认情况下使用Ubuntu 10.10自带的kernel：
barry@barry-VirtualBox:/lib/modules$ uname -a 2.6.35-32-generic #67-Ubuntu SMP Mon Mar 5 19:35:26 UTC 2012 i686 GNU/Linux
在Ubuntu 10.10，apt-get install rt-tests安装rt测试工具集，运行其中的cyclictest测试工具，默认创建5个SCHED_FIFO策略的realtime线程，优先级76-80，运行周期是1000,1500,2000,2500,3000微秒：
barry@barry-VirtualBox:~/development/panda/android$ sudo cyclictest -p 80 -t5 -n [sudo] password for barry: policy: fifo: loadavg: 9.22 8.57 6.75 11/374 21385 T: 0 (20606) P:80 I:1000 C: 18973 Min: 26 Act: 76 Avg: 428 Max: 12637 T: 1 (20607) P:79 I:1500 C: 12648 Min: 31 Act: 68 Avg: 447 Max: 10320 T: 2 (20608) P:78 I:2000 C: 9494 Min: 28 Act: 151 Avg: 383 Max: 9481 T: 3 (20609) P:77 I:2500 C: 7589 Min: 29 Act: 889 Avg: 393 Max: 12670 T: 4 (20610) P:76 I:3000 C: 6325 Min: 37 Act: 167 Avg: 553 Max: 13673
由此可见在标准Linux内，rt线程投入运行的jitter非常不稳定，最小值在26-37微秒，平均值为68-889微秒，而最大值则分布在9481-13673微秒之间。
我们还是运行这个测试，但是在运行这个测试的过程中引入更多干扰，如mount /dev/sdb1 ~/development，则结果变为：
barry@barry-VirtualBox:~$ sudo cyclictest -p 80 -t5 -n policy: fifo: loadavg: 0.14 0.29 0.13 2/308 1908 T: 0 ( 1874) P:80 I:1000 C: 28521 Min: 0 Act: 440 Avg: 2095 Max: 331482 T: 1 ( 1875) P:79 I:1500 C: 19014 Min: 2 Act: 988 Avg: 2099 Max: 330503 T: 2 ( 1876) P:78 I:2000 C: 14261 Min: 7 Act: 534 Avg: 2096 Max: 329989 T: 3 ( 1877) P:77 I:2500 C: 11409 Min: 4 Act: 554 Avg: 2073 Max: 328490 T: 4 ( 1878) P:76 I:3000 C: 9507 Min: 12 Act: 100 Avg: 2081 Max: 328991
mount过程中引入的irq、softirq和spinlock导致最大jitter明显地加大甚至达到了331482us，充分显示出了标准Linux内核中RT线程投入运行时间的不可预期性(硬实时要求意味着可预期)。
如果我们编译一份kernel，选择的是“Voluntary Kernel Preemption (Desktop)“，这类似于2.4不支持kernel抢占的情况，我们运行同样的case，时间的不确定性大地几乎让我们无法接受：
barry@barry-VirtualBox:~$ sudo /usr/local/bin/cyclictest -p 80 -t5 -n # /dev/cpu_dma_latency set to 0us policy: fifo: loadavg: 0.23 0.30 0.15 3/247 5086 T: 0 ( 5082) P:80 I:1000 C: 5637 Min: 60 Act:15108679 Avg:11195196 Max:15108679 T: 1 ( 5083) P:80 I:1500 C: 5723 Min: 48 Act:12364955 Avg:6389691 Max:12364955 T: 2 ( 5084) P:80 I:2000 C: 4821 Min: 32 Act:11119979 Avg:8061814 Max:11661123 T: 3 ( 5085) P:80 I:2500 C: 3909 Min: 27 Act:11176854 Avg:4563549 Max:11176854 T: 4 ( 5086) P:80 I:3000 C: 3598 Min: 37 Act:9951432 Avg:8761137 Max:116026155
RT-Preempt Patch使能
RT-Preempt Patch对Linux kernel的主要改造包括：
- Making in-kernel locking-primitives (using spinlocks) preemptible though reimplementation with rtmutexes:
- Critical sections protected by i.e. spinlock_t and rwlock_t are now preemptible. The creation of non-preemptible sections (in kernel) is still possible with raw_spinlock_t (same APIs like spinlock_t)
- Implementing priority inheritance for in-kernel spinlocks and semaphores. For more information on priority inversion and priority inheritance please consultIntroduction
to Priority Inversion - Converting interrupt handlers into preemptible kernel threads: The RT-Preempt patch treats soft interrupt handlers in kernel thread context, which is represented by a task_struct like a common userspace process. However it is also possible to register an
IRQ in kernel context. - Converting the old Linux timer API into separate infrastructures for high resolution kernel timers plus one for timeouts, leading to userspace POSIX timers with high resolution.
在本试验中，我们取的带RT-Preempt Patch的kernel tree是git://git.kernel.org/pub/scm/linux/kernel/git/rt/linux-stable- rt.git，使用其v3.4-rt-rebase branch，编译kernel时选中了"Fully Preemptible Kernel"抢占模型:
───────────────────────── Preemption Model ─────────────────────────┐
│ │ ( ) No Forced Preemption (Server)
│ │ ( ) Voluntary Kernel Preemption (Desktop)
│ │ ( ) Preemptible Kernel (Low-Latency Desktop)
│ │ ( ) Preemptible Kernel (Basic RT)
│ │ (X) Fully Preemptible Kernel (RT)
另外，kernel中需支持tickless和高精度timer：
┌───────────────────Processor type and features ─────────────────────────┐
│ │ [*] Tickless System (Dynamic Ticks)
│ │ [*] High Resolution Timer Support
make modules_install、make install、mkintramfs后，我们得到一个可以在Ubuntu中启动的RT kernel。具体编译方法可详见http://www.linuxidc.com/Linux/2012-01/50749.htm，根据该文修改版本号等信息即可，我们运行的命令包括：
安装模块
barry@barry-VirtualBox:~/development/linux-2.6$ sudo make modules_install .... INSTALL /lib/firmware/whiteheat_loader.fw INSTALL /lib/firmware/whiteheat.fw INSTALL /lib/firmware/keyspan_pda/keyspan_pda.fw INSTALL /lib/firmware/keyspan_pda/xircom_pgs.fw INSTALL /lib/firmware/cpia2/stv0672_vp4.bin INSTALL /lib/firmware/yam/1200.bin INSTALL /lib/firmware/yam/9600.bin DEPMOD 3.4.11-rt19
安装kernel
barry@barry-VirtualBox:~/development/linux-2.6$ sudo make install sh /home/barry/development/linux-2.6/arch/x86/boot/install.sh 3.4.11-rt19 arch/x86/boot/bzImage \ System.map "/boot"
制作initrd
barry@barry-VirtualBox:~/development/linux-2.6$ sudo mkinitramfs 3.4.11-rt19 -o /boot/initrd.img-3.4.11-rt19
修改grub配置
在grub.conf中增加新的启动entry，仿照现有的menuentry，增加一个新的，把其中的相关版本号都变更为3.4.11-rt19，我们的修改如下：
menuentry 'Ubuntu, with Linux 3.4.11-rt19' --class ubuntu --class gnu-linux --class gnu --class os {
recordfail
insmod part_msdos
insmod ext2
set root='(hd0,msdos1)'
search --no-floppy --fs-uuid --set a0db5cf0-6ce3-404f-9808-88ce18f0177a
linux /boot/vmlinuz-3.4.11-rt19 root=UUID=a0db5cf0-6ce3-404f-9808-88ce18f0177a ro quiet splash
initrd /boot/initrd.img-3.4.11-rt19
}
开机时选择3.4.11-rt19启动：
RT-Preempt Patch试用
运行同样的测试cyclictest benchmark工具，结果迥异：
barry@barry-VirtualBox:~$ sudo cyclictest -p 80 -t5 -n WARNING: Most functions require kernel 2.6 policy: fifo: loadavg: 0.71 0.42 0.17 1/289 1926 T: 0 ( 1921) P:80 I:1000 C: 7294 Min: 7 Act: 89 Avg: 197 Max: 3177 T: 1 ( 1922) P:79 I:1500 C: 4863 Min: 10 Act: 85 Avg: 186 Max: 2681 T: 2 ( 1923) P:78 I:2000 C: 3647 Min: 15 Act: 93 Avg: 160 Max: 2504 T: 3 ( 1924) P:77 I:2500 C: 2918 Min: 23 Act: 67 Avg: 171 Max: 2114 T: 4 ( 1925) P:76 I:3000 C: 2432 Min: 19 Act: 134 Avg: 339 Max: 3129
我们还是运行这个测试，但是在运行这个测试的过程中引入更多干扰，如mount /dev/sdb1 ~/development，则结果变为：
barry@barry-VirtualBox:~$ sudo cyclictest -p 80 -t5 -n # /dev/cpu_dma_latency set to 0us policy: fifo: loadavg: 0.11 0.12 0.13 1/263 2860 T: 0 ( 2843) P:80 I:1000 C: 28135 Min: 5 Act: 198 Avg: 200 Max: 7387 T: 1 ( 2844) P:80 I:1500 C: 18756 Min: 22 Act: 169 Avg: 188 Max: 6875 T: 2 ( 2845) P:80 I:2000 C: 14067 Min: 7 Act: 91 Avg: 149 Max: 7288 T: 3 ( 2846) P:80 I:2500 C: 11254 Min: 19 Act: 131 Avg: 155 Max: 6287 T: 4 ( 2847) P:80 I:3000 C: 9378 Min: 25 Act: 58 Avg: 172 Max: 6121
时间在可预期的范围内，没有出现标准kernel里面jitter达到331482的情况。需要说明的是，这个jitter大到超过了我们的预期，达到了10ms量级，相信是受到了我们的测试都是在Virtualbox虚拟机进行的影响。按照其他文档显示，这个jitter应该在数十us左右。
我们在这个kernel里面运行ps aux命令，可以看出线程化了的irq：
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 1 0.8 0.1 2880 1788 ? Ss 18:39 0:03 init root 2 0.0 0.0 0 0 ? S 18:39 0:00 kthreadd ... root 45 0.0 0.0 0 0 ? S 18:39 0:00 irq/14-ata_piix root 46 0.0 0.0 0 0 ? S 18:39 0:00 irq/15-ata_piix root 50 0.0 0.0 0 0 ? S 18:39 0:00 irq/19-ehci_hcd root 51 0.0 0.0 0 0 ? S 18:39 0:00 irq/22-ohci_hcd root 55 0.0 0.0 0 0 ? S 18:39 0:00 irq/12-i8042 root 56 0.0 0.0 0 0 ? S 18:39 0:00 irq/1-i8042 root 57 0.0 0.0 0 0 ? S 18:39 0:00 irq/8-rtc0 root 863 0.0 0.0 0 0 ? S 18:39 0:00 irq/19-eth0 root 864 0.0 0.0 0 0 ? S 18:39 0:00 irq/16-eth1 root 1002 0.5 0.0 0 0 ? S 18:39 0:01 irq/21-snd_inte ...
在其中编写一个RT 线程的应用程序，通常需要如下步骤：
- Setting a real time scheduling policy and priority.
- Locking memory so that page faults caused by virtual memory will not undermine deterministic behavior
- Pre-faulting the stack, so that a future stack fault will not undermine deterministic behavior
例子test_rt.c，其中的mlockall是为了防止进程的虚拟地址空间对应的物理页面被swap出去，而stack_prefault()则故意提前导致stack往下增长8KB，因此其后的函数调用和局部变量的使用将不再导致栈增长（依赖于page fault和内存申请）：
#include <stdlib.h>
#include <stdio.h>
#include <time.h>
#include <sched.h>
#include <sys/mman.h>
#include <string.h>
#define MY_PRIORITY (49) /* we use 49 as the PRREMPT_RT use 50
as the priority of kernel tasklets
and interrupt handler by default */
#define MAX_SAFE_STACK (8*1024) /* The maximum stack size which is
guaranteed safe to access without
faulting */
#define NSEC_PER_SEC (1000000000) /* The number of nsecs per sec. */
void stack_prefault(void) {
unsigned char dummy[MAX_SAFE_STACK];
memset(dummy, 0, MAX_SAFE_STACK);
return;
}
int main(int argc, char* argv[])
{
struct timespec t;
struct sched_param param;
int interval = 50000; /* 50us*/
/* Declare ourself as a real time task */
param.sched_priority = MY_PRIORITY;
if(sched_setscheduler(0, SCHED_FIFO, ¶m) == -1) {
perror("sched_setscheduler failed");
exit(-1);
}
/* Lock memory */
if(mlockall(MCL_CURRENT|MCL_FUTURE) == -1) {
perror("mlockall failed");
exit(-2);
}
/* Pre-fault our stack */
stack_prefault();
clock_gettime(CLOCK_MONOTONIC ,&t);
/* start after one second */
t.tv_sec++;
while(1) {
/* wait until next shot */
clock_nanosleep(CLOCK_MONOTONIC, TIMER_ABSTIME, &t, NULL);
/* do the stuff */
/* calculate next shot */
t.tv_nsec += interval;
while (t.tv_nsec >= NSEC_PER_SEC) {
t.tv_nsec -= NSEC_PER_SEC;
t.tv_sec++;
}
}
}
编译之：gcc -o test_rt test_rt.c -lrt。本节就到这里，后续我们会有一系列博文来描述RT-Preempt Patch对kernel的主要改动，以及其工作原理。