AbstractThis article is based on a masters thesis written by Martin Andersson and Jens-Henrik Lindskov. The work was carried out at Axis Communications AB during autumn 2002.



Real-time, embedded systems and Linux are commonly used words in these days. The thesis on which this article is based looks deeper into the possibility of turning Linux into a real-time operating system. Particularly, it investigates available hard real-time solutions for Linux, but also looks into the soft variants for completeness.

RTAI is selected as a suitable solution for Axis and is ported to the ETRAX 100LX, an Axis-developed system-on-chip processor designed with networking and embedded systems in mind. A large number of performance tests are conducted during the evaluation to make sure that the implementation meets the demands of a real-time operating system.

The evaluation shows that RTAI provides good real-time performance, especially when compared to standard Linux.

---

Introduction

The purpose of this article is to, in a not so technical way, describe the results found in the master thesis upon which it is based. The investigation covers how real-time performance can be achieved in Linux and the availability of hard real-time extensions. Axis' need for real-time is examined in order to select a suitable extension for implementation on ETRAX. The goals of the evaluation are to verify the functionality of the system as well as measure its real-time performance on the ETRAX platform.

Axis Communications and many other device manufacturers are turning to Linux for embedded systems. Using a full-featured UNIX-like operating system obviously has many advantages. However, the Linux kernel can not support hard real-time processing, which may be required by certain embedded applications. There exists a number of extensions to Linux that provide support for hard real-time tasks.

Axis uses the ETRAX processors in most of its products and has previously ported Linux and some real-time operating systems to ETRAX. However, Axis and its customers have an interest in being able to combine Linux with hard real-time, as this combination would provide both a full-featured free UNIX-like system with many available applications, hardware drivers, etc. and hard real-time capabilities.

As a developer of network cameras, video servers, print servers etc., Axis and its customers would benefit from guaranteed response times in their systems. Possible applications include: sampling of data under time restrictions; surveillance cameras that must respond quickly; and wireless baseband in software, for example Bluetooth implementations; control loops.

Real-time Systems

The purpose of this section is to explain in short the concept of real-time and what can be expected from a real-time operating system. The different approaches towards providing real-time in Linux are discussed. Then a comparison of two hard real-time extensions to Linux is made and finally a selection of one of them is motivated, based on the requirements of Axis.

A real-time system is a system in which the correctness of the system depends not only on the logical results that the system produces, but also on the time at which the results are produced [Note 1]. This is a formal definition of a real-time system. Before proceeding it is appropriate to define and explain some other related concepts.

• The response time of an application is the time interval from when the application receives a stimulus, usually provided via a hardware interrupt, to when the application has produced a result based on that stimulus [Note 2].
  
• The deadline of a certain task in an application is the longest acceptable response time for the task.

For example, say we have a robot arm picking up components from a conveyor belt. An optical sensor informs the robot when a component is approaching the arm. After the robot has received the information there is a certain small amount of time available for the robot to react and for the arm to move down to the right position over the conveyor belt. This time is the deadline for the "move-arm"-task of the robot application. If the arm is not there in time, i.e. if the deadline is missed, the component may be lost. If the arm reaches the position in time we say that the deadline is met.

A hard real-time system is a system in which all the deadlines of the system must be met at all times. It is the system designer s responsibility to make sure that the deadlines can be met, that is the system must not be overloaded. A soft real-time system, on the other hand, is a system in which the deadlines usually must be met, but it may be acceptable if a small number of deadlines occasionally are missed [Note 2]. An example of a hard real-time system is an air-traffic controller, here it is critical that every deadline is met. An example of a soft real-time system is an audio sampling application where it may be acceptable if some samples are lost from time to time, as long as it does not happen too often.

Linux and Real-Time

A real-time operating system (RTOS) can be described as a system that meets timing requirements of the processes under its control. Linux is not designed to provide real-time performance. It provides a good average performance for applications. For a real-time application with relatively long deadlines it may be sufficient if the environment can be controlled properly (fixed number of processes, well-tested drivers etc.). However, for applications that require very low response times, or hard real-time, the standard Linux kernel is not sufficient. It is clear that in order to have hard real-time, guaranties must exist that no deadline is missed. This type of guaranties require a deterministic environment. For an operating system this means that it must be possible to predict the maximum time it takes to perform different tasks, such as interrupt handling and scheduling. Also, the kernel must be preemptible, that is if a lower priority process is running a system call in the kernel, it must be possible to interrupt it if a higher priority process is ready to run. This is currently not the case with Linux (2.4).

• Improving the Kernel -- There are two different approaches towards providing real-time in Linux. In the first one, the standard Linux kernel is improved, either by attempting to make the kernel preemptible (by altering the kernel in different clever ways -- [Note 3]) or by adding preemption points to the code (i.e. checking more often if a higher priority process is ready to run). This results in a kernel more responsive to applications without any need for alterations in these applications [Note 3]. This approach is sometimes used together with a new improved scheduler implementation. It should be noted that in the coming releases of the Linux kernel, the preemption patches are included by default and the functionality is available as a configure option. Thus, the Linux kernel is expected to provide better real-time performance in the future.
  
  
• Adding a Real-Time Kernel -- Another approach is to make the Linux kernel fully preemptible by adding a hardware abstraction layer between the system hardware and Linux. Also a new separate real-time scheduler is used which runs Linux as its lowest priority thread. The abstraction layer takes control over the system interrupts and passes them on to Linux only if no real-time task is running. When Linux tries to disable interrupts it only sets a flag in the abstraction layer and cannot really turn off the interrupts. Thus, the real-time scheduler has full control over the system and Linux runs virtually unmodified. A small real-time kernel has been added to the system. The real-time tasks are written as kernel modules and executed within kernel-space. They have access to a special real-time API. There are two projects providing this technique, RTLinux [Note 4] and RTAI [Note 5]. RTLinux is the oldest project of the two, in fact, RTAI is based on the ideas behind RTLinux.

The two approaches both have their pros and cons. In the preemption improvement approach the major disadvantage is the lack of guaranties it can provide. Unless every possible code path in the kernel is examined, it is impossible to provide a guarantee about the latency [Note 6]. It is clear that analyzing all possible paths is very hard, and even if it was possible under some restrictions, the development of the kernel and addition of new drivers etc. would make the analysis hard to maintain. Also new code would have to meet the requirement not to introduce additional long non-preemptible kernel code paths. The motivation for the preemption improvement approach is the fact that it improves the Linux kernel without the users having to modify their applications.

In the approach based on a hardware abstraction layer the application is preferably split up into two parts:

• The part with timing requirements (e.g. data sampling). It executes as a real-time task in kernel-space.
  
• The other non time-critical part (e.g. user interface) executing in user-space.

The different parts communicate with each other using for example FIFO-queues or shared memory. The split often requires a new design of the application. Also, the real-time tasks are written as kernel modules using the special real-time API, not the standard Linux API. Writing Linux kernel modules requires different programming skills than it does writing a Linux application process [Note 6]. However this is a small price to pay, since this approach can provide the deterministic environment required for hard real-time. This is possible mainly because the API-functions available to the real-time tasks are well tested and analyzed, as are the internal functions of the small real-time kernel. The amount of code which must be analyzed in this approach is relatively small and contained, compared to the entire Linux kernel. Another advantage with this approach is that Linux runs virtually unmodified. Thus, it is easy to maintain this solution when new kernels and drivers are released.

To summarize, in the preemption improvement approach, all applications can without modification benefit from an improved Linux kernel with low latency; soft real-time can be obtained. But only the hardware abstraction approach can provide the guarantees required to support hard real-time, with the extra programming effort as the only major downside.

As hard real-time is the most interesting and useful for Axis, the two alternatives that will be further examined are RTAI and RTLinux. It is also these extensions that provide the best response times.

Evaluation of RTAI vs. RTLinux

The two projects chosen for evaluation use the same basic design. We will start with a short description of this design and then move on to examine the projects one at a time. The figure below shows a Linux system with a small real-time kernel. As shown in the figure, a hardware abstraction layer is added between the system hardware and the standard Linux kernel. A preemptive fixed-priority scheduler handles the real-time tasks as well as the Linux kernel, which runs as the lowest priority task. The abstraction layer intercepts all hardware interrupts from the underlying system. If an interrupt is not related to any real-time task, it is passed on to Linux, but only when no real-time task is running.



A Linux system with a small real-time kernel

The real-time kernel is in itself non-preemptible. Unpredictable delays within the real-time kernel are eliminated by its small size and limited operations. A real-time task is written as a kernel module and when loaded it informs the real-time scheduler about its deadline, period and releasetime. A real-time task should not make Linux system calls as that would infer unpredictability. Also, it should be made as simple as possible and leave the non real-time parts of the code to the user-space application.

Overview of RTLinux:

Real-Time Linux (RTLinux) began as a research project in 1995 at the Dept. of Computer Science at New Mexico Institute of Technology [Note 4]. The immediate goal was to develop a Linux kernel that would support real-time control of scientific instruments [Note 7]. The work by Victor Yodaiken and Michael Barabanov resulted in a small real-time executive running Linux as a completely preemptible task.

RTLinux was quickly adopted as the real-time processing software of choice in a variety of production projects. In 1998, the company called Finite State Machine Labs (or FSMLabs) was formed around RTLinux. FSMLabs now has a number of strategic partners, one of them is Red Hat, Inc., which has selected RTLinux as its standard approach to hard real-time Linux application requirements.

The RTLinux API is well documented, especially since a big part of it is POSIX. Installation instructions, getting started documentation and a FAQ are available as well as a mailing list.

FSMLabs holds a patent on the basic process for real-time in RTLinux. The patent describes the technique of using a software emulation of the interrupt control hardware to prevent the non-real-time OS from causing delays in real-time operations [Note 4]. It is a software patent only valid in the U.S. [Note 8].

Currently, RTLinux is available in different forms from FSMLabs. RTLinuxpro and RTLinux/BSD (which have the same code base) are commercial distributions shipped with a development kit. Also, the remains of the open-source project that RTLinux once was, are available in OpenRTLinux . OpenRTLinux is released under GPL and OpenRTLinux Patent License [Note 9]. Any modification of the code covered by the Open RTLinux License must be released under GPL and it must be open and available on the web [Note 8].

Summary: RTLinux is a widely used and tested solution. It has been around in some form since 1995 and is now used in many applications around the world. The technology is solid, and the use of POSIX means that an application programmer hopefully does not have to learn yet another API.

Overview of RTAI:

Development of the DIAPM (Dipartimento di Ingegneria Aerospaziale Politecnico di Milano) RTLinux variant started immediately after RTLinux was released because the people at DIAPM were not satisfied with the performance offered by the first version. They had been using a self made DOS based real-time variant earlier and thought they could take some of the techniques developed there and put them into RTLinux.

In the beginning of 1999, RTAI (Real-Time Application Interface) was released by Paolo Mantegazza. RTAI is now an open source project with an active development community. The project has been ported to a wide variety of architectures such as x86, PPC, ARM and MIPS.

The RTAI documentation is well written, although not quite up-to-date. The mailing list has a lot of traffic and questions get answered quickly.

RTAI is an open-source project. It was earlier released under LGPL 2, but the core has recently changed to GPL 2, while the rest remains LGPL 2. The parts of RTAI released under GPL are the parts that potentially may be claimed to implement the teachings of the RTLinux patent. With this move from LGPL to GPL there should be no problem with the Version 2 of the Open RTLinux Patent License [Note 9], which says: "The Patented Process may be used, without any payment of a royalty, with two types of software. The first type is software that operates under the terms of a GPL...". Eben Moglen, general counsel of the Free Software Foundation appears to support this position [Note 10].

Summary: RTAI was derived from an early version of RTLinux, but has since followed its own track and evolved into a mature and feature-rich environment which is fully devoted to open source software. It is under active development in an open community, continuously contributing to the development process. RTAI seems to be mature enough to use with most applications and a lot of work is devoted to further improve it.

Conclusions from RTLinux vs. RTAI Evaluation

Here, we summarize our conclusions based on the investigation.

In short, RTLinux and RTAI are based on the same idea and they work very much alike. The performance provided is essentially the same. There are, however, some more or less important differences outlined in the following table:

Summary of Comparision between RTLinux and RTAI

RTAI	RTLinux
RTAI uses a small kernel patch.	RTLinux uses a large kernel patch.
RTAI is an open-source initiative and is likely to continue that way.	RTLinux started as open-source, but is commercial today. Any development seems to be in the commercial versions.
RTAI is actively developed.	RTLinux development seems to have stalled in the free versions.
RTAI has its own API, but has a POSIX module that supports some POSIX calls.	RTLinux API is fully compatible with POSIX 1003.13 "minimal real-time operating system"
RTAI has many features.	RTLinux has a minimalistic approach.

Using a small kernel patch, as RTAI does, makes it easy to maintain the system between different kernel versions. As to features it is not certain that more are necessarily better; it could be convenient with many features but it also makes the system more complex and harder to understand. The real-time part of an application should be made as simple as possible and leave the rest to the non real-time part.

It is mostly an advantage to use a standardized API such as POSIX because it may be easier for newcomers to migrate to the system. RTLinux has full support for POSIX 1003.13 while RTAI provide only some POSIX compatibility. However, the RTAI API is not difficult to understand.

We ended up selecting RTAI as the extension of choice for the rest of the thesis. This was mainly because of its . . .

• commitment to open-source,
  
• active development,
  
• and small modifications of the Linux kernel.

System Performance Evaluation

During the thesis, we have ported RTAI to ETRAX and in order to evaluate the performance and verify the basic functionality of the port, a number of tests have been conducted. In this article we will only present one of these tests.

For a description of the test environment (hardware configuration, software versions, load definitions etc.) see the full report.

Interrupt Latency

Interrupt latency is the amount of time between when an interrupt is generated (internally or by an external device) and when an installed interrupt handler starts to execute. When the system is in an idle state this time is very short, but will be longer when for example other interrupts are processed. Interrupt latency is a very important measurement in a real-time system. It affects many other performance aspects such as scheduling precision and interrupt task latency. A worst case interrupt latency yields for example a lower boundary for the worst case interrupt task latency. The most interesting tests are those with load as the worst latencies show up under load.

Comparison of plain Linux and RTAI

A comparison between Linux and RTAI is made. It is based upon the two normal configurations: RTAI with an RT-handler and plain Linux with a normal interrupt handler. The values presented are in µs.

	Min	Max	Mean	Median	Std. Dev.	#Values
Linux	5.2	162.9	14.3	11.3	6.4	476052
RTAI	4.1	68.5	14.3	15.6	6.0	918598

The difference between RTAI and Linux is small without system load. RTAI has a slightly higher mean value compared to Linux, which is due to the hardware abstraction layer. It imposes a slight, predictable increase in latency. It appears not much has been gained with RTAI when there is no system load.

As load is applied, the difference becomes more obvious. While Linux still has a better average, the worst case measured latency is 2.5 times higher for Linux compared to RTAI (162.9 µs for Linux and 68.5 µs for RTAI). The two figures below show plots of measured latencies for Linux (upper graph) and RTAI (lower graph) respectively. The samples appear much more limited in RTAI, Linux has for example several samples over 70 µs while RTAI has none.


Above: Performance of Standard Linux under Load


Above: Performance of RTAI under Load

Observations and Conclusions

Porting a system such as RTAI is not easy. The documentation available at this level for both RTAI and Linux is very limited. In order to fully understand, one can not rely on written documentation, but must instead read the code itself. This is a time consuming task. The process of debugging the kernel in an embedded system is also very time consuming and the addition of RTAI does not make it any easier.

In order to make RTAI a commercially attractive solution for the ETRAX platform, some of the limitations mentioned in the full report could need further attention.

When one compares real-time performance, RTAI is much better than Linux. The improved performance is visible in all of the tests. For example, in the interrupt latency test where the plain Linux kernel had a maximum response time of 163 µs, RTAI never went over 68 µs.

The evaluation leads to three primary conclusions:

• The basic functionality of RTAI is working on ETRAX.
  
• The interrupt latency in RTAI is much more limited than in Linux. The measurement shows a 2.5 times higher worst case interrupt latency in Linux compared to RTAI.
  
• All measurements indicate a good average real-time performance in RTAI.

To sum up, it can be said that RTAI is a solution that provides good real-time performance, while preserving the entire functionality of Linux.

---

Resources:

• The authors' full masters thesis is available here (625KB PDF download)
  
• RTAI 24.1.9 with support for CRIS/ETRAX 100LX can be downloaded here
  
• A description of how to install RTAI on the Axis ETRAX 100LX developer board can be found here
  
• Further information on RTAI is here
  
• Further information the Axis ETRAX is here

---

Bibliography and Notes:

1. John A. Stankovic, et al. What is predictability for Real-Time Systems? Editorial, Dept. of Computer and Information Science, University of Massachusetts, 1993.
   
   
2. Kevin Dankwardt. Real-Time and Linux, Part 1. Embedded Linux Journal, January 2002. Available here.
   
   
3. Kevin Dankwardt. Real-Time and Linux, Part 2: the Preemptible Kernel. Embedded Linux Journal, March 2002. Available here.
   
   
4. FSMLabs, Inc. homepage. Available here.
   
   
5. The DIAPM RTAI project homepage. Available here.
   
   
6. Tim Bird. Comparing two approaches to real-time Linux.Guest column at Linuxdevices.com, Dec 2000. Available here.
   
   
7. Victor Yodaiken and Michael Barabanov. A Real-Time Linux. New Mexico Institute of Technology.
   
   
8. Ismael Ripoll. RTLinux versus RTAI. 2002. Available here.
   
   
9. The Open RTLinux Patent License, version 2.0. Available here.
   
   
10. Eben Moglen. About the RT-Linux patent and RTAI. Available here.

This article is copyright © 2003 Martin Andersson and Jens-Henrik Lindskov and is reproduced by LinuxDevices.com with permission of the authors.

---

About the authors:  

Martin Andersson received an MSc in Computer Science and Engineering from Lund Institute of Technology in 2003. After his graduation he was employed at the Department of Automatic Control in Lund, where he is currently a PhD student. He likes to solve problems and make things work, and has a special interest in Real-Time and Linux and is currently the maintainter of the CRIS port of RTAI.


Jens-Henrik Lindskov studied Computer Science and Engineering at Lund Institute of Technology, where he received his MSc degree in 2003. He is currently employed at Axis Communications, working as a Software Engineer. Lindskov has been tinkering with Linux for some years, and as been enthusiastic about computers and programming since the Commodore 64, back in the 1980s.

---

Related stories:

• The Real-time Linux Software Quick Reference Guide

Discuss Running real-time Linux on the Axis ETRAX system-on-chip   >>> Be the FIRST to comment on this article!

>>> More Linux For Devices Articles Articles          >>> More By Linux Devices