Foreword -- This whitepaper by FSMLabs product manager Matt Sherer looks at "VxIT," a VxWorks emulation layer that aims to provide developers with a strategy for transitioning VxWorks applications to real-time Linux. In particular, it shows how real-time VxWorks applications running under VxIT can transparently access the Linux filesystem.

Enjoy . . . !

---

Linux filesystem I/O for hard real-time applications
by Matt Sherer
Abstract

Legacy systems such as those based on VxWorks have had access to the capabilities of a single operating system, limited to what the vendor provides. In contrast, RTCore applications have access to a small, high performance POSIX RTOS but rely on Linux or BSD for non-real-time services. With the addition of VxIT, a VxWorks migration tool, applications now have access to non-real-time Linux services like file I/O directly, in addition to VxWorks and RTCore-provided POSIX APIs. We'll demonstrate this in action for native RTCore and migrated VxWorks applications.

Some background

Some of you are familiar with RTLinux, and some are probably familiar with VxWorks. Both are hard real-time systems, deployed in many demanding environments. Both provide hard real-time performance near the limits of the hardware they're running on. RTCore's interface is based on the POSIX standard, and VxWorks' API is a collection of legacy interfaces with POSIX added on when needed, but most users that have been on the system for some time are generally bound to the legacy interfaces.

In the VxWorks case, all services are provided by the single OS, and are limited to what is provided by the vendor and any third party add-ons. This limits the functionality of the RTOS and applications considerably, generally to the scope of what the original vendor and your development team can put together and support.

In the RTCore model, the RTOS provides a minimal set of hard real-time services. Linux or BSD UNIX then provides the non-real-time services -- anything that Linux could do before it still does, without any changes. (Apache, Oracle, MySQL, Python, and so on.) Linux can provide just about any service you need, generally right out of the box, or with minimal effort.

The third piece we'll be covering is VxIT, which is an add-on to RTCore that provides access to VxWorks API calls. It allows legacy applications built on VxWorks to move over to RTCore with minimal changes. Once they are migrated, they have access to their expected VxWorks interfaces, but now have access to everything under Linux.

A quick demo

Before we demonstrate how to do non-real-time file I/O from VxIT and RTCore applications, let's fill in the blanks a bit regarding what some VxWorks applications look like under VxIT.

First, let's look at a simple semaphore application: (These examples are hosted here).

#include "vxWorks.h"
#include "taskLib.h"
#include "semLib.h"
#include "stdio.h"

void taskOne(void);
void taskTwo(void);

#define ITER 10
SEM_ID semBinary;
int global = 0;

void binary(void) {
	int taskIdOne, taskIdTwo;

	semBinary = semBCreate(SEM_Q_FIFO, SEM_FULL);

	semTake(semBinary,WAIT_FOREVER);

	taskIdOne = taskSpawn("t1",90,0x100,2000,
		(FUNCPTR)taskOne,0,0,0,0,0,0,0,0,0,0);
	taskIdTwo = taskSpawn("t2",90,0x100,2000,
		(FUNCPTR)taskTwo,0,0,0,0,0,0,0,0,0,0);
}

void taskOne(void) {
	int i;
	for (i=0; i  ITER; i++) {
		semTake(semBinary,WAIT_FOREVER);
		printf("I am taskOne and global = %d\n", ++global);
		semGive(semBinary); 
	} 
}

void taskTwo(void)
{
	int i;
	semGive(semBinary); 
	for (i=0; i  ITER; i++) {
		semTake(semBinary,WAIT_FOREVER);
		printf("I am taskTwo and global = %d\n", --global);
		semGive(semBinary); 
	} 
}

There are two threads that are created, and a semaphore. The real-time threads coordinate on a semaphore until each has taken and given it 10 times. To make this a VxIT application, we add these lines:

... [ Unchanged VxWorks Code ]
#include "pthread.h"
... [ Unchanged VxWorks Code ]
int main(void) { /* Was "void binary(void) {" */
... [ Unchanged VxWorks Code ]
pthread_join(taskIdOne, NULL);
pthread_join(taskIdTwo, NULL);
... [ Unchanged VxWorks Code ]

The entry function is replaced with POSIX main() so that it can be run as a normal UNIX application. Then one line to #include "pthread.h" so we can call pthread_join() to join with the threads after they complete. (This is important - you can see that you can access RTCore's POSIX calls n addition to the VxWorks APIs.) Let's run it:

user@host # ./sema.rtl
I am taskOne and global = 1
I am taskTwo and global = 0
...
user@host # 

How about a simpler one, with just 10 VxWorks threads, moved over to RTCore and VxIT? Here's the original code, straight from the web site:

#define ITERATIONS 10

void print(void) {
	printf("Hello, I am task %d\n",taskIdSelf()); 
}

spawn_ten() {
	int i, taskId;
	for(i=0; i  ITERATIONS; i++)  
		taskId = taskSpawn("tprint",90,0x100,2000,
			print,0,0,0,0,0,0,0,0,0,0);
}

We spawn 10 threads. Now let's move it to VxIT, with the requisite headers:

#include stdio.h
#include taskLib.h
#include unistd.h
... [ Unchanged VxWorks Code ]
int main(void) { /* Was "spawn_ten() {" */
... [ Unchanged VxWorks Code ]
       usleep(10000000);
... [ Unchanged VxWorks Code ]

Yawn, right? Yeah. There's nothing spectacular here either. That's pretty much the point. In this case we usleep() after the thread creation to wait instead of joining. For the sake of completeness, let's run it:

user@host # ./thread.rtl
Hello, I am task 1
Hello, I am task 2
...
user@host # 

I won't bore you any more here - let's move on.

Direct access to non-real-time services (file I/O)

RTLinux applications have been able to coordinate with Linux and BSD apps for years to shuttle data to and from storage. The hard real-time OS implements the subset of POSIX specified for hard real-time systems, and the rest of the (rather large) specification is left for the general purpose system to manage. Why have the real-time system care at all about shell behavior, or cron, or anything else that is a non-real-time activity? Let the capable non-real-time system handle it - Linux and BSD are really good at this kind of thing.

Let's look at a basic real-time application that needs to get data to a Linux disk. First we need to have the RT app put the data in a FIFO:

#include stdio.h
#include sys/types.h
#include sys/stat.h
#include unistd.h
#include fcntl.h
#include pthread.h

pthread_t thread;
int fd;

void *thread_code(void *arg) {
	while (1) {
		write(fd,"abcde",5);
		usleep(100);
	}
	return NULL;
}

int main(int argc, char **argv) {
	mkfifo("/tmp/fifo", 0755);
	fd = open("/tmp/fifo", O_NONBLOCK | O_WRONLY, 0);
	pthread_create(&thread, 0, thread_code, 0);
	rtl_main_wait();
	pthread_cancel(thread);
	pthread_join(thread, NULL);
	close(fd);
	unlink("/tmp/fifo");
	return 0;
}

This app has a real-time thread that generates 5 bytes to go to disk every 100 microseconds. This is sent down a real-time FIFO to be picked up by a Linux application, which then puts it to the disk. The non-real-time application can handle all the non-real-time work this requires - waiting to be scheduled (and maybe suspended when the kernel wants it to be), waiting for the kernel to get the data to disk, and so on. Here's the Linux app:

#include stdio.h
#include sys/types.h
#include sys/stat.h
#include unistd.h
#include fcntl.h

int fd, disk_fd;
char buf[4096];
int ret;

int main(int argc, char **argv) {
	disk_fd = open("/var/log/data", O_CREAT | O_WRONLY);
	fd = open("/tmp/fifo", O_NONBLOCK | O_RDONLY);
	while (1) {
		ret = read(fd,buf,4096);
		write(disk_fd,buf,ret);
		sleep(1);
	}
	close(fd);
	close(disk_fd);
	return 0;
}

Pretty simple - just open a handle to the FIFO, and a handle to a file on disk, and read from one and write to the other.

A better way

The previous method is fine, but it requires a bit of indirection between the real-time application and the Linux filesystem. Linux does all of the heavy work, but you need to have an application to do the shuttling. Also, for more complex applications, a protocol needs to be defined that both applications need to understand.

If you're writing your application under RTCore, this is not new, and generally pretty easy to do. However, it's very common that large amounts of POSIX code already exists from a legacy system, and can't be modified to suit a new real-time OS, but has to move over anyway. Or maybe a provided simulation environment has a large existing codebase that does not allow modification.

In these situations, a piece of the real-time application may need to log data to disk, read initial configuration data from disk, or any other normal file I/O activity.

For these applications, FSMLabs has extended the VxIT tool to handle these kinds of activities transparently. VxIT is a legacy transition add-on to RTCore that provides VxWorks APIs to users. With this tool, code developed for VxWorks can be recompiled and run as an RTCore application with Linux.

With this extension comes the ability to transparently access anything on a mounted Linux filesystem from a real-time application. Need to log data to a file on a USB memory stick? Just mount it in Linux and it's available. If the device is mounted on /mnt/usb_drive, the real-time application can:

open("/mnt/usb_drive/log_data", O_WRONLY);

and dump data there directly.

An example

We'll stop now and do a quick example, starting with a Linux application, and then move it over to an RTCore app with VxIT. Here's a Linux process that spawns a POSIX thread that opens and reads /etc/hosts.

#include stdio.h
#include stdlib.h
#include sys/stat.h
#include pthread.h
#include fcntl.h
#include sys/types.h
#include unistd.h

pthread_t thread;

void *thread_code(void *arg) {
	int ret, fd;
	char buf[11];
	printf("Contents of /etc/hosts:\n");
	fd = open("/etc/hosts",O_RDONLY,0);
	while ((ret = read(fd,buf,10)) != 0) {
		buf[ret] = '\0';
		printf("%s",buf);
	}
	close(fd);
	return NULL;
}

int main(int argc, char **argv) {
	pthread_create(&thread, 0, thread_code, 0);
	pthread_join(thread, NULL);
	return 0;
}

This file can be built and run with the expected output, which is usually something like (shortened for this demo):

# ./read_app
Contents of /etc/hosts:
127.0.0.1	localhost.localdomain	localhost
#

Now let's build and run this as an RTCore application with VxIT. First, under our running Linux, we load RTCore and VxIT:

# rtcore &
# vxit.rtl &
# vxit_gpos &

We're now running a hard real-time operating system along with Linux, and we have an API layer to make VxWorks applications run transparently. (Although for this application we're just sticking to the POSIX API visible through RTCore for file I/O.) Since RTCore is POSIX, and VxIT allows for Linux file I/O, we do not change a single line. It'll just work like before - here's the output:

# ./vxit_app.rtl
Contents of /etc/hosts:
127.0.0.1	localhost.localdomain	localhost
#

Of course, this could also be done with VxIT using VxWorks API calls. For example, here's a similar application that does the same thing but with the the VxWorks thread creation API:

#include taskLib.h
#include vxWorks.h
#include unistd.h
#include stdio.h

void taskOne(void) {
	int ret, fd;
	char buf[11];
	printf("Contents of /etc/hosts:\n");
	fd = open("/etc/hosts",O_RDONLY,0);
	while ((ret = read(fd,buf,10)) != 0) {
		buf[ret] = '\0';
		printf("%s",buf);
	}
	close(fd);
}

int main(int argc, char **argv) {
	int taskIdOne;
	taskIdOne = taskSpawn("t1",90,0x100,2000,
		(FUNCPTR)taskOne,0,0,0,0,0,0,0,0,0,0);
	usleep(10000000);
	return 0;
}

The only difference here is that we're running a hard real-time application that is scheduled and managed by RTCore instead of Linux. Now, since Linux is still managing the filesystem, the open(), read(), close(), and so forth of Linux files is still a non-deterministic operation, but it can be done easily within the RTCore application. Of course, these calls applied to real-time devices (like FIFOs, shared memory, device interfaces, etc.) are hard real-time calls - the non-real-time aspect only applies to files hosted by the non-real-time operating system.

This makes things like configuration file parsing, data logging, and other normal file I/O activities trivial and indistinguishable from what you would normally do in a non-real-time POSIX environment. Existing codebases can come over easily, developers can just do what they normally do, and there are no surprises.

More information

Many people think that writing real-time applications under Linux is a difficult chore, and in fact, some people still think it can't be done at all. But as you can see here, we have standard POSIX APIs available to in-kernel hard real-time applications, and it just loads and runs like a normal Linux application. Along with this, you get worst case scheduling jitter in the low microsecond range, memory protected real-time threads, hard real-time networking, XML integration, VxWorks compatibility, BSD integration, and more.

FSMLabs is doing other work along these lines to make hard real-time development even easier for developers, in VxIT and elsewhere. If you're curious, let me know, and I can fill you in on the details.

---

About the author -- Matt works at the central FSMLabs office in New Mexico. Before coming to FSMLabs in 2001, he spent years desperately trying to use Linux in applications that needed hard real-time. Now he says he spends his time developing systems "to save people from having to going through that process themselves."

---

Related Stories:

• Migration kit gives VxWorks users an exit strategy to Linux
  
• An open source software library to enhance Linux/RTAI
  
• Soccer-playing robot runs real-time embedded Linux
  
• A comparison of hard real-time Linux alternatives
  
• 25 papers on real-time Linux -- proceedings of the 6th Real-time Linux Workshop
  
• Guest Editorial: Thoughts on secure operating systems
  
• RTLinuxPro CPU reservation technology
  
• The Linux "real time interrupt patch"
  
• Coding examples illustrate dual-kernel real-time Linux programming
  
• Coding examples tout hard real-time Linux ease-of-use
  
• CEO Interview: Victor Yodaiken, FSMLabs
  

Discuss Linux filesystem I/O for hard real-time applications   >>> Be the FIRST to comment on this article!

>>> More Linux For Devices Articles Articles          >>> More By Linux Devices