REAL TIME DATA ACQUISITION SYSTEM
Empress Real Time Data Collector

Introduction

Real-time applications must be deterministic. These applications control highly sophisticated systems, such as robotic manipulators on factory floors, ground satellite stations or next generation IP-based routers. Most systems require a constant ingest of data in order to obtain sufficient information. Modern real-time applications depend on vast amount of data, hence they require some kind of a database management system (DBMS) component. However, since a database grows incessantly it lacks predictability – an important criteria for a real time application.

Furthermore, many real-time applications depend to a large degree on constant feedback collected by monitoring stations or sensors. Such data must be ingested in real time and often needs to be permanently stored for later analysis (e.g. temperature, humidity, air pressure, etc.)

The challenge therefore is to transform a variable database management system into a precise and deterministic tool to satisfy the uncompromising requirements of real time sensory data collection.

One of such tools is Empress Real-Time Data Collector (Empress RTDC).

Overview of the Empress RTDC

Empress RTDC is a collection of APIs, utilities, clients and servers for facilitating real-time data collection into an Empress database in a client-server environment

RTDC is expected to run primarily in an environment where real-time data is first collected on a set of client machines, then sent to a server machine, and then stored in an Empress database. The client machines are expected to be less powerful machines running real-time or stripped-down operating systems possibly without consoles, file systems and hard drives. The server machine is expected to be a normal disk-based system running Unix-like or Microsoft Windows 2K/XP-like operating systems.

The flow of the real-time data in the Empress RTDC is shown in the Figure 1.



Figure 1. The Flow of the Real-Time Data in the Empress RTDC

RTDC consists of the following components.

• On the client machines:
   
1. RTDC data collection applications (RTDC dc app) - these are developed by the RTDC application developer to collect the real-time data typically from operating system functions and to pass the data to RTDC buffers through the RTDC dc app api;
    
2. RTDC dc app api - a set of library functions to be called from the RTDC dc apps to pass the real-time data to the RTDC buffers; the real-time data may be augmented automatically with such data as timestamps;
    
3. RTDC buffers - sets of memory buffers to hold the real-time data from the RTDC dc app api to be retrieved through the RTDC uploader api; each RTDC dc app on a client machine uses different sets of buffers; each set of buffers act in a circular fashion;
    
4. RTDC uploader api - a set of library functions to be called by the RTDC uploader to retrieve the real-time data from the RTDC buffers;
    
5. RTDC uploader - an Empress client that runs in the background that retrieves the real-time data through the RTDC uploader api and sends the data to the RTDC server.

• On the server machine:
   
1. RTDC server - a server that receives the real-time data from the RTDC uploaders on the client machines and stores the data in the RTDC database;
    
2. RTDC database - an Empress database to store the real-time data; each set of RTDC buffers on a client machine corresponds to a single table; the table may be shared by the same set of RTDC buffers on different client machines;
    
3. RTDC database applications (RTDC db app) - these are developed by the RTDC developer to operate on the data in the RTDC database.

• On the development (server) machine:
   
1. RTDC configuration info generator - a utility that assists the developer in generating human-readable configuration info from the schema of the RTDC database; the configuration info is required by many of the RTDC components;
    
2. RTDC dc app maker - a utility that (cross) compiles and links an RTDC dc app with the appropriate libraries to form an executable or thread-object for the client machine; it may also convert configuration info files to static data and bind this data to the dc app;
    
3. RTDC uploader maker - a utility that links the RTDC uploader with the appropriate libraries to form an executable or thread-object for the client machine; it may also convert configuration info files to static data and bind this data to the RTDC uploader.

 

On the Client Machine

The RTDC dc app and the RTDC dc app api together form a real-time task. The task can be a process or a thread depending on the developer's inclination and the support of the operating system. The task must run fast enough to collect one set of real-time data and pass the data on to the RTDC buffers before the next set of data is ready. Empress has no control over the RTDC dc app portion of the task as it is developed by the real-time application developer. The RTDC dc app api portion of the task must be developed to run as fast as possible and be as small as possible.

The RTDC uploader and the RTDC uploader api together form another task. The task can be a process or a thread depending on the support of the operating system. The task should run fast enough to retrieve the real-time data from the RTDC buffers and send the data to the RTDC server so that no data is lost. The task is designed to run as fast as possible and be relatively small. However, since the task usually runs in the background with lower priority, and its speed is subject to the vagaries of client-server communications as well as the responsiveness of the RTDC server, no guarantees can be made. Since the architectures of the client machine and the server machine can be expected to be different, the RTDC client must address the issue of cross-platform data exchange.

The RTDC uploader task normally creates and initializes the RTDC buffers on startup and removes them on exit. Invocation parameters can alter this behavior.

The RTDC buffers form an intermediary between the RTDC dc app tasks and the RTDC uploader task. The buffers are implemented in shared memory or in thread global data depending on the support of the operating system. Each RTDC dc app task uses a different set of buffers. The RTDC uploader task deals with all the sets of buffers. Each set of buffers act in a circular fashion, so that if all the buffers are full of un-retrieved data, the buffer with the earliest data is the one overwritten with new data. In this way, if the RTDC dc app task passes data to the RTDC buffers faster than the RTDC uploader task can retrieve the data, some data will be lost, albeit the earlier data. The number of buffers in a given set reflects the bridging of the behavior of the two tasks trading off the desire of zero data loss versus the size of memory occupied by the buffers.

There is no locking mechanism implemented on the RTDC buffers. This eliminates the potential problem of the RTDC uploader task blocking the RTDC dc app task which must run in real-time. Two RTDC dc app tasks do not share the same set of RTDC buffers.

In addition to actual real-time data, the RTDC buffers may optionally contain timestamps and application IDs.

The client machines are expected to be less powerful machines running real-time or stripped down operating systems possibly without consoles, file systems and hard drives. Some of these operating systems do not support processes and shared memory, but only threads. Thus, they present several porting considerations for Empress development. For the RTDC components that run on the client machines, they must be able to run as processes or threads, so that effectively they must be multi-threaded. They must highly restrict the operating system functions that they depend on for maximum portability.

Without consoles, file systems and hard drives, the RTDC components cannot depend on console or file I/O functionality from the operating system. They cannot get configuration info from files. The configuration info must be bound into the object codes of the components themselves as static data. For the RTDC dc app api, any error detected can be indicated to the RTDC dc app as an error code and subsequent handling of the error will be the responsibility of the RTDC dc app.

On the Server Machine

The RTDC server is a conventional Empress server. It receives real-time data from RTDC uploaders tasks on various client machines and stores the data in an RTDC database. Since the architecture of the client machines and the server machine are expected to be different, the RTDC server must address the issue of cross-platform data exchange. As the RTDC server works on a conventional machine, it reads configuration info from files.

The RTDC database is a conventional Empress database. Each set of RTDC buffers on a client machine corresponds to a single table; the table may be shared by the same set of RTDC buffers on different client machines.

The RTDC db app is a conventional Empress application that may use any of the Empress database interfaces.

On the Development (Server) Machine

The RTDC configuration info generator is a utility that assists the RTDC developer in generating human-readable configuration info from the schema of the RTDC database. The configuration info is required by many of the RTDC components. While the developer can create and hand-edit any configuration information, a significant portion of the configuration info must correspond exactly to the RTDC database. For example, the size and structure of an RTDC buffer depends entirely on the schema of the corresponding RTDC database table. Generating this type of configuration info with a utility can eliminate many potential sources of error. This utility can also generate C header files with data structure definitions for the convenience of the developer in developing the RTDC dc apps.

The RTDC dc app maker is a utility that (cross) compiles and links an RTDC dc app with the appropriate libraries to form an executable or a thread-object for the client machine. It is effectively a wrapper around the appropriate (cross) compiler.

The RTDC uploader maker is a utility that links the RTDC uploader with the appropriate libraries to form an executable or thread-object for the client machine. It is effectively a wrapper around the appropriate (cross) compiler.

An additional task of the RTDC dc app maker and the RTDC uploader maker is to convert configuration info files to static data and bind this data to the dc app and uploader. Different pieces of configuration info are required by the different components of the dc app and uploader tasks. For example, the RTDC buffers require their size and structure and the RTDC uploader require client-server communications configuration info. All of them require the standard Empress initialization info.

Various Scenarios/Configurations Using Empress RTDC

In the most common case there will be one RTDC data collection application per client machine/device. However, Empress RTDC allows many different scenarios in using multiple number of RTDC data collection applications, RTDC uploaders or RTDC servers. Figure 2 shows only some of them.

 



Figure 2. Different Scenarios using the Empress RTDC

 

There could be multiple RTDC data collection applications per one RTDC uploader.

There could be multiple RTDC uploaders per one client machine.

One RTDC uploader can upload data only to one RTDC server.

One RTDC server can upload data from many RTDC uploaders.

Conclusion

Empress RTDC is a small footprint data collection (data ingest) tool that is deployed on real time devices.

With the addition of the RTDC product to its portfolio, Empress Software Inc. becomes the only commercial database vendor to offer deterministic database behavior in its server and client offerings.

USA: (301) 220-1919   Canada & International: (905) 513-8888 Copyright © Empress Software Inc.  All rights reserved