INDEX

1) Introduction

The ERT Toolkit from Empress is an optional product for EMPRESS RDBMS developers.  The ERT Toolkit is a "C" Language API that implements a number of abstract data objects in the Empress RDBMS such as Array, Circular Buffer, Queue, Stack, Table and Variable, as well as, several special attributes as Record Number, Timestamp, Key and Serial Number.

The ERT Toolkit is distributed in source form and has been compiled on QNX and Linux.  The ERT Toolkit uses the Empress "Extreme C Interface" (The "MR" C API) to implement the database functionality.

The ERT Toolkit is both useful in providing higher level data abstraction for the embedded real-time developer as well as real examples of the power and flexibility of Empress "Extreme C Interface".  The ERT Toolkit was originally written by Empress Software real-time developers in Germany.  As a result, the documentation has a distinct German technical flavor and orientation.  If you can read "Tom's Hardware", you should have no problem here.

Empress welcomes any feedback and source code modifications for possible inclusion in future releases of the ERT Toolkit.

Enjoy.

2) Design Goals

The ERT Toolkit adds extensive functionality to the Empress database system for use in embedded and real-time systems. The design and implementation of the ERT Toolkit is optimized for the special demands of technical applications such as:

• Fast performance
• Determinism
• The maximum response time of important operations can be calculated
• No performance issues in 7/24 activity through expansion and growth of files
• The caching strategy is optimized for determinism
• Reliability, fail-safe operation
• Robustness against failure of other system components (hardware, software, network)
• Flexibility

3) Application Areas

Possible application areas of the ERT Toolkit are:

• Collect, administer and analyze process control data
• Administer and analyze historical data
• Record and analyze failure data
• Administer configuration data
• Uncouple and synchronize processes
• Prototype and test components and whole systems
• over systems against hardware, software and network failures
• Integrate with other systems through standard interfaces

The ERT Toolkit  allows the developer to create objects or containers to store structured data. The objects, containers or abstract data types (ADT) can be manipulated using the "C" function in the ERT Toolkit. The objects and their data are stored inside Empress database tables. The Empress database tables can be stored in Shared Memory (temporary data) or on hard disk (persistent data). Applications interact with the objects through the ERT Toolkit special libraries and utilities. The Empress database tables can also be accessed through standard Empress interfaces such as ODBC, JDBC, HTML, Perl, Tcl/Tk, Embedded SQL, etc.

4) The Abstract Data Types

The ERT Toolkit supports the following abstract data types (ADT):

• Array
• Circular Buffer
• Queue (FIFO)
• Stack (LIFO)
• Table
• Variable

5) What Can You Store in an ADT?

Each ADT or container can store one or more data elements which can have a simple or complex structure. Basic types of data elements are all data types of the EMPRESS database and all scalar types of the C programming language. Complex structures are made up of combinations of basic types with constant or variable size and can be stored in the EMPRESS data type BULK.  This allows storage of any structured C structures/arrays or binary files.

Basic EMPRESS types:

• Integer: shortinteger, integer, longinteger
• Fixed point: decimal, dollar => size required
• Floating point: real, float, longfloat
• Date and time: date, time, microtimestamp
• Strings with constant size: char, nlschar => size required
• Strings with variable size: text, nlstext
• Binary data with constant size: bulk => size required
• Binary data with variable size: bulk

Basic C-Types:

• char, unsigned char
• short, unsigned char
• int, unsigned int
• long, unsigned long
• float
• double
• char * => size required

Complex Types:

• struct value { char a[10], longinteger b, float c }

6) Storage of Data Elements

Every data element will be stored in a record of an EMPRESS table. Therefore you can choose between two different storage methods:

1. Every value will be stored in a separate database attribute with the matching basic type. In this case, a record has a normal attribute for every value which can be selected through standard interfaces of the database.  This would allow creation of reports with MS Access, Crystal Reports etc., through ODBC.
2. All values will be stored binary in a single bulk attribute.  In this case, it is possible to read the database attribute with standard interfaces, but you can only work with the data if you know the internal structure of the bulk value. This method has the following advantage:
• Better performance, no need to convert individual values  
• Possible smaller footprint

7) General Operation of the API on ADT's:

• Create => create a new object
• Delete => delete an existing object
• Attach => get a handle for an object, Initialization (open)
• Connect => get a handle for an object ( client )
• Detach => close an object
• Disconnect => disconnect an object ( client )
• Purge => mark all entries as deleted
• Entries => count of valid entries
• MaxEntries => Maximum count of entries
• ReadOkay => TRUE, if entries > 0
• WriteOkay => TRUE, if entries < max_entries  

8) Array

An Array has space to store a defined number (n) of data elements. Each array can be indexed directly through a longinteger index value between 0 and n-1.

Possible Operations for Array:

• WriteArray => overwrite the value on a position
• ReadArray => read the value on a position and mark as 'invalid'
• Del => set the value on a position to NULL
• DelKey => set the value with a given key to NULL
• Peek => read the value on a position

9) Circular Buffer

A circular buffer is a buffer with space for (n) data elements. If a circular buffer has no data, the first value will be written to the first position, the second value to the second position and so on. If (n) values are written and the circular buffer is full, the next insert will overwrite the oldest value. This means that only the last (n) values can be read. You can read a circular buffer sequentially or through sequence numbers. For this, a process can hold multiple read positions or cursors. Entries of a circular buffer can not be deleted individually. You can only delete the whole circular buffer.

Possible operations for Circular Buffer:

• Write => write the value to the circular buffer
• Read => read the oldest value and mark it as 'read'
• First => get the read position for the oldest value >
• Last => get the read position for the newest value
• Next => move the read position to the right (newer value)
• Prev => move the read position to the left (older value)
• PeekLast => read the last value
• Peek => read the value on the actual position

10) Queue

A Queue implements a FIFO (First-In-First-Out) buffer. The Queue has space for (n) data elements. The write operation inserts a value to the end of the queue. If the maximum count of entries is reached than no more write operations are possible until values are read from the queue. A normal read operation removes the first (oldest) element of the Queue and marks this element as 'read' (deleted). If the queue is empty no read operations are possible. The developer implements how the operation should react:

• Operation returns an error
• The reading process waits for a new value (without time-out)
• The reading process waits at least x seconds for a new value

All read operations for circular buffers are also available for a queue (First, Last, Next, Prev, Peek, PeekLast). It is possible to read a value which is marked as 'read' until it is overwritten.

It is possible to have multiple write and multiple read processes. A queue can be used for synchronization of multiple processes which create (write) or use (read) data.

Additional operations for Queue:

• Fill(n) => the Queue gets a place for (n) data elements (old values are safe)
• Write => write a value on the end of the Queue
• Read => read the oldest value from the Queue and mark it as 'read'

11) Stack

A Stack implements a LIFO (Last-In-First-Out) buffer. A Stack can store (n) data elements. A write operation puts a new value to the top of the stack. If the maximum count of entries is reached no more write operations are possible. A read operation reads the value at the top of the stack and marks this element as 'used' (deleted). If a stack does not hold data then no read operations are possible.

All read operations for circular buffers are also available for a queue (First, Last, Next, Prev, Peek, PeekLast). It is possible to read a value which is marked as 'read' until it is overwritten.

A Stack can be used for many things such as storing the results of an arithmetic routine to implement an RPN (Reverse Polish Notation) engine.

Additional operations for Stack:

• Fill(n) => the Stack gets a place for (n) data elements ( old values are safe )
• Write => writes a value on the top of the Stack
• Read => reads the value from the top of the Stack and marks it as 'used'

12) Table

A Table is like a sorted list. It can have as many data elements as attributes in a table of an EMPRESS database. It is really a normal EMPRESS database table and so every Empress table can be manipulated with the routines of the ERT Toolkit.

In comparison to other ADTs, operations on a table are not deterministic. e.g. the time for an insert of a value in a Table can vary depending on the actual state of the database engine, the table and the file system.

13) Variable

The abstract data type Variable stores a single data element. This element can have any structure and can include multiple values with different types.

Possible Operation for Variable:

• WriteVariable => overwrite the current value
• PeekLast => read the current value
• ReadVariable => read the current value and mark the value as 'invalid'

The current value will be buffered in memory. So a read operation (PeekLast) does not need the database engine nor the file system.

If a Variable was created as temporary, that value is only stored in memory and will not be written to the disk. In this case the Variable uses Shared Memory and different processes are able to manipulate the data in memory. A Variable can be used to store the value of a process control sensor.

14) How to Deal with Data and the ADT's

The parameters of the data elements from the read/write operations require a special C structure. The C structure will be allocated and initialized with given functions. The developer can choose between two possible ways to put his data into this structure:

1. The developer gives a memory address to the structure. At this address the developer writes the data which should be stored in an ADT. In this case the developer must take care that the size of the ADT and the actual data fits together. For example, if the ADT was created to store a char(20) the address should not point to a longinteger.

2. The memory for the user data of the C structure will be allocated from the maximum size of the data element of the ADT. In this case the developer can ask for the address of this memory and can now put the data at this memory address. The developer must take care that the data fits into the allocated memory.

15) Special Attributes in General

The ERT Toolkit supports the following special attributes:

• Record Number
• Timestamp
• Key
• Serial Number

Every ADT can be administered automatically for special attributes. For this, some parameters must be set at creation time of the ADTs. An exception is the ADT Table. A Table can not be created by a function call to the ERT Toolkit. Therefore, the developer has to create matching attributes (to the special attributes wanted) in the original Empress table which will be manipulated by the ERT Toolkit. Parameter names of the attributes are shown in parenthesis.

16) Record Number

Every record receives a record number attribute. This attribute stores the absolute position of a record in the EMPRESS database table and will be used internally for the administration of a list of free records. The developer can not manipulate this attribute and it is always available.

17) Timestamp (timestamp)

The Timestamp attribute stores the last instance time for every data element. Normally it is the time of the last change of the data element (day with time to microseconds). It will be set automatically when a write operation is performed. Alternatively, developers can store their own time stamps such as the time stamp at which an event occurred. The type of the Timestamp can be one of Date, Time or Microtimestamp. When reading a data element, the Timestamp is read as well. An ADT Variable always contains a Timestamp attribute.

The Timestamp attribute can be used to get the exact chronological order in which events occurred.

Warning: An automatically refreshed Timestamp can hold duplicate values if the system time was set back on a running system. This happens if the system time in a network must be synchronized or set from summer to winter time.

18) Key (key)

The Key attribute may have type longinteger or char. The developer can ensure that for a write/read operation the key value is write/read as well. An ADT of type Variable does not have a key attribute. A Key attribute can be used to store identification numbers from an external system together with corresponding data elements.

19) Serial Number (seq_id)

A Serial Number attribute stores a monotonically increasing integer sequence for every new data element. This number identifies the data element uniquely. The last value of the serial number used will exist even if the ADT does not hold any valid data. The serial number will only be reset if the ADT is deleted and recreated.

In an EMPRESS table, the Serial Number can be stored as longinteger or DECIMAL. If you create an ADT you have to specify how many decimal places the serial number should have. The Serial Number attribute can only be read. It is used as a 'cursor' to select a specific entry with the Peek function.

Through "C" libraries you can do arithmetic with serial numbers:

• +, -, *, /, %
• inc, dec
• whole Math-Library (where it makes sense for 'integer' numbers, e.g. abs, pow, sqrt...)

In addition, you can compare Serial Numbers:

• seq1 < seq2 => seq1 was inserted before seq2
• seq2 - seq1 = 1 => seq2 was inserted next to seq1
• seq2 - seq1 = 10 => 9 other elements were inserted between seq1 and seq2

20)  Example 1 - Using Circular Buffers for Fast Data Collection

The source code includes an example program that shows how you can write into one circular buffer and than read these records and store the data into a second circular buffer. Both circular buffer has a default size of 10,000 records.

First a new database 'db' is created and initialized to work with the ERT Toolkit. The MemMaster program starts up and creates two circular buffers. After MemMaster has filled up one of the Circular Buffers it exits normally. A Shared-Memory segment for the database is created. Tabzero, needed files and tables are mapped into the Shared-Memory segment. MemMaster runs in the background to hold the shared-memory segment for the ERT Toolkit. First Read and than Write for 10,000 operations are performed. Once both Read and Write finish, they print out the time they need.

Write Process

Assume that data needs to be writen every 10 msecs into the Circular Buffer. Set a delay of 10 msecs in the write loop. When the Write process finishes it calculates the time needed without the delay time. Also the time before and after the write operation are shown. Although, the Write process can be scheduled between operations, showing the before and after time may be instructive.

Read Process

The Read process reads data from a Circular Buffer and stores this data into a second Circular Buffer. If there is data available in the first Circular Buffer the Read process will read the data without a delay. If there is no data in the Circular Buffer the Read process performs a delay of 10 msecs and then looks for new data.

PeekLast Process

The PeekLast process reads the last data from a Circular Buffer and stores this data into a second Circular Buffer only if the data is different. After every read, the PeekLast process performs a delay of 10 msecs and then looks for new data. The delay times for PeekLast, Read and Write processes can be set as options.

21)  Example 2 - Using Circular Buffers for a System Log Daemon

The 'emplogd' daemon stores syslog messages in an EMPRESS database using a Circular Buffer and makes the messages available through database queries.

The design goals of 'emplogd' are to use fixed size disk space and to provide the convenience and power of a utility, 'emploginfo.sh' to query the syslog messages in the Empress database. When 'emplogd' is started the first time, a Circular Buffer is created with the number of syslog messages to be stored (default 1000). The 'emploginfo.sh' script uses pattern matching to read the data in the Circular Buffer.

Once 'emplogd' is started, it receives syslog messages and stores them in its Circular Buffer on an ongoing basis.

emploginfo.sh:

This shell script provides a convenient way to read information with pattern matching from the database used by 'emplogd'. A call of

        emploginfo.sh <database> <table><option list>

without the <option list> will show you all log messages.  To get a subset of the messages, qualify them with pattern matching by using the following options:

• -a                time after
• -b                time before
• -k                key match pattern (AND)
• -K               key match pattern (OR)
• -m               message match pattern (AND)
• -M              message match pattern (OR)

For example, the call                

emploginfo.sh db emplog -a "today - 10 days" -b "today - 20 days" -m error failed

will select the messages where the timestamp is between (today - 20 days) and (today - 10 days) and the syslog message matches or contains the word "error" and the word "failed".