(The following article was originally written for the .Net Developer Journal where it featured in the Nov 2003 issue. This was co-written by Pooja and me and basically serves as an intoroduction to our DDL language. The original language was co-written by us and two of our friends Rakesh Nandakumar and Tony Sakariya as our final year college project. The DDL in its present .Net avatar is work by Pooja and me. The copyright perios of this article has expired, hence this post. You can find the article hosted on the magazine website here: http://www.sys-con.com/dotnet/article.cfm?id=460. I hope Pooja will be mirroring this article too)
Enter the Data Definition Language: a developer perspective
Roshan James & Pooja Malpani
(Authors of the DDL)
Greetings, In this short write-up we are going to discuss our new language for the .Net platform called the Data Definition Language (DDL). We hope that by the time you have finished reading this, you will want to log onto the internet, download the DDL and try it for yourself (and drop us mail about it).
The DDL is a unique language that aims at solving a common problem. We all would have faced situations where we needed to read data from a file of some given format and there are no real library routines for this purpose, other than the standard file handling libraries. In such cases, we would have to develop code that will calculate the addresses of various values that we want to retrieve from the file and write file handling code that will retrieve the data. While this is probably feasible for simple file types, the need to support a complex file type, or multiple file types or a changing file type can easily become a development bottleneck.
Once, we had to work with a development team that was creating an application for analyzing data collected in aircraft black boxes. Data dumps from black boxes have very complex formats. The team needed to read values of various aircraft parameters such as left-aileron-angle, right-engine-exhaust-temperature etc, which were scattered across various bits that seemed to have no respect for word boundaries or any such thing. To add to the fray there was no apparent standard between black box formats of the same manufacturer, version of the same aircraft or any such thing. The team needed to have their product support multiple black box formats without a huge development cycle between these. A little thought made it apparent that there was no real solution to supporting data retrieval from arbitrary file formats. This is the problem that the DDL was designed to solve.
How does the DDL understand the format of the file I want to read data from? The DDL is a language. It is designed to be simple and intuitive for expressing data formats. To develop a solution with the DDL a developer needs to first write a DDL script that represents the file format. Once the script is developed, you can run the DDL engine/interpreter on the script, provide the engine with your data file and you are ready to start reading information out of the file.
So how can I use the DDL in developing an application? The DDL engine is designed to be an interpreter that be hosted by any .Net application – which means that your VB.Net or C# programs can act as hosts for the DDL. A typical DDL solution would consist of your parent application that does all the UI and has all the business logic. This would host the DDL engine with which you can programmatically interact. The DDL engine simply acts as a substitute for your complex file handling code; it does not dictate what you do with the data that you have retrieved.
Hosting the DDL.
Once you have hosted the DDL engine, you tell it to do tasks such as ‘load this ddl script file’, to make it understand your file format. Then you tell it to ‘use this data file’ so that it can apply your ddl script (which defines the data format) of the actual data file. Then you tell it ‘give me the value of left-aileron-angle’ and the DDL engine looks at the script, understands where in the data file ‘left-aileron-angle’ would exist and reads out the value for your host application.
What are the DDL script files like? The DDL language provides you with primitive types that represent bits – these are available from single bit definitions to 32 bit definitions. These are named i1 i2, i3 etc to i32. Variables can be declared to represent any of these types.
i16 width
This represents a 16 bit value called width.
Now declarations like this should be grouped into DDL structures. Structures would look like this:
struct Window
{
i16 width
i16 height
}
A DDL script file can contain multiple structures like the one above. Similar to the declaration of primitive bit types you could also define a structure to contain a member of another structure type.
struct Bitmap
{
i8 BitsPerPixel
Window w
}
The DDL script requires that you mark one structure in the script file as the ‘init’ structure. The DDL uses the “init” structure as the first structure of your data file – the init structure is mapped to offset zero of your data file. The init structures and its members (which maybe instances of some of the other structures declared in the script file) are expected to represent the entire data file.
The tree-like structure formed by structures in a DDL script file.
The DDL structures are different in concept from structures in languages such as C. The first difference is that a DDL structure can contain members depending on conditions. Secondly, a member in a DDL structure simply represents a region in the data file; an i8 type member would represent a byte-sized region. Members in a structure have their offset addresses from the base address of the structure automatically calculated or can have explicit addresses provided. This little script demonstrates both of these.
struct EmployeeData
{
i32 empNumber
i1 fPhoneNumberProvided
i7 fUndefined
when( fPhoneNumberProvided == 1 )
{
i8[10] phoneNumberString
};
@ 0,0 i24 empSerialNumber
i8 empDesigCode
}
This simple script demonstrates some interesting things. An instance of the EmployeeData structure will contain a member called phoneNumberString of 10 bytes, only if the fPhoneNumberProvided flag bit is set. Similarly the notation “@ 0,0” is an explicit address specification that causes the address of the declaration immediately following it to fall at an offset of 0 bits from the start of the structure. Thus, with respect to little-endian Intel architecture, the lower 3 bytes of empNumber are the empSerialNumber and the most significant byte stands for the employee’s designation code.
Layout of the structure with respect to the bits of the data file.
The script also demonstrates a simple array of i8 (or byte) type of ten elements. Unlike C, the size of an array can be specified via an expression. To illustrate the power of arrays consider the following snippet that uses the above employee structure.
struct EmployeeHeader : init
{
i16 empCount
EmployeeData [empCount] emps
}
The first member decides the number of employees whose data is provided and ‘emps’ represents as array of EmployeeData types. You can now ask the DDL engine for the designation code of the 5th employee and DDL engine sets about to determine the location of the 5th employee and retrieve its ‘empDesigCode’ value for you. Now remember that the EmployeeData had a member that would occur conditionally – this means that the size of each EmployeeData instance can be different. The DDL engine internally determines that there is a dependency and checks the flag in each of the preceding EmployeeData instances to determine the actual location of the 5th instance that you requested.
The ddl script provides only a minimal set of programming constructs whose purpose is centered on being able to define data formats. The present ddl language is rich enough to support a wide variety of common file formats. There may however be some format types that may not be expressible in the ddl or are hard to express.
The DDL language has constructs for representing address specs, size specs, conditional dependencies, different kinds of array constructs etc. This is a just a brief description of the language, the complete language description document is available from the homepage.
Hosting the DDL Interpreter: This is probably the simplest part. Imagine that EmployeeHeader and EmployeeData together formed a ddl file called employee.ddl and that we had a data file in this format called empinfo.bin. The following C# snippet is all you will need to start off using the DDL.
using System;
using System.DDL;
class DDLTestClass
{
static void Main()
{
//initialise the DDL
ManagedDDLEngine ddl = new ManagedDDLEngine();
ddl.LoadSourceFile("employee.ddl");
ddl.OpenDataFile("empinfo.bin");
ddl.InterpretData(); //this call is needed to map the scoure to the data
Console.WriteLine("No of employees = {0}",
ddl.GetValue("empCount")); // GetValue() read the value of a variable
//contents of emps[0]
ddl.Seek(".emps:0"); //seek changes path into a member
Console.WriteLine("Data of 0th Employee \n\t "+
"empNumber={0}",
ddl.GetValue("empNumber"));
//contents of emps[5]
ddl.Seek(".emps:5");
Console.WriteLine("Data of 5th Employee \n\t "+
"empNumber={0} \n\t "+
"empDesigCode={1}, \n\t "+
"empSerialNumber={2}",
ddl.GetValue("empNumber"),
ddl.GetValue("empDesigCode"),
ddl.GetValue("empSerialNumber"));
ddl.Dispose(); //clean up
}
}
This little snippet loads the ddl with the script file and a data file and reads values from it. This snippet does not show any of the error handling code that would be required in a production quality application. One concept you need to be familiar with is that of path.
The init structure is represented as “.” (dot). Any member under it is represented using its member name. Any child of that member is separated by a dot, and so on. If there is an array in the path, then the array instance is separate by a “:” (colon).
Ex
init.emp[0] will be represented as “.emps:0”
At any point the GetValue() call will return the values on any of the variables in the current path. The Seek() is used to set the current path to another location, subsequent GetValue() calls will read values from that location. Bit values that are read are treated as unsigned integer types. All values are returned as ‘double’ type.
A document describing the API exposed by the DDL is available on the homepage for details.
What is available? The DDL engine itself is currently for download as System.DDL.dll. This is a mixed mode .net assembly developed in Managed C++ and can be hosted in any .net application. The entire source code of the DDL is also available for download. The DDL is offered for use free of cost and is currently not under any licensing or royalty restrictions. We however expect that in return we get feedback that can help improve the DDL. If the DDL is used for commercial purposes we hope that the authors drop us a note and possibly give credit where credit is due – this would help in popularizing the DDL. You are however not required to do any of these and are free to use the DDL without any acknowledgements at all.
Also a console program is available which can be used to test-run DDL scripts that you are developing. This is also available in source form as an example of hosting the DDL. The site also has tutorial material about the DDL console as well as documentation about the language, API, internal algorithms and such.
Present and Future: The DDL in its .Net avatar is presently in Beta 1 status and is the work of two people. We believe that the idea of a generally useful DDL system has substance and are hoping to work towards it.
For future development we are hoping to strengthen to the DDL language so that the language can be used to express data formats that are currently not expressible or are hard to express. Also plans are underway for a compiler for the DDL language. The compiler is expected to take a .ddl file as input and generate a .net assembly as output that will be code streamlined for your ddl script, rather than be a general purpose ddl interpreter.
We are hoping to build a community around the DDL and would like to invite you to join the DDL development project. Inputs for future design aspects, known issues etc would be appreciated.
The DDL project homepage is http://ddl.sscli.net .
Contact us: spark at sscli dot net, dolly at sscli dot net.
(Download the article and related code sample here
The homepage for the DDL project remains http://ddl.sscli.net
You may also want to visit: http://www.thinkingms.com/pensieve/homepage/work/system.ddl/index.htm
And http://www34.brinkster.com/mpooja/work/Other/ddl/index.htm
A complete description of the language is available at http://www34.brinkster.com/mpooja/work/Other/ddl/html/DDL-Language.htm)
Please give us feedback about the language and what you would like to see implemented. One of the plans we have for the future is a compiler for the DDL – this has been postponed because various other projects kept getting in the way.