DEL - Data Extraction Language

W3C Note 31 October 2001

This version:: http://www.w3.org/TR/2001/NOTE-data-extraction-20011031
Latest version:: http://www.w3.org/TR/data-extraction
Previous version:
Editors:: Eero Lempinen, Republica Corp.
Harri Saarikoski, Republica Corp.

Abstract

This document outlines the Data Extraction Language. DEL is an XML format for describing data conversion processes from other data formats to XML. A DEL script specifies how to locate and extract fragments from input data and where to insert them in the resulting XML format. The DEL processor executing the DEL script can use the extracted data to either create a new XML document or modify an existing XML document by creating new elements and attributes at locations specified with XPath expressions.

Status of this document

This document was submitted to the World Wide Web Consortium on June 21 2001 (see Submission Request, W3C Staff Comment) intention that the W3C use it as a basis for furthering the work on any-to-XML transformations. For a full list of all acknowledged Submissions, please see Acknowledged Submissions to W3C.

This document is a NOTE made available by the W3C for discussion only. Publication of this Note by W3C indicates no endorsement by W3C or the W3C Team, or any W3C Members. W3C has had no editorial control over the preparation of this Note. This document is a work in progress and may be updated, replaced, or rendered obsolete by other documents at any time.

A list of current W3C technical documents can be found at the Technical Reports page.

1 INTRODUCTION
2 DEFINITION OF DATA EXTRACTION LANGUAGE
2.1 wrapper
2.2 template
2.3 repeat
2.4 map
2.5 extract
2.6 test
2.7 runtemplate
2.8 set
2.9 charsetmap (values)
2.10 stringconvert (register)
2.11 count
3 BASIC EXAMPLE OF DATA EXTRACTION LANGUAGE
3.1 input data
3.2 DEL script
3.3 output XML
4 ADVANCED EXAMPLE WITH CHARACTER SET MAPPING AND OUTPUT TEMPLATE
4.1 input data
4.2 DEL script
4.3 XML template
4.4 output XML
APPENDIX 1. DATA EXTRACTION LANGUAGE DTD

1 INTRODUCTION

This document outlines the Data Extraction Language. DEL is an XML [XML] format for describing data conversion processes from other data formats to XML. A DEL script specifies how to locate and extract fragments from input data and where to insert them in the resulting XML format.

The DEL processor executing the DEL script can use the extracted data to either create a new XML document or modify an existing XML document by creating new elements and attributes at locations specified with XPath [XPath] expressions. A DEL script along with the source data are given to the DEL processor which performs the actual data conversion according to the script. The output from the DEL processor is a well-formed XML document containing the desired parts of the source data.

Locating data fragments in the input data can be done by searching for patterns and the matching regular expressions [REGEX]. The extracted data fragments are first temporarily stored to DEL processor's registers (or stack) in order to be refined before outputting and possibly re-used as a search pattern. The data is then read from the registers or stack and placed into its proper position in the DOM [DOM] tree of the resulting XML document. In placing the data to XML, a cursor function is used to keep track of the current position. The cursor position can be modified using XPath expressions.

2 DEFINITION OF DATA EXTRACTION LANGUAGE

The following sections describe the use of Data Extraction Language elements, their attributes and attribute values.

NOTE: The attribute values of "stack" and "regX" (where X is a user-defined register name with one character at least) are parsed before use. Then the actual value is taken from memory (stack or register). Otherwise, the actual value is the given value of the attribute.

2.1 wrapper

Function: wrapper element is the container (root element) for DEL script rules.

NOTE: When creating a new output XML, the first map element should include maptype attribute with value "createDocument".

Contains: First template and map elements and then optionally one of the following elements: repeat, map, extract, test, set or runtemplate.

Attributes: No attributes.

Syntax:

</wrapper>

2.2 template

Function: template element is used as container for common sequences of other elements. runtemplate element (for loading the template, see 2.7) is then used to call the content of template element.

Contains: Optionally one of the following elements:

repeat, map, extract, test, set or runtemplate.

Attributes: name (REQUIRED)

Syntax:

</template>

Example:

</template>

2.3 repeat

Function: repeat element repeats its content, ie. the elements located under it.

Contains: Optionally one of the following elements:

repeat, map, extract, test, set or runtemplate.

Attributes: times (REQUIRED)

Times attribute:

When times attribute is "*", it repeats the content until one of the extract elements sets "dataStreamError" status.

When times attribute is a number (e.g. "8"), it repeats the content as many times. If "dataStreamError" status is set, repeating is terminated.

NOTE: In repeat loop, when trying to extract data but not finding the given (regular) expression where expected (at the beginning of the data under processing), the DEL processor sets "dataStreamError" status to stop the repeat loop. In case "dataStreamError" status is set outside repeat loop, the whole wrapping process stops.

Syntax:

</repeat>

</repeat>

2.4 map

Function: map element inserts content to the output as XML node(s).
It moves the cursor in the output XML to specify the insertion point for the node(s). It also keeps track of the current element.

Contains: No elements

Attributes: The attributes and their possible values are:

maptype (REQUIRED): Tells the DEL processor how and where in the output XML document (specified by node attribute) to insert the content (specified in content attribute).

"createDocument": Creates a new output XML document and specifies its root element. The root element gets its name from the value of node attribute (XPath expression). After this command, the cursor moves to this root element.

"createElement": Creates a new element to the output XML. The new element is the last child element for the element at the current cursor position. The name of the element is the value of node attribute. When content attribute is specified, a new text node is also created inside the new element containing the parsed value of content attribute. After this command, the cursor moves to the new element.

"createElementBefore": Works as "createElement" but creates the new element in front of the element where the cursor is located (at the same level in result XML).

"createElementAfter": Works as "createElement" but creates a new element after the element where the cursor is located (at the same level in result XML).

"createAttribute": Creates an attribute to the output XML. The attribute is created to the element where the cursor is located. The name of the new attribute is the value of node attribute. The content of the attribute is the parsed value of the content attribute. If there is already an attribute with the same name, its content is overridden.

"createTextNode": Appends a text node to the output XML. The text node is added to the last node of the element where the cursor is located. The content of the text node is the parsed value of the content attribute.

"createComment": Appends a comment to the output XML. The comment is added before the element where the cursor is located. The content of the comment is the parsed value of the content attribute.

"createCDATA": Appends a CDATA section to the output XML. The section is added to the element where the cursor is located. The content of the CDATA section is the parsed value of the content attribute.

"createProcessingInstruction": Appends a processing instruction to the output XML. The processing instruction is added before the element where the cursor is located. The content of the processing instruction is the parsed value of the content attribute.

"moveCursor": Moves the cursor to an element which has already been created in the output XML. The place of the cursor is specified in the node attribute as an XPath expression. If the XPath expression points to more than one element, the last one is selected.

"documentReady": Cuts the source data into chunks to enable more effective processing. Produces a DOM tree which is saved in the memory.

"getTemplate": Loads a result XML template as a framework for result XML. You can use the template's XML elements instead of creating elements one by one. You can even use it to fix the values of some result XML elements, if necessary. Value attribute sets the template's root element (e.g "root").

node (REQUIRED, will be parsed): Specifies the cursor position in the result XML as XPath expression.

content (OPTIONAL, will be parsed): Gives the content of e.g. attributes in the result XML.

value (REQUIRED with "getTemplate"): Result XML document's root element.

Syntax:

Example:

2.5 extract

Function: extract element gets data from the source data.

Contains: Optionally one of the following elements:

repeat, map, extract, test, set or runtemplate.

Attributes: exptype (REQUIRED), expression (REQUIRED, will be parsed), save (OPTIONAL, will be parsed)

Exptype and expression attributes: exptype attribute tells the DEL processor which part of the data should be extracted and where it is save.

expression attribute specifies exptype.

Possible exptype attribute values are:

"length": Extracts as many characters from the beginning as the expression attribute shows.

"upto": Extracts data until ("up to") the content of expression attribute.

"over": Extracts the section of the source data that contains the content of expression attribute.

"re_upto": Extracts data until ("up to") the source document contains a section matching the regular expression given in expression attribute.

"re_over": Extracts the section of the source data that contains the regular expression given in expression attribute.

"content": Extracts the section of the source data that contains the regular expression given in expression attribute. As "re_over" but the extractable data must start at current cursor position.

"mark" (expression): Marks the current XML position and stores it into the register specified by expression attribute (cf. "set").

"set" (expression): Returns to the point previously stored to a register specified in expression attribute (cf. "mark").

Save attribute: save attribute tells where the extractable data is saved.

Possible save values are:

"stack": Data will be saved on top of the stack.

"regX": Data will be saved in register X (e.g "regRegister1").

Syntax:

Example

Tip!

In case of large source data, try a multi-level extraction script. In such a script, extract element contains other DEL elements (which in turn can contain more). This allows you to 'chop up' the data into more manageable chunks.

Consider an XML source file with <table> as root element containing <tr> elements (table rows) and those containing <td> (cells):

<table>

<tr>

</tr>

<tr>

</tr>

<tr>

</tr>

</table>

Now look at the script below. The first extract (top level) element moves the cursor to a larger section from the source data (section within "<tr>" tags). Then the second extract element puts the cursor within the "<td>" tags located in that larger section. And the third extract element extracts the content within "<td>" tags and saves it to a register from where it will be later placed to result XML:

</extract>

Such a multi-level script can also be reused and edited more effectively than one where all extract elements are on the same level.

2.6 test

Function: test element compares two values. The content of the test element will be processed when the test result is "true".

Contains: Optionally one of the following elements:

repeat, map, extract, test, set or runtemplate.

Attributes: testtype (REQUIRED), value1 (REQUIRED, parsed), value2 (OPTIONAL, parsed)

Testtype, value1 and value2: testtype attribute indicates what kind of test is processed.

Possible testtype values are:

"equal": Tests whether value1 attribute (any data, e.g "12") is equal to value2 attribute (e.g "reg1"). If test is "true", the script inside test element will be run.

"unequal": Tests whether value1 attribute (any data) is not equal to value2 attribute (any data). If test is "true", the script inside test element will be run.

"lesser": Tests whether value1 attribute (number) is a smaller number than value2 attribute (e.g "reg1"). If test is "true", the script inside test element will be run.

"greater": Tests whether value1 attribute (number) is greater than value2 attribute (e.g "reg1"). If test is "true", the script inside test element will be run.

"re_equal": Tests whether value1 attribute (regular expression) is equal to value2 attribute (e.g number 23 in "reg1"). If test is "true", the script inside test element will be run.

"re_unequal": Tests whether value1 attribute (regular expression) is not equal to value2 attribute (e.g number 23 in "reg1"). If test is "true", the script inside test element will be run.

"contains": Tests whether value1 attribute (regular expression) appears in a register or a value specified in value2 attribute (e.g "11"). If test is "true", the script inside test element will be run.

NOTE: Content of the attributes value1 and value2 are parsed before comparing.

Syntax:

Example

2.7 runtemplate

Function: runtemplate element runs elements from a predefined template (see defining the template, chapter 2.2).

Contains: No elements.

Attributes: nameref (REQUIRED)

nameref attribute:

nameref attribute points to the template to run.

Syntax:

2.8 set

Function: set element gives instructions to the DEL parser for processing the data.

Contains: No elements.

Attributes:

parameter (REQUIRED)

value1 (REQUIRED)

value2 (OPTIONAL)

value3 (OPTIONAL with "append")

parameter attribute values (value1 and value2):

"doctype": Sets the content of value1 attribute to SYSTEM identifier and content of value2 attribute to PUBLIC identifier of the result XML document.

"encoding": Sets the content of value1 attribute to the encoding type of result XML document.

"memory": Sets the content of value2 attribute to memory which points to value1 attribute.

"length": Sets the length of content of value2 attribute to memory which points to value1 attribute.

"omitXMLDeclaration": Omits the XML declaration if the content of value1 attribute is "true".

"serializeOutput": Serializes all the DOM trees in the memory if the content of value1 attribute is "true". Default value is "false".

"append": Appends strings and register contents.

if only two attributes are specified (value1 and value2): Combines the values in value1 and value2 and stores the combination in the register specified in value1 (e.g. "reg1").

if three attributes are specified (value1, value2 and value3): Combines the values in value1 and value2 and stores the combination in the register specified in value3 (e.g. "regCombo").

NOTE: The place for storing the combination ("append") must be a register (e.g. "regCombo"). Other values can either be registers or strings.

Syntax:

Example:

2.9 charsetmap (values)

Function: Creates a character set map where you can define which characters to replace with which characters.

For running the character set map, stringconvert needs to be defined (see below).

Attributes: name

Contains: values

search: Contains the string to replace.

replace: Contains the replacing string.

Syntax:

</charsetmap>

Example:

</charsetmap>

2.10 stringconvert (register)

Function: Loads and runs a character set map defined by charsetmap (see above).

Attributes:

use: Specifies the name of the character set map to be run.

overlapping: Specifies whether to convert the content of all registers, ie. also the previously converted ones ("true") or just those parts of the data so far unconverted ("false").

Contains: register

nameref attribute: Specifies the registers to process.

Syntax:

</stringconvert>

Example

</stringconvert>

2.11 count

Function: A simple arithmetic calculator.

Contains: No elements.

Attributes: parameter (REQUIRED), value1 (REQUIRED), value2, value3

parameter attribute values:

"plus": Calculates the sum of two numbers (or contents of registers) in value1 and value2 and stores it in the register specified in value3.

"minus": Subtracts (the content of) value2 from (the content of) value1 and stores it in value3.

"divide": Divides (the content of) value1 by (the content of) value2 and stores it in value3.

"multiply": Multiplies the two numbers (or contents of registers) in value1 and value2 and stores it in the register specified in value3.

"decimal": Sets the number of decimals (in value1) for the above arithmetic functions. The default is "0".

Syntax:

Example:

3 BASIC EXAMPLE OF DATA EXTRACTION LANGUAGE

Below are example rules (3.2) that extract data from normal HTML pages (3.1), producing a result XML file (3.3).

3.1 input data

Here is an example input data HTML file that will be processed below:

<html>

<body>

<p>Test Material</p>

<table>

<tr><td>Some numbers<td>Other numbers</tr>

<tr><td>1<td>2

</table>

</body>

</html>

3.2 DEL script

This example DEL script contains the following rules:

The root element is wrapper:

The below map elements create a root element called "root" and an element called "description" for its child element. The third map element adds the text "Extracted data" to that element.

The following extract element extracts data over "<table>" from the input data. Note that XML notation writes <table> as "<table>".

The following extract element tries to find "</table>" from the input data and send the extracted data to its inner DEL elements.

The following repeat element repeats its inner DEL elements as many times as possible.

The following extract elements extract data from the input data. First over "<tr>", then over "<td>" and finally, up to "<". The third element also saves the extracted data into the stack.

Below the first map element moves to the root element (/root) and the second one creates a "row" element there.

The following map element creates "field1" element and moves the content of the stack there.

The following map element moves the current cursor position one level up.

The first extract element extracts data from the input data. First over "<td>", then up to "<". The second element then saves the extracted data to register 1.

The following map element creates "field2" element and moves the content of the register 1 there.

The following test element tests the equality of the value "4" and the values in register 1. If the values are equal, the map element will create a new attribute called test with the value "success".

</test>

</repeat>

</extract>

</wrapper>

3.3 output XML

The output XML from the above input data (3.1) and rules (3.2) is as follows:

<?xml version="1.0" encoding="ISO-8859-1"?>

<root>

<description>Extracted data</description>

<row>

<field1>Some numbers</field1>

<field2>Other numbers</field2>

</row>

<row>

</row>

<row>

</row>

</root>

4 ADVANCED EXAMPLE WITH CHARACTER SET MAPPING AND OUTPUT TEMPLATE

Below is another example where DEL script modifies an output XML template is modified with source data. Character set mapping feature is used to search and replace characters.

4.1 input data

Input data can be complex and hard to extract, as follows:

#0,12.10

#23,1.514

#1,4.444

#abba

#amount

#contact

#end

4.2 DEL script

This example script makes a character set conversion using charsetmap and stringconvert elements (first with "false" overlapping and then with "true" overlapping). It produces two result XML files using "getTemplate". It also uses count to add register values and "append" to combine register values.

First the default root element is set:

The below charsetmap element creates following character set map:

',' (comma) à '.' (dot)

'.' (dot) à '_'

'a' à 'b'

'b' à 'c'

Note that no data conversion is made yet. The actual conversion is made using stringconvert command.

</charsetmap>

"getTemplate" loads the XML template to be used as framework for result XML (in this example, it is an empty XML document):

The index of the current "extract_cursor_position" is stored to register regBegin:

Value "0" is stored to register regC:

"regC + 1" is calculated and the result is stored to regCC:

value_is combined with regCC and the result is stored to regRes:

</repeat>

</extract>

The current "extract_cursor_position" is moved to the saved index (in regBegin):

regValue register's content is converted using charsetmap map1:

</stringconvert>

</repeat>

</extract>

</stringconvert>

</repeat>

</extract>

</stringconvert>

</repeat>

</extract>

</repeat>

</wrapper>

4.3 XML template

The DEL script (in 4.2) uses the following kind of XML template (using "getTemplate" to call this template):

</charsetmap>

<count>

</count>

</append>

</document>

4.4 output XML

The output XML from the above input data and rules is as follows:

<?xml version="1.0" encoding="ISO-8859-1" ?>

<value_1>0,12.10</value_1>

<value_1>23,1.514</value_1>

<value_1>1,4.444</value_1>

<value_1>abba</value_1>

</original>

<value_1>0.12-10</value_1>

<value_1>23-1.514</value_1>

<value_1>1.4-444</value_1>

<value_1>bbcc</value_1>

</overlappingfalse>

<value_1>0-12-10</value_1>

<value_1>23-1-514</value_1>

<value_1>1-4-444</value_1>

<value_1>cccc</value_1>

</overlappingtrue>

</charsetmap>

<count>

<value_1>3</value_1>

<value_1>5</value_1>

</original>

</count>

</append>

<value1>amount</value1>

<value1>contact</value1>

</original>

<result>amountcontact</result>

</append>

</document>

APPENDIX 1. DATA EXTRACTION LANGUAGE DTD

<!DOCTYPE wrapper [

<!ATTLIST template

name ID #REQUIRED >

<!ATTLIST repeat

times CDATA #REQUIRED >

<!ATTLIST test

value1 CDATA #REQUIRED

value2 CDATA #IMPLIED >

<!ELEMENT map EMPTY >

<!ATTLIST map

node CDATA #REQUIRED

content CDATA #IMPLIED >

<!ATTLIST extract

expression CDATA #REQUIRED

save CDATA #IMPLIED >

<!ELEMENT set EMPTY >

<!ATTLIST set

parameter (doctype | encoding | memory | append | serializeOutput) #REQUIRED

value1 CDATA #REQUIRED

value2 CDATA #IMPLIED

value3 CDATA #IMPLIED >

<!ELEMENT runtemplate EMPTY >

<!ATTLIST runtemplate

nameref IDREF #REQUIRED >

<!ELEMENT charsetmap (values)*>

<!ATTLIST charsetmap

name CDATA #REQUIRED >

<!ELEMENT values EMPTY >

<!ATTLIST values

search CDATA #REQUIRED

replace CDATA #REQUIRED >

<!ELEMENT stringconvert (register)*>

<!ATTLIST stringconvert

use CDATA #REQUIRED

overlapping (true | false) #REQUIRED >

<!ELEMENT register EMPTY>

<!ATTLIST register

nameref CDATA #REQUIRED >

<!ELEMENT count EMPTY

<!ATTLIST count

parameter (plus | minus | multiply | divide | decimal) #REQUIRED

value1 CDATA #REQUIRED

value2 CDATA #REQUIRED

value3 CDATA #IMPLIED >

References

XML: World Wide Web Consortium. Extensible Markup Language (XML) 1.0. February 1998. <http://www.w3.org/TR/REC-xml>
XPath: World Wide Web Consortium. XML Path Language (XPath) 1.0. November 1999. <http://www.w3.org/TR/xpath>
DOM: World Wide Web Consortium. Document Object Model (DOM) Level 2 Core Specification 1.0. November 2000. <http://www.w3.org/TR/DOM-Level-2-Core>
REGEX: GNU Regular Expressions, Regex Edition 0.12a. September 1992. <http://sunsite.utk.edu/gnu/regex/regex_toc.html>