XSLT 2.0 and XQuery 1.0 Serialization

1 Introduction

This document defines serialization of the W3C XQuery 1.0 and XPath 2.0 Data Model, which is the data model of at least [XPath 2.0], [XSLT 2.0], and [XQuery 1.0], and any other specifications that reference it.

Serialization is the process of converting an instance of the [Data Model] into a sequence of octets. Serialization is well-defined for most data model instances.

Ed. Note:

The document assumes the reader already knows generally what serialization is. A brief explanation will be added, especially to disabuse any reader who thinks it might mean Java (or .NET) serialization.

Ed. Note:

The editor has yet to align the description of serialization errors with the description of errors in related specifications. That will be done in a future public working draft.

In this specification the words must, must not, should, should not, may, required, and recommended are to be interpreted as described in [RFC2119].

2 Serializing Arbitrary Instances of the Data Model

An instance of the data model that is input to the serialization process is a sequence. Prior to serializing a sequence using any of the output methods whose behavior is specified by this document (3 Serialization Parameters) the serialization process must first place that input sequence into a normalized form for serialization; it is the normalized sequence that is actually serialized. The normalized form for serialization is constructed by applying all of the following rules in order, with the initial sequence being input to the first step, and the sequence that results from any step being used as input to the subsequent step. For any implementation-defined output method, it is implementation-defined whether this normalization process takes place.

Where the process of converting the input sequence to a normalized form indicates that a value must be cast to xs:string, that operation is as defined in Section 17.7 Casting to xs:string and xdt:untypedAtomic^FO of [Functions and Operators].

1. Replace an empty sequence with a zero-length string.

2. If the instance of the data model contains any atomic values, convert the atomic values to strings: obtain the lexical representation of each value by casting it to an xs:string and replace the value with its string representation. It is a serialization error if the value cannot be cast to xs:string.

3. Replace all adjacent strings in the sequence, with a single string equal to the values of the strings concatenated, each separated by a single space.

4. Replace any string in the sequence with a text node whose string value is equal to the string.

5. Replace any document node in the sequence with its children.

6. It is a serialization error if an item in the sequence is an attribute node or a namespace node. Otherwise, create a new document node and make all the items in the sequence, which are all nodes, children of that document node.

The tree rooted in the document node that is created by the final step of this normalization process is the instance of the data model to which the rules of the appropriate output method are applied. If the normalization process results in a serialization error, the processor must signal the error.

Note:

The normalization process for a sequence $seq is equivalent to constructing a document node using the XSLT instruction: <xsl:result-document> <xsl:copy-of select="$seq"/> </xsl:result-document> or the XQuery expression: document { for $s in $seq return if ($s instance of document-node()) then $s/child::node() else $s } This process results in a serialization error with certain sequences, for example sequences containing parentless attribute and namespace nodes, or atomic values of types that cannot be cast to a string, such as xs:QName.

3 Serialization Parameters

There are a number of parameters that influence how serialization is performed. Host languages may allow users to specify any or all of these parameters, but they are not required to be able to do so.

The following serialization parameters are defined:

The method parameter identifies the overall output method that should be used for serializing. The value of the method parameter must be a valid QName. If the QName is in no namespace, then it identifies a method specified in this document and must be one of xml, html, xhtml, or text; in this case, the output method specified must be used for serializing. If the QName is in a namespace, then it identifies an implementation-defined output method; the behavior in this case is not specified by this document.

The detailed semantics of each parameter will be described separately for each output method for which it is applicable. If the semantics of a parameter are not described for an output method, then it is not applicable to that output method.

4 Phases of Serialization

Following the normalization process described in 2 Serializing Arbitrary Instances of the Data Model, serialization can be regarded as involving four phases of processing, carried out sequentially as follows:

1. Markup generation produces the representation of start and end tags for elements, and other constructs such as XML declarations, processing instructions, and so on. This is influenced by the parameters method, doctype-system, doctype-public, include-content-type, indent, omit-xml-declaration, standalone, undeclare-namespaces and version.

2. Character expansion is concerned with the representation of characters appearing in text and attribute nodes in the instance of the data model. The substitution processes that may apply are listed below, in priority order: a character that is handled by one process in this list will be unaffected by processes appearing later in the list:

   • URI escaping (in the case of URI-valued attributes in the HTML and XHTML output methods), as determined by the escape-uri-attributes parameter

   • Creation of CDATA sections, as determined by the cdata-section-elements parameter. Note that this is also affected by the encoding parameter, in that characters not present in the selected encoding cannot be represented in a CDATA section.

   • Character mapping, as determined by the use-character-maps parameter.

   • Escaping of special characters according to XML or HTML rules, for example replacing < by <

3. Unicode Normalization, if requested by the normalization-form parameter. Unicode normalization is applied to the character stream that results after all markup generation and character expansion has taken place.

   For the definitions of the various normalization forms, see [Character Model for the World Wide Web 1.0]

   The meanings associated with the possible values of the normalization-form parameter are as follows:

   • NFC specifies the serialized result should be in Unicode Normalization Form C.

   • NFD specifies the serialized result should be in Unicode Normalization Form D.

   • NFKC specifies the serialized result should be in Unicode Normalization Form KC.

   • NFKD specifies the serialized result should be in Unicode Normalization Form KD.

   • fully-normalized specifies the serialized result should be in fully normalized form.

   • none specifies that no Unicode normalization should be applied.

   • An implementation-defined value has an implementation-defined effect.

   
   
   

   Note:

   Any characters produced under the effect of the use-character-maps parameter are not subject to Unicode normalization. If the normalization-form parameter has a value other than none and the use-character-maps parameter is not empty, the whole of the serialized document may not be in the normalization form specified by the normalization-form parameter.

4. Encoding, as controlled by the encoding parameter. This converts the character stream produced by the previous phases into a byte stream.

   Note:

   Serialization is only defined in terms of encoding the result as a stream of bytes. However, a processor may provide an option that allows the encoding phase to be skipped, so that the result of serialization is a stream of Unicode characters. The effect of any such option is implementation-defined, and a processor is not required to support such an option.

5 XML Output Method

The xml output method outputs the instance of the data model as an XML entity that must satisfy the rules for either a well-formed XML document entity or a well-formed XML external general parsed entity, or both, unless the processor is unable to satisfy those rules due to either serialization errors or the requirements of the character expansion phase of serialization, as described in 4 Phases of Serialization. If the processor is unable to satisfy those requirements for any other reason, a serialization error results. The processor must signal the error.

If the document node of the instance of the data model has a single element node child and no text node children, then the serialized output is a well-formed XML document entity, and the serialized output must conform to the XML Namespaces Recommendation [XML Names]. If the instance of the data model does not take this form, then the serialized output is a well-formed XML external general parsed entity, which, when referenced within a trivial XML document wrapper like this

<!DOCTYPE doc [
<!ENTITY e SYSTEM "entity-URI">
]>
<doc>&e;</doc>

where entity-URI is a URI for the entity, produces a document which must itself be a well-formed XML document conforming to the XML Namespaces Recommendation [XML Names].

In addition, the output must be such that if a new tree was constructed by parsing the XML document and converting it into an instance of the data model as specified in [Data Model], then the new instance of the data model would be the same as the instance of the data model that resulted from the normalization process described in 2 Serializing Arbitrary Instances of the Data Model, with the following possible exceptions:

• If the document was produced by adding a document wrapper, as described above, then it will contain an extra doc element as the document element.

• The order of attribute and namespace nodes in the two trees may be different.

• The following properties of corresponding nodes in the two trees may be different:

  • the base-uri property of document nodes and element nodes;

  • the document-uri and unparsed-entities properties of document nodes;

  • the type-name and typed-value properties of element and attribute nodes;

  • the nilled property of element nodes;

  • the content property of text nodes, due to the effect of the indent and use-character-maps parameters.

  
  
  

• The new tree may contain additional attributes and text nodes resulting from the expansion of default and fixed values in its DTD or schema.

• The type annotations of the nodes in the two trees may be different. Type annotations in a result tree are discarded when the tree is serialized. Any new type annotations obtained by parsing the document will depend on whether the serialized XML document is assessed against a schema, and this may result in type annotations that are either more or less precise than those in the original result tree.

  Note:

  In order to influence the type annotations in the instance of the data model that would result from processing a serialized XML document, the author of the XSLT stylesheet, XQuery expression or other process may wish to create the instance of the data model that is input to the serialization process so that it makes use of mechanisms provided by [XML Schema], such as xsi:type and xsi:schemaLocation attributes. The serialization process will not automatically create such attributes in the serialized document if those attributes were not part of the result tree that is to be serialized. Similarly, it is possible that an element node in the instance of the data model that is to be serialized has the nilled property with the value true, but no xsi:nil attribute. The serialization process will not create such an attribute in the serialized document simply to reflect the value of the property. The value of the nilled property has no direct effect on the serialized result.

• Additional namespace nodes may be present in the new tree if the serialization process did not undeclare one or more namespaces, as described in 5.7 XML Output Method: the undeclare-namespaces Parameter, and the starting instance of the data model contained an element node with a namespace node that declared some prefix, but a child element of that node did not have any namespace node that declared the same prefix.

  The original tree may contain namespace nodes that are not present in the new tree, as the process of creating an instance of the data model may ignore namespace declarations in some circumstances. See Section 7.2.3 Construction from an Infoset^DM and Section 7.2.4 Construction from a PSVI^DM of [Data Model] for additional information.

• If the indent parameter has the value yes,

  • additional text nodes consisting of whitespace characters may be present in the new tree; and

  • text nodes in the original tree that contained only whitespace characters may correspond to text nodes in the new tree that contain additional whitespace characters that were not present in the original tree

  See 5.3 XML Output Method: the indent Parameter for more information on the indent parameter.

• Additional nodes may be present in the new tree, and the values of attribute nodes and text nodes in the new tree may be different from those in the original tree, due to the character expansion phase of serialization.

  Note:

  The use-character-maps parameter can cause arbitrary characters to be inserted into the serialized XML document in an unescaped form, including characters that would be considered to be part of XML markup. Such characters could result in arbitrary new element nodes, attribute nodes, and so on, in the new tree that results from processing the serialized XML document.

A consequence of this rule is that certain whitespace characters must be output as character references, to ensure that they survive the round trip through serialization and parsing. Specifically, CR, NEL and LINE SEPARATOR characters in text nodes must be output respectively as 
, …, and 
, or their equivalents; while CR, NL, TAB, NEL and LINE SEPARATOR characters in attribute nodes must be output respectively as 
, 
, 	, …, and 
, or their equivalents.

For example, an attribute with the value "x" followed by "y" separated by a newline will result in the output "x
y" (or with any equivalent character reference). The XML output cannot be "x" followed by a literal newline followed by a "y" because after parsing, the attribute value would be "x y" as a consequence of the XML attribute normalization rules.

Note:

XML 1.0 did not permit processors to normalize NEL or LINE SEPARATOR characters to a LINE FEED character. However, if a document entity that specifies version 1.1 invokes an external general parsed entity with no text declaration or a text declaration that specifies version 1.0, the external parsed entity is processed according to the rules of XML 1.1. For this reason, NEL and LINE SEPARATOR characters in text and attribute nodes must always be escaped using character references or CDATA sections, regardless of the value of the version parameter.

It is a serialization error to request the output of a document type declaration, or of a standalone parameter, if the instance of the data model contains text nodes or multiple element nodes as children of the root node. The processor must either signal the error, or recover by ignoring the request to output a document type declaration or standalone parameter.

The result of serialization using the XML output method is not guaranteed to be well-formed XML if character maps have been specified (see 9 Character Maps) or if nodes in the instance of the data model contain characters that are invalid in XML (introduced, perhaps, by calling a user-written extension function: this is an error but the processor is not required to signal it).

<x:foo xmlns:x="http://example.org/x"
       xmlns:y="http://example.org/y"
       xmlns:z="http://example.org/z">

<x:bar xmlns:x="http://example.org/x"
       xmlns:y="http://example.org/y">...

<x:foo xmlns:x="http://example.org/x"
       xmlns:y="http://example.org/y"
       xmlns:z="http://example.org/z">
      <x:bar xmlns:z="">...</x:bar>
</x:foo>

6 XHTML Output Method

The xhtml output method serializes the instance of the data model as XML, using the HTML compatibility guidelines defined in the XHTML specification.

It is entirely the responsibility of the person or process that creates the instance of the data model to ensure that the instance of the data model conforms to the [XHTML 1.0] or [XHTML 1.1] specification. It is not an error if the instance of the data model is invalid XHTML. Equally, it is entirely under the control of the person or process that creates the instance of the data model whether the output conforms to XHTML Strict, XHTML Transitional, XHTML Frameset, or XHTML Basic.

The serialization of the instance of the data model follows the same rules as for the xml output method, with the exceptions noted below. These differences are based on the HTML compatibility guidelines published in Appendix C of [XHTML 1.0], which are designed to ensure that as far as possible, XHTML is rendered correctly on user agents designed originally to handle HTML.

• Given an empty instance of an XHTML element whose content model is not EMPTY (for example, an empty title or paragraph) the serializer must not use the minimized form. That is, it must output <p></p> and not <p />.

• Given an XHTML element whose content model is EMPTY, the serializer must use the minimized tag syntax, for example <br />, as the alternative syntax <br></br> allowed by XML gives uncertain results in many existing user agents. The serializer must include a space before the trailing />, e.g. <br />, <hr /> and <img src="karen.jpg"  alt="Karen" />.

• The serializer must not use the entity reference ' which, although legal in XML and therefore in XHTML, is not defined in HTML and is not recognized by all HTML user agents.

• The serializer should output namespace declarations in a way that is consistent with the requirements of the XHTML DTD if this is possible. The DTD requires the declaration xmlns="http://www.w3.org/1999/xhtml" to appear on the html element, and only on the html element. The serializer must output namespace declarations that are consistent with the namespace nodes present in the result tree, but it should avoid outputting redundant namespace declarations on elements where the DTD would make them invalid.

  Note:

  Where the process used to construct the input instance of the data model does not provide complete control over the prefix used for an element name in the instance of the data model or control of whether the element is in the default namespace (for instance, the XSLT namespace fixup process), implementors are encouraged to provide means or endeavor to preserve the obvious intent of a user to place the html element in the default namespace, wherever possible. For example, implementors of XSLT processors are encouraged to place the html element that results from a literal result element like the following in the default namespace. <html xmlns="http://www.w3.org/1999/xhtml"> ... </html>

• If the instance of the data model includes a head element in the XHTML namespace, and the include-content-type parameter has the value yes, the xhtml output method must add a meta element immediately after the start-tag of the head element specifying the character encoding actually used.

  For example,

  <head>
  <meta http-equiv="Content-Type" content="text/html; charset=EUC-JP"/>
  ...
  

  The content type should be set to the value given for the media-type parameter; the default value for XHTML is text/html. The value application/xhtml+xml, registered in [RFC3236], may also be used.

  If the instance of the data model includes a head element that has a meta element child, the processor should replace any content attribute of the meta element, or add such an attribute, with the value as described above, rather than output a new meta element.

• If the escape-uri-attributes parameter has the value yes, the xhtml output method must escape non-ASCII characters in URI attribute values using the method recommended in Seciton 2.4.1 of [RFC2396].

Note:

This escaping is deliberately confined to non-ASCII characters, because escaping of ASCII characters is not always appropriate, for example when URIs or URI fragments are interpreted locally by the HTML user agent. Even in the case of non-ASCII characters, escaping can sometimes cause problems. More precise control of URI escaping is therefore available by setting escape-uri-attributes to no, and controlling the escaping of URIs by means of the fn:escape-uri function defined in [Functions and Operators].

Note:

As with the XML output method, the XHTML output method outputs an XML declaration unless it is suppressed using the omit-xml-declaration parameter. Appendix C.1 of [XHTML 1.0] provides advice on the consequences of including, or omitting, the XML declaration.

Note:

Appendix C of [XHTML 1.0] describes a number of compatibility guidelines for users of XHTML who wish to render their XHTML documents with HTML user agents. In some cases, such as the guideline on the form empty elements should take, only the serialization process itself has the ability to follow the guideline. In such cases, those guidelines are reflected in the requirements on the processor described above. In all other cases, the guidelines can be adhered to by the instance of the data model that is input to the serialization process. The guideline on the use of whitespace characters in attribute values is one such example. It is the responsibility of the person or process that creates the instance of the data model that is input to the serialization process to ensure it is created in a way that is consistent with the guidelines. No serialization error results if the input instance of the data model does not adhere to the guidelines.

7 HTML Output Method

The html output method outputs the instance of the data model as HTML.

<xsl:stylesheet version="2.0"
                xmlns:xsl="http://www.w3.org/1999/XSL/Transform">

<xsl:output method="html"/>

<xsl:template match="/">
<html>
<xsl:apply-templates/>
</html>
</xsl:template>

...

</xsl:stylesheet>

The version parameter indicates the version of the HTML Recommendation [HTML] to which the serialized result should conform. If the processor does not support the version of HTML specified by the version parameter, it must signal a serialization error.

<script>if (a &lt; b) foo()</script>

<script><![CDATA[if (a < b) foo()]]></script>

<script>if (a < b) foo()</script>

<script type="text/javascript">
      document.write ("<em>This won't work</em>")
</script>

<script type="text/javascript">
      document.write ("<em>This won't work</em>")
</script>

<script type="text/javascript">
      document.write ("&lt;em&gt;This won't work&lt;/em&gt;")
</script>

<script type="text/javascript">
      document.write ("&lt;em&gt;This will work&lt;\/em&gt;")
</script>

<OPTION selected="selected">

<OPTION selected>

<BODY bgcolor='&{{randomrbg}};'>

<BODY bgcolor='&{randomrbg};'>

<HEAD>
<META http-equiv="Content-Type" content="text/html; charset=EUC-JP">
...

8 Text Output Method

The text output method outputs the instance of the data model by outputting the string-value of every text node in the instance of the data model in document order without any escaping.

A newline character in the instance of the data model may be output using any character sequence that is conventionally used to represent a line ending in the chosen system environment.

The media-type parameter is applicable for the text output method.

The encoding parameter identifies the encoding that the text output method must use to convert sequences of characters to sequences of bytes. Processors are required to support values of UTF-8 and UTF-16. A serialization error occurs if the implementation does not support the encoding specified by the encoding parameter. The processor must signal the error. If the instance of the data model contains a character that cannot be represented in the encoding that the processor is using for output, the implementation must signal a serialization error.

The normalization-form parameter is applicable for the text output method. The values NFC and none must be supported by the processor. A serialization error results if the value of the normalization-form parameter specifies a normalization form that is not supported by the processor; the processor must signal the error.

The use-character-maps parameter is applicable for the xml output method.

The use-character-maps parameter is applicable for the text output method.

9 Character Maps

The use-character-maps parameter is a list of characters and corresponding string substitutions.

Character maps allow a specific character appearing in a text or attribute node in the instance of the data model to be substituted by a specified string of characters during serialization. The string that is substituted is output "as is", and the serializer performs no checks that the resulting document is well-formed. This mechanism can therefore be used to introduce arbitrary markup in the serialized output.

Character mapping is applied to the characters that actually appear in a text or attribute node in the instance of the data model, before any other serialization operations such as escaping or Unicode normalization are applied. If a character is mapped, then it is not subjected to XML or HTML escaping, nor to Unicode normalization. The string that is substituted for a character is not validated or processed in any way by the serializer, except for translation into the target encoding. In particular, it is not subjected to XML or HTML escaping, it is not subjected to Unicode normalization, and it is not subjected to further character mapping. If the string cannot be represented using the target encoding, the serializer takes the same action as it would if the offending characters appeared directly in the instance of the data model.

Character mapping is not applied to characters in text nodes whose parent elements are listed in the cdata-section-elements parameter, nor to characters in attribute values that are subject to the URI escaping defined for the HTML and XHTML output methods, unless URI escaping has been disabled using the escape-uri-attributes parameter in the output definition.

On serialization, occurrences of a character specified in the use-character-maps in text nodes and attribute values are replaced by the corresponding string from the use-character-maps parameter.

Note:

Using a character map can result in non-well-formed documents if the string contains XML-significant characters. For example, it is possible to create documents containing unmatched start and end tags, references to entities that are not declared, or attributes that contain tags or unescaped quotation marks.

Character mapping is applied to the characters that actually appear in a text or attribute node in the instance of the data model, before any other serialization operations such as escaping or Unicode normalization are applied.

Character mapping is not applied to characters for which output escaping has been disabled (disabling output escaping is an [XSLT 2.0] feature), nor to characters in text nodes whose parent elements are listed in the cdata-section-elements parameter, nor to characters in attribute values that are subject to the URI escaping defined for the HTML and XHTML output methods, unless URI escaping has been disabled using the escape-uri-attributes parameter.

If a character is mapped, then it is not subjected to XML or HTML escaping.

A serialization error occurs if character mapping causes the output of a string containing a character that cannot be represented in the encoding that the processor is using for output. The processor must signal the error.