Character Model for the World Wide Web 1.0: Normalization

1.1 Goals and Scope

The goal of this document is to facilitate use of the Web by all people, regardless of their language, script, writing system, and cultural conventions, in accordance with the W3C goal of universal access. One basic prerequisite to achieve this goal is to be able to transmit and process the characters used around the world in a well-defined and well-understood way.

The main target audience of this document is W3C specification developers. This document and parts of it can be referenced from other W3C specifications.

Other audiences of this document include software developers, content developers, and authors of specifications outside the W3C. Software developers and content developers implement and use W3C specifications. This document defines some conformance requirements for software developers and content developers that implement and use W3C specifications. It also helps software developers and content developers to understand the character-related provisions in other W3C specifications.

The character model described in this document provides authors of specifications, software developers, and content developers with a common reference for consistent, interoperable text manipulation on the World Wide Web. Working together, these three groups can build a more international Web.

Topics addressed include character encoding identification, early uniform normalization, string identity matching, string indexing, and URI conventions. Some introductory material on characters and character encoding is also provided.

Topics not addressed or barely touched include collation (sorting), fuzzy matching and language tagging. Some of these topics may be addressed in a future version of this specification.

At the core of the model is the Universal Character Set (UCS), defined jointly by the Unicode Standard [Unicode] and ISO/IEC 10646 [ISO/IEC 10646]. In this document, Unicode is used as a synonym for the Universal Character Set. The model will allow Web documents authored in the world's scripts (and on different platforms) to be exchanged, read, and searched by Web users around the world.

1.2 Background

This section provides some historical background on the topics addressed in this document.

Starting with Internationalization of the Hypertext Markup Language [RFC 2070], the Web community has recognized the need for a character model for the World Wide Web. The first step towards building this model was the adoption of Unicode as the document character set for HTML.

The choice of Unicode was motivated by the fact that Unicode:

• is the only universal character repertoire available,

• provides a way of referencing characters independent of the encoding of the text,

• is being updated/completed carefully,

• is widely accepted and implemented by industry.

W3C adopted Unicode as the document character set for HTML in [HTML 4.0]. The same approach was later used for specifications such as XML 1.0 [XML 1.0] and CSS2 [CSS2]. W3C specifications and applications now use Unicode as the common reference character set.

The IETF has adopted some policies on the use of character sets on the Internet (see [RFC 2277]).

When data transfer on the Web remained mostly unidirectional (from server to browser), and where the main purpose was to render documents, the use of Unicode without specifying additional details was sufficient. However, the Web has grown:

• Data transfers among servers, proxies, and clients, in all directions, have increased.

• Non-ASCII characters [ISO/IEC 646] are being used in more and more places.

• Data transfers between different protocol/format elements (such as element/attribute names, URI components, and textual content) have increased.

• More and more APIs are defined, not just protocols and formats.

In short, the Web may be seen as a single, very large application (see [Nicol]), rather than as a collection of small independent applications.

While these developments strengthen the requirement that Unicode be the basis of a character model for the Web, they also create the need for additional specifications on the application of Unicode to the Web. Some aspects of Unicode that require additional specification for the Web include:

• Choice of Unicode encoding forms (UTF-8, UTF-16, UTF-32).

• Counting characters, measuring string length in the presence of variable-length character encodings and combining characters.

• Duplicate encodings of characters (e.g. precomposed vs decomposed).

• Use of control codes for various purposes (e.g. bidirectionality control, symmetric swapping, etc.).

It should be noted that such aspects also exist in legacy encodings (where legacy encoding is taken to mean any character encoding not based on Unicode), and in many cases have been inherited by Unicode in one way or another from such legacy encodings.

The remainder of this document presents additional specifications and requirements to ensure an interoperable character model for the Web, taking into account earlier work (from W3C, ISO and IETF).

For information about the requirements that informed the development of important parts of this specification, see Requirements for String Identity Matching and String Indexing [CharReq].

1.3 Terminology and Notation

For the purpose of this specification, the producer of text data is the sender of the data in the case of protocols, and the tool that produces the data in the case of formats. The recipient of text data is the software module that receives the data.

Unicode code points are denoted as U+hhhh, where "hhhh" is a sequence of at least four, and at most six hexadecimal digits.

Characters have been used in various examples that will not appear as intended unless you have the appropriate font. Care has been taken to ensure that the examples nevertheless remain understandable.

3.1 Motivation

3.2 Definitions for W3C Text Normalization

For use on the Web, this document defines Web-related text normalization forms by starting with Unicode Normalization Form C (NFC), and additionally addressing the issues of legacy encodings, character escapes, includes, and character and markup boundaries. Examples illustrating these definitions can be found in 3.3 Examples.

A normalizing transcoder is a transcoder that converts from a legacy encoding to a Unicode encoding form and ensures that the result is in Unicode Normalization Form C (see 3.2.1 Unicode-normalized text). For most legacy encodings, it is possible to construct a normalizing transcoder (by using any transcoder followed by a normalizer); it is not possible to do so if the encoding's repertoire contains characters not represented in Unicode.

3.3 Examples

In some of the following examples, '¸' is used to depict the character U+0327 COMBINING CEDILLA, for the purposes of illustration. Had a real U+0327 been used instead of this spacing (non-combining) variant, some browsers might combine it with a preceding 'c', resulting in a display indistinguishable from a U+00E7 'ç' and a loss of understandability of the examples. In addition, if the sequence c + combining cedilla were present, this document would not be include-normalized and would therefore not conform to itself.

It is also assumed that the example strings are relevant constructs for the purposes of full-normalization.

3.4 Responsibility for Normalization

This section defines the W3C Text Normalization Model, based on early uniform normalization.

Unless otherwise specified, the word 'normalization' in this section may refer to 'include-normalization' or 'full-normalization', depending on which is most appropriate for the specification or implementation under consideration.

An operation is normalization-sensitive if its output(s) are different depending on the state of normalization of the input(s); if the output(s) are textual, they are deemed different only if they would remain different were they to be normalized. These operations are any that involve comparison of characters or character counting, as well as some other operations such as ‘delete first character’ or ‘delete last character’.

A text-processing component is a component that recognizes data as text. This specification does not specify the boundaries of a text-processing component, which may be as small as one line of code or as large as a complete application. A text-processing component may receive text, produce text, or both.

Certified text is text which satisfies at least one of the following conditions:

1. it has been confirmed through inspection that the text is in normalized form

2. the source text-processing component is identified and is known to produce only normalized text.

Suspect text is text which is not certified.

Given the definitions and considerations above, specifications, implementations and content have some responsibilities which are listed below. @@@Specifications, implementations and content ought to follow as many of the responsibilities as possible and make sure that this is done in a way that is consistent overall.

• C300  [C]  Text content SHOULD be in fully-normalized form and if not SHOULD at least be in include-normalized form.

• C301  [S]  Specifications of text-based formats and protocols SHOULD, as part of their syntax definition, require that the text be in normalized form.

• C302  [S]  [I]  A text-processing component that receives suspect text SHOULD NOT perform any normalization-sensitive operations unless it has first confirmed through inspection that the text is in normalized form, and MUST NOT normalize the suspect text. Private agreements MAY, however, be created within private systems which are not subject to these rules, but any externally observable results SHOULD be the same as if the rules had been obeyed.

• C303  [I]  A text-processing component which modifies text and performs normalization-sensitive operations SHOULD behave as if normalization took place after each modification, so that any subsequent normalization-sensitive operations always behave as if they were dealing with normalized text.

  EXAMPLE: If the 'z' is deleted from the (normalized) string cz¸ (where '¸' represents a combining cedilla, U+0327), normalization is necessary to turn the denormalized result c¸ into the properly normalized ç. If the software that deletes the 'z' later uses the string in a normalization-sensitive operation, it needs to normalize the string before this operation to ensure correctness; otherwise, normalization may be deferred until the data is exposed. Analogous cases exist for insertion and concatenation (e.g. xf:concat(xf:substring('cz¸', 1, 1), xf:substring('cz¸', 3, 1)) in XQuery [XQuery Operators]).

  NOTE: Software that denormalizes a string such as in the deletion example above does not need to perform a potentially expensive re-normalization of the whole string to ensure that the string is normalized. It is sufficient to go back to the last non-composing character and re-normalize forward to the next non-composing character; if the string was normalized before the denormalizing operation, it will now be re-normalized.

• C304  [S]  Specifications of text-based languages and protocols SHOULD define precisely the construct boundaries necessary to obtain a complete definition of full-normalization. These definitions SHOULD include at least the boundaries between markup and character data as well as entity boundaries (if the language has any include mechanism) and SHOULD include any other boundary that may create denormalization when instances of the language are processed.

• C305  [C]  Even when authoring in a (formal) language that does not mandate full-normalization, content developers SHOULD avoid composing characters at the beginning of constructs that may be significant, such as at the beginning of an entity that will be included, immediately after a construct that causes inclusion or immediately after markup.

• C306  [I]  Authoring tool implementations for a (formal) language that does not mandate full-normalization SHOULD prevent users from creating content with composing characters at the beginning of constructs that may be significant, such as at the beginning of an entity that will be included, immediately after a construct that causes inclusion or immediately after markup, or SHOULD warn users when they do so.

• C307  [I]  Implementations which transcode text from a legacy encoding to a Unicode encoding form SHOULD use a normalizing transcoder.

  NOTE: Except when an encoding's repertoire contains characters not represented in Unicode, it is always possible to construct a normalizing transcoder by using any transcoder followed by a normalizer.

• C308  [S]  Where operations may produce unnormalized output from normalized text input, specifications of API components (functions/methods) that implement these operations MUST define whether normalization is the responsibility of the caller or the callee. Specifications MAY state that performing normalization is optional for some API components; in this case the default SHOULD be that normalization is performed, and an explicit option SHOULD be used to switch normalization off. Specifications SHOULD NOT make the implementation of normalization optional.

  EXAMPLE: The concatenation operation may either concatenate sequences of codepoints without normalization at the boundary, or may take normalization into account to avoid producing unnormalized output from normalized input. An API specification must define whether the operation normalizes at the boundary or leaves that responsibility to the application using the API.

• C309  [S]  Specifications that define a mechanism (for example an API or a defining language) for producing textual data object SHOULD require that the final output of this mechanism be normalized.

  EXAMPLE: XSL Transformations [XSLT] and the DOM Load & Save specification [DOM3 LS] are examples of specifications that define text output and that should specify that this output be in normalized form.

  NOTE: As an optimization, it is perfectly acceptable for a system to define the producer to be the actual producer (e.g. a small device) together with a remote component (e.g. a server serving as a kind of proxy) to which normalization is delegated. In such a case, the communications channel between the device and proxy server is considered to be internal to the system, not part of the Web. Only data normalized by the proxy server is to be exposed to the Web at large, as shown in the illustration below:

  

  Illustration of a text producer defined as including a proxy.

  A similar case would be that of a Web repository receiving content from a user and noticing that the content is not properly normalized. If the user so requests, it would certainly be proper for the repository to normalize the content on behalf of the user, the repository becoming effectively part of the producer for the duration of that operation.

C310  [S]  [I]  Specifications and implementations MUST document any deviation from the above requirements. C311  [S]  Specifications MUST document any known security issues related to normalization.