Internationalization Best Practices for Spec Developers

It should be possible to associate a language with any piece of localizable text or natural language content.

Where possible, there should be a way to label natural language changes in inline text.

Consider whether it is useful to express the intended linguistic audience of a resource, in addition to specifying the language used for text processing.

Values for language declarations must use BCP 47.

Refer to BCP 47, not to RFC 5646.

Be specific about what level of conformance you expect for language tags: BCP 47 defines two levels of conformance, "valid" and "well-formed".

Specifications may require implementations to check if language tags are "valid", but in most circumstances should only require that the language tags be "well-formed".

Specifications should require content and content authors to use "valid" language tags.

Reference BCP47 for language tag matching.

The specification should indicate how to define the default text-processing language for the resource as a whole.

Content within the resource should inherit the language of the text-processing declared at the resource level, unless it is specifically overridden.

Consider whether it is necessary to have separate declarations to indicate the text-processing language versus metadata about the expected use of the resource.

If there is only one language declaration for a resource, and it has more than one language tag as a value, it must be possible to identify the default text-processing language for the resource.

By default, blocks of content should inherit any text-processing language set for the resource as a whole.

It should be possible to indicate a change in language for blocks of content where the language changes.

It should be possible to indicate language for spans of inline text where the language changes.

It must be possible to indicate base direction for each individual paragraph-level item of natural language text that will be read by someone.

It must be possible to indicate base direction changes for embedded runs of inline bidirectional text for all localizable text.

Annotating right-to-left text must require the minimum amount of effort for people who work natively with right-to-left scripts.

Values for the default base direction should include left-to-right, right-to-left, and auto.

Provide metadata constructs that can be used to indicate the base direction of any natural language string.

Specify that consumers of strings should use heuristics, preferably based on the Unicode Standard first-strong algorithm, to detect the base direction of a string except where metadata is provided.

Where possible, define a field to indicate the default direction for all strings in a given resource or document.

Do NOT assume that a creating a document-level default without the ability to change direction for any string is sufficient.

If metadata is not available due to legacy implementations and cannot otherwise be provided, specifications MAY allow a base direction to be interpolated from available language metadata.

Specifications MUST NOT require the production or use of paired bidi controls.

It must be possible to indicate spans of inline text where the base direction changes. If markup is available, this is the preferred method. Otherwise your specification must require that Unicode control characters are recognized by the receiving application, and correctly implemented.

It must be possible to also set the direction for a span to auto. This means that the base direction will be determined by examining the content itself. A typical approach here would be to set the direction based on the first strong directional character outside of any markup.

If users use Unicode bidirectional control characters, the isolating RLI/LRI/FSI with PDI characters must be supported by the application and recommended (rather than RLE/LRE with PDF) by the spec.

Use of RLM/LRM should be appropriate, and expectations of what those controls can and cannot do should be clear in the spec.

For markup, provide dedicated attributes for control of base direction and bidirectional overrides; do not rely on the user applying style properties to arbitrary markup to achieve bidi control.

For markup, allow bidi attributes on all inline elements in markup that contain text.

For markup, provide attributes that allow the user to (a) create an embedded base direction or (b) override the bidirectional algorithm altogether; the attribute should allow the user to set the direction to LTR or RTL or the aforementioned Auto in either of these two scenarios.

Specifications SHOULD use specific terms, when available, instead of the general term 'character'.

When specifications use the term 'character' the specifications MUST define which meaning they intend, and SHOULD explicitly define the term 'character' to mean a Unicode code point.

Specifications, software and content MUST NOT require or depend on a one-to-one relationship between characters and units of physical storage.

Specifications, software and content MUST NOT require or depend on a one-to-one correspondence between characters and the sounds of a language.

Specifications, software and content MUST NOT require or depend on a one-to-one mapping between characters and units of displayed text.

Specifications and software MUST NOT require nor depend on a single keystroke resulting in a single character, nor that a single character be input with a single keystroke (even with modifiers), nor that keyboards are the same all over the world.

Textual data objects defined by protocol or format specifications MUST be in a single character encoding.

All specifications that involve processing of text MUST specify the processing of text according to the Reference Processing Model described by the rest of the recommendations in this list.

Specifications MUST define text in terms of Unicode characters, not bytes or glyphs.

For their textual data objects specifications MAY allow use of any character encoding which can be transcoded to a Unicode encoding form.

Specifications MAY choose to disallow or deprecate some character encodings and to make others mandatory. Independent of the actual character encoding, the specified behavior MUST be the same as if the processing happened as follows: (a) The character encoding of any textual data object received by the application implementing the specification MUST be determined and the data object MUST be interpreted as a sequence of Unicode characters - this MUST be equivalent to transcoding the data object to some Unicode encoding form, adjusting any character encoding label if necessary, and receiving it in that Unicode encoding form, (b) All processing MUST take place on this sequence of Unicode characters, (c) If text is output by the application, the sequence of Unicode characters MUST be encoded using a character encoding chosen among those allowed by the specification.

If a specification is such that multiple textual data objects are involved (such as an XML document referring to external parsed entities), it MAY choose to allow these data objects to be in different character encodings. In all cases, the Reference Processing Model MUST be applied to all textual data objects.

Specifications SHOULD NOT arbitrarily exclude code points from the full range of Unicode code points from U+0000 to U+10FFFF inclusive.

Specifications MUST NOT allow code points above U+10FFFF.

Specifications SHOULD NOT allow the use of codepoints reserved by Unicode for internal use.

Specifications MUST NOT allow the use of surrogate code points.

Specifications SHOULD exclude compatibility characters in the syntactic elements (markup, delimiters, identifiers) of the formats they define.

Specifications SHOULD allow the full range of Unicode for user-defined values.

Specifications MUST NOT require the use of private use area characters with particular assignments.

Specifications MUST NOT require the use of mechanisms for defining agreements of private use code points.

Specifications and implementations SHOULD NOT disallow the use of private use code points by private agreement.

Specifications MAY define markup to allow the transmission of symbols not in Unicode or to identify specific variants of Unicode characters.

Specifications SHOULD allow the inclusion of or reference to pictures and graphics where appropriate, to eliminate the need to (mis)use character-oriented mechanisms for pictures or graphics.

Specifications MUST either specify a unique character encoding, or provide character encoding identification mechanisms such that the encoding of text can be reliably identified.

When designing a new protocol, format or API, specifications SHOULD require a unique character encoding.

When basing a protocol, format, or API on a protocol, format, or API that already has rules for character encoding, specifications SHOULD use rather than change these rules.

When a unique character encoding is required, the character encoding MUST be UTF-8, or UTF-16.

Specifications SHOULD avoid using the terms 'character set' and 'charset' to refer to a character encoding, except when the latter is used to refer to the MIME charset parameter or its IANA-registered values. The term 'character encoding', or in specific cases the terms 'character encoding form' or 'character encoding scheme', are RECOMMENDED.

If the unique encoding approach is not taken, specifications SHOULD require the use of the IANA charset registry names, and in particular the names identified in the registry as 'MIME preferred names', to designate character encodings in protocols, data formats and APIs.

Character encodings that are not in the IANA registry SHOULD NOT be used, except by private agreement.

If an unregistered character encoding is used, the convention of using 'x-' at the beginning of the name MUST be followed.

If the unique encoding approach is not chosen, specifications MUST designate at least one of the UTF-8 and UTF-16 encoding forms of Unicode as admissible character encodings and SHOULD choose at least one of UTF-8 or UTF-16 as required encoding forms (encoding forms that MUST be supported by implementations of the specification).

Specifications that require a default encoding MUST define either UTF-8 or UTF-16 as the default, or both if they define suitable means of distinguishing them.

Specifications MUST NOT propose the use of heuristics to determine the encoding of data.

Specifications MUST define conflict-resolution mechanisms (e.g. priorities) for cases where there is multiple or conflicting information about character encoding.

Specifications should provide a mechanism for escaping characters, particularly those which are invisible or ambiguous.

Specifications SHOULD NOT invent a new escaping mechanism if an appropriate one already exists.

The number of different ways to escape a character SHOULD be minimized (ideally to one).

Escape syntax SHOULD require either explicit end delimiters or a fixed number of characters in each character escape. Escape syntaxes where the end is determined by any character outside the set of characters admissible in the character escape itself SHOULD be avoided.

Whenever specifications define character escapes that allow the representation of characters using a number, the number MUST represent the Unicode code point of the character and SHOULD be in hexadecimal notation.

Escaped characters SHOULD be acceptable wherever their unescaped forms are; this does not preclude that syntax-significant characters, when escaped, lose their significance in the syntax. In particular, if a character is acceptable in identifiers and comments, then its escaped form should also be acceptable.

Protocols, data formats and APIs MUST store, interchange or process text data in logical order.

Independent of whether some implementation uses logical selection or visual selection, characters selected MUST be kept in logical order in storage.

Specifications of protocols and APIs that involve selection of ranges SHOULD provide for discontiguous logical selections, at least to the extent necessary to support implementation of visual selection on screen on top of those protocols and APIs.

Specifications SHOULD NOT define a string as a 'byte string'.

The 'character string' definition SHOULD be used by most specifications.

Use U+XXXX syntax to represent Unicode code points in the specification.

Since specifications in general need both a definition for their characters and the semantics associated with these characters, specifications SHOULD include a reference to the Unicode Standard, whether or not they include a reference to ISO/IEC 10646.

A generic reference to the Unicode Standard MUST be made if it is desired that characters allocated after a specification is published are usable with that specification. A specific reference to the Unicode Standard MAY be included to ensure that functionality depending on a particular version is available and will not change over time.

All generic references to the Unicode Standard MUST refer to the latest version of the Unicode Standard available at the date of publication of the containing specification.

All generic references to ISO/IEC 10646 MUST refer to the latest version of ISO/IEC 10646 available at the date of publication of the containing specification.

The character string is RECOMMENDED as a basis for string indexing.

Grapheme clusters MAY be used as a basis for string indexing in applications where user interaction is the primary concern.

Specifications that define indexing in terms of grapheme clusters MUST either: (a) define grapheme clusters in terms of extended grapheme clusters as defined in Unicode Standard Annex #29, Unicode Text Segmentation (UTR #29), or (b) define specifically how tailoring is applied to the indexing operation.

The use of byte strings for indexing is NOT RECOMMENDED.

A UTF-16 code unit string is NOT RECOMMENDED as a basis for string indexing, even if this results in a significant improvement in the efficiency of internal operations when compared to the use of character string.

Specifications that need a way to identify substrings or point within a string SHOULD consider ways other than string indexing to perform this operation.

Specifications SHOULD understand and process single characters as substrings, and treat indices as boundary positions between counting units, regardless of the choice of counting units.

Specifications of APIs SHOULD NOT specify single characters or single 'units of encoding' as argument or return types.

When the positions between the units are counted for string indexing, starting with an index of 0 for the position at the start of the string is the RECOMMENDED solution, with the last index then being equal to the number of counting units in the string.

String identity matching for identifiers and syntactic content should involve the following steps: (a) Ensure the strings to be compared constitute a sequence of Unicode code points (b) Expand all character escapes and includes (c) Perform any appropriate case-folding and Unicode normalization step (d) Perform any additional matching tailoring specific to the specification, and (e) Compare the resulting sequences of code points for identity.

The default recommendation for matching strings in identifiers and syntactic content is to do no normalization (ie. case folding or Unicode Normalization) of content.

'ASCII case fold' and 'Unicode canonical case fold' approaches should only be used in special circumstances.

A 'Unicode compatibility case fold' approach should not be used.

Specifications of vocabularies MUST define the boundaries between syntactic content and character data as well as entity boundaries (if the language has any include mechanism).

Specifications SHOULD NOT specify a Unicode normalization form for encoding, storage, or interchange of a given vocabulary.

Implementations MUST NOT alter the normalization form of textual data being exchanged, read, parsed, or processed except when required to do so as a side-effect of text transformation such as transcoding the content to a Unicode character encoding, case folding, or other user-initiated change, as consumers or the content itself might depend on the de-normalized representation.

Specifications SHOULD NOT specify compatibility normalization forms (NFKC, NFKD).

Specifications MUST document or provide a health-warning if canonically equivalent but disjoint Unicode character sequences represent a security issue.

Where operations can produce denormalized output from normalized text input, specifications MUST define whether the resulting output is required to be normalized or not. Specifications MAY state that performing normalization is optional for some operations; in this case the default SHOULD be that normalization is performed, and an explicit option SHOULD be used to switch normalization off.

Specifications that require normalization MUST NOT make the implementation of normalization optional.

Normalization-sensitive operations MUST NOT be performed unless the implementation has first either confirmed through inspection that the text is in normalized form or it has re-normalized the text itself. Private agreements MAY be created within private systems which are not subject to these rules, but any externally observable results MUST be the same as if the rules had been obeyed.

A normalizing text-processing component which modifies text and performs normalization-sensitive operations MUST behave as if normalization took place after each modification, so that any subsequent normalization-sensitive operations always behave as if they were dealing with normalized text.

Specifications and implementations that define string matching as part of the definition of a format, protocol, or formal language (which might include operations such as parsing, matching, tokenizing, etc.) MUST define the criteria and matching forms used. These MUST be one of: (a) case-sensitive (b) Unicode case-insensitive using Unicode full case-folding (c) ASCII case-insensitive.

Case-sensitive matching is RECOMMENDED for matching syntactic content, including user-defined values.

Specifications that define case-insensitive matching in vocabularies that include more than the Basic Latin (ASCII) range of Unicode MUST specify Unicode full casefold matching.

Specifications that define case-insensitive matching in vocabularies limited to the Basic Latin (ASCII) subset of Unicode MAY specify ASCII case-insensitive matching.

If language-sensitive case-sensitive matching is specified, Unicode case mappings SHOULD be tailored according to language and the source of the language used for each tailoring MUST be specified.

Specifications that define case-insensitive matching in vocabularies SHOULD NOT specify language-sensitive case-insensitive matching.

Specifications SHOULD NOT limit the size of data fields unless there is a specific practical or technical limitation.

Specifications that limit the length of a string MUST specify which type of unit (extended grapheme clusters, Unicode code points, or code units) the length limit uses.

Specifications that limit the length of a string SHOULD specify the length in terms of Unicode code points.

If a specification sets a length limit in code units (such as bytes), it MUST specify that truncation can only occur on code point boundaries.

Specifications that limit the length of a string SHOULD require truncation on grapheme boundaries, as truncation in the midst of a combining or joining sequence can alter the meaning of the string.

If a specification specifies a length limit, it SHOULD specify that any string that is truncated include an indicator, such as ellipses, that the string has been altered.

When specifying a length limitation in code units (such as bytes), specifications SHOULD set the maximum length in a way that accommodates users whose language requires multibyte code unit sequences.

Specify the UTF-8 [Unicode] encoding for the storage and processing of file names and file paths.

File names SHOULD be restricted to 255 bytes in length.

This restriction is related to limitations found in certain file systems, originally MS-DOS, but also certain Unix file systems—as well as packaging schemes such as PKZIP that depend on these file systems or subsumed their limitations—in which the limit for a specific "path element" (including directory names) is limited to 255 bytes.

Path names SHOULD be restricted to 65535 bytes in length.

This restriction is related to limitations found in file systems such as FAT32 or NTFS, which restrict the path length to 32760 (32K) code units in the UTF-16 character encoding. Each UTF-16 code unit takes 16 bits (or 2 bytes), making the limit 65,535 when measured in bytes. Note that a path name limited to 64K bytes in UTF-8 can exceed the path length limits on these file systems, since UTF-8 is a variable width encoding.

File name and path name definitions MUST NOT use the following Unicode code points.

These characters are known to cause interoperability problems with various file systems. Specifications and implementations should use an abundance of caution in their file naming when interoperability of content is key. The list of restricted characters is intended to help avoid some known problem areas, but it does not ensure that all other Unicode characters are supported.

• " [U+0022 QUOTATION MARK]
• * [U+002A ASTERISK]
• / [U+002F SOLIDUS]
• : [U+003A COLON]
• < [U+003C LESS-THAN SIGN]
• > [U+003E GREATER-THAN SIGN]
• \ [U+005C REVERSE SOLIDUS]
• | [U+007C VERTICAL LINE]
• U+007F DEL
• U+E0001 LANGUAGE TAG
• U+E007F CANCEL TAG
• Codepoints in the following ranges:
  • C0 Controls [U+0000...U+001F]
  • C1 Controls [U+0080...U+009F]
  • Private Use [U+E000...U+F8FF]
  • Specials [U+FFF0...U+FFFF]
  • Supplementary Private Use [U+F0000...U+FFFFF]
  • Supplementary Private Use [U+100000...U+10FFFF]
• . [U+002E FULL STOP] as the last character (Note that this includes the file names . and .., which have special meaning to many file systems)
• All Unicode non-character code points, specifically:
  • The 32 contiguous characters in the Basic Multilingual Plane (U+FDD0 … U+FDEF)
  • The last two code points of the Basic Multilingual Plane (U+FFFE and U+FFFF)
  • The last two code points at the end of the Supplementary Planes (U+1FFFE, U+1FFFF … U+EFFFE, U+EFFFF)

Software that sorts or searches text for users SHOULD do so on the basis of appropriate collation units and ordering rules for the relevant language and/or application.

Where searching or sorting is done dynamically, particularly in a multilingual environment, the 'relevant language' SHOULD be determined to be that of the current user, and may thus differ from user to user.

Software that allows users to sort or search text SHOULD allow the user to select alternative rules for collation units and ordering.

Specifications and implementations of sorting and searching algorithms SHOULD accommodate text that contains any character in Unicode.

Resource identifiers must permit the use of characters outside those of plain ASCII.

Specifications MUST define when the conversion from IRI references to URI references (or subsets thereof) takes place, in accordance with Internationalized Resource Identifiers (IRIs).

Do not define attribute values that will contain user readable content. Use elements for such content.

If you do define attribute values containing user readable content, provide a means to indicate directional and language information for that text separately from the text contained in the element.

Provide a way for authors to annotate arbitrary inline content using a span-like element or construct.

Specifications that define application internal identifiers (which are never shown to users and are always used for matching or processing within an application or protocol) should limit the content to a printable subset of ASCII. ASCII case-insensitive matching is recommended.

When identifiers are visible or potentially visible to users, specifications should allow the use of non-ASCII Unicode characters, in order to ensure that users in all languages can use the resulting document format or protocol with equal access. Case sensitivity (i.e. no case folding) is recommended.

If application internal identifiers are not restricted to ASCII, specifications should define the characters that are allowed to start and be part of a valid identifier.

Specifications should not allow surrogate code points (U+D800 to U+DFFF) or non-character code points in identifiers.

Specifications should not allow the C0 (U+0000 to U+001F) and C1 (U+0080 to U+009F) control characters in identifiers.

Identifiers should be case-sensitive when non-ASCII characters are allowed and case insensitive when only ASCII characters are allowed.

Application internal identifier fields or values must be wrapped with a localizable display value when displayed to end-users.

Avoid natural language text in elements or attribute values that only allow for plain text.

Avoid defining attribute values whose content will be natural language text.

Provide a span-like element that can be used for any text content to apply information needed for internationalization.

Text decoration such as underline and overline should allow lines to skip ink.

It should be possible to specify the distance of overlines and underlines from the text.

It should be possible to render text vertically for languages such as Japanese, Chinese, Korean, Mongolian, etc.

Vertical text must support line progression from LTR (eg. Mongolian) and RTL (eg. Japanese).

By default, text decoration, ruby, and the like in vertical text where lines are stacked from left to right (eg. Mongolian) should appear on the same side as for CJK vertical text. Placement should not rely on the before and after line locations.

Vertical writing modes that are equivalent to the vertical- values in CSS (only) should use UTR50 to apply default text orientation of characters. (This does not apply to writing modes that are equivalent to sideways- in CSS.)

By default, glyphs of scripts that are normally horizontal should run along a line in vertical text such that the top of the character is toward the right side of the vertical line, but there should also be a mechanism to allow them to progress down the line in upright orientation. Such a mechanism should use grapheme clusters as a minimum text unit, but where necessary allow syllabic clusters to be treated as a unit when they involve more than one grapheme cluster.

Upright Arabic text in vertical lines should use isolated letter forms and the order of text should read top to bottom.

It should be possible for some sequences of characters (particularly digits) to run horizontally within vertical lines (tate chu yoko).

Writing modes should provide values like sideways-lr and sideways-rl in CSS to allow for vertical rotation of lines of horizontal script text. UTR50 is not applicable for these cases.

Overlaps should not be exposed when transparency is applied to the joined letters in cursive text, such as for Arabic, Mongolian, and N'Ko.

When adding a text stroke or shadow, joined letters should not be separated from their neighbors in cursive script text.

Box positioning coordinates must take into account whether the text is horizontal or vertical.

'Ruby' style annotations alongside base text should be supported for Chinese, Japanese, Korean and Mongolian text, in both horizontal and vertical writing modes.

Ruby implementations should support zhuyin fuhao (bopomofo) ruby for Traditional Chinese.

Ruby implementations should support a tabular content model (such that ruby contents can be arranged in a sequence approximating to rb rb rt rt).

Ruby implementations should make it possible to use an explicit element for ruby bases, like the rb element in HTML.

Ruby implementations should allow annotations to appear on either or both sides of the base text.

Line heights must allow for characters that are taller than English.

Box sizes must allow for text expansion in translation.

Line wrapping should take into account the special rules needed for non-Latin scripts.

Avoid specifying presentational tags, such as b for bold, and i for italic.

When definining data formats, use locale-neutral serialization forms.

When defining calendar and date systems, be sure to allow for dates prior to the common era, or at least define handling of dates outside the most common range.

When defining time or date data types, ensure that the time zone or relationship to UTC is always defined.

Provide a health warning for conversion of time or date data types that are "floating" to/from incremental types, referring as necessary to the Time Zones WG Note.

Allow for leap seconds in date and time data types.

Use consistent terminology when discussing date and time values. Use 'floating' time for time zone independent values.

Keep separate the definition of time zone from time zone offset.

Use IANA time zone IDs to identify time zones. Do not use offsets or LTO as a proxy for time zone.

Use a separate field to identify time zone.

When defining rules for a "week", allow for culturally specific rules to be applied.

When defining rules for week number of year, allow for culturally specific rules to be applied.

When non-Gregorian calendars are permitted, note that the "month" field can go to 13 (undecimber).

Check whether you really need to store or access given name and family name separately.

Avoid placing limits on the length of names, or if you do, make allowance for long strings.

Try to avoid using the labels 'first name' and 'last name' in non-localized contexts.

Consider whether it would make sense to have one or more extra fields, in addition to the full name field, where users can provide part(s) of their name that you need to use for a specific purpose.

Allow for users to be asked separately how they would like you be addressed when someone contacts them.

If parts of a person's name are captured separately, ensure that the separate items can capture all relevant information.

Be careful about assumptions built into algorithms that pull out the parts of a name automatically.

Don't assume that a single letter name is an initial.

Don't require that people supply a family name.

​ Don't forget to allow people to use punctuation such as hyphens, apostrophes, etc. in names.

Don't require names to be entered all in upper case.

Don't normalize the casing in names.

Allow the user to enter a name with spaces.

Don't assume that members of the same family will share the same family name.

It may be better for a form to ask for 'Previous name' rather than 'Maiden name' or 'née'.

You may want to store the name in both Latin and native scripts, in which case you probably need to ask the user to submit their name in both native script and Latin-only form, as separate items.

In standards and standards related documents containing examples that include names of persons, use a variety of names to reflect a global audience. Avoid a bias of names specific to certain regions.

When defining email field validation, allow for EAI (smtputf8) names.

When parsing user input of numeric values, allow for digit shaping (non-ASCII digits).

When formatting numeric values for display, allow for culturally sensitive display, including the use of non-ASCII digits (digit shaping).

APIs and protocols SHOULD include language and base direction metadata for all natural language messages and data fields.

All natural language fields or messages, including error messages, defined by a given API or protocol SHOULD be localized into the preferred locale of the user or, if that language is not available, supplied with a suitable fallback or default.

Specifications for APIs or protocols SHOULD define how the user's locale is determined (this is sometimes called language negotiation).

Specifications MAY define a specific default language for messages or errors in an API or protocol.

In a multilingual environment it must be possible for the user to receive text in the language they prefer. This may depend on implicit user preferences based on the user's system or browser setup, or on user settings explicitly negotiated with the user.