1. Scope

This technical report addresses the issue of Japanese characters in XML documents thereby complementing [XML].

For high interoperability of XML documents containing Japanese characters, this technical report studies character encoding schemes (CESs) for Japanese characters. CESs for files for information interchange and CESs for each protocol (SMTP, HTTP/1.1 and others) are specified separately. Furthermore, this technical report lists conversion tables for converting these CESs to [JIS X 0221]/[Unicode 3.0], and provides names for use in encoding declarations and MIME headers.

Most parts of this technical report are mere adaptations of IETF RFCs and W3C recommendations for Japanese characters.

When some part of this technical report is adopted from [XML] and other IETF RFCs or W3C recommendations, the source reference is indicated in a subsequent note or appendice.

2. Normative References

Any of the following JISs refers to its latest version. If an RFC or I-D is superseded by another RFC, the new RFC is referenced.

Character Mapping Tables

The Unicode Consortium. Character Mapping Tables 1.0, Proposed Draft Unicode Technical Report 22.http://www.unicode.org/unicode/reports/tr22/tr22-1 , 1999.

IANA

IANA (Internet Assigned Numbers Authority). Official Names for Character Sets, ed. Keld Simonsen et al. ftp://ftp.isi.edu/in-notes/iana/assignments/character-sets

IETF RFC 1468

IETF (Internet Engineering Task Force). RFC 1468: Japanese character encoding for Internet messages, ed. J. Murai, M. Crispin, and E. van der Poel. 1993.

IETF RFC 1766

IETF (Internet Engineering Task Force). RFC 1766: Tags for the Identification of Languages, ed. H. Alvestrand. 1995.

IETF RFC 2045

IETF (Internet Engineering Task Force). RFC 2045: Multipurpose Internet Mail Extensions (MIME) Part One: Format of Internet Message Bodies, ed. N. Freed and N. Borenstein. 1996.

IETF RFC 2046

IETF (Internet Engineering Task Force). RFC 2046: Multipurpose Internet Mail Extensions (MIME) Part Two: Media Types, ed. N. Freed and N. Borenstein. 1996.

IETF RFC 2130

IETF (Internet Engineering Task Force). RFC 2130: The Report of the IAB Character Set Workshop held 29 February - 1 March, 1996, C. Weider, C. Preston, K. Simonsen, H. Alvestrand, R. Atkinson, M. Crispin, P. Svanberg. 1997.

IETF RFC 2277

IETF (Internet Engineering Task Force). RFC 2277: IETF policy on character sets and languages, ed. H. Alvestrand. 1998.

IETF RFC 2278

IETF (Internet Engineering Task Force). RFC 2278: IANA Charset Registration Procedures, ed. N. Freed and J. Postel. 1998.

IETF RFC 2279

IETF (Internet Engineering Task Force). RFC 2279: UTF-8, a transformation format of ISO 10646, ed. F. Yergeau. 1998.

IETF RFC 2376

IETF (Internet Engineering Task Force). RFC 2376: XML media types, ed. E. Whitehead and M. Murata. 1998.

IETF RFC 2396

IETF (Internet Engineering Task Force). RFC 2396: Uniform Resource Identifiers (URI): Generic Syntax, ed. T. Berners-Lee, R. Fielding, and L. Masinter. 1998.

IETF RFC 2616

IETF (Internet Engineering Task Force). RFC2616: Hypertext Transfer Protocol --- HTTP/1.1, ed. R. Fielding, J. Gettys, J. Mogul, H. Frystyk, and T. Berners-Lee. 1999.

IETF RFC 2781

IETF (Internet Engineering Task Force). RFC 2781: UTF-16, an encoding of ISO 10646, ed. P. Hoffman. and F. Yergeau. 2000.

ISO/IEC 646

International Organization for Standardization. Information technology ? ISO 7-bit coded character set for information interchange, ISO/IEC 646:1991, International Organization for Standardization, 1991.

ISO/IEC10646

International Organization for Standardization. Information technology ? Universal Multiple-Octet Coded Character Set (UCS) ? Part 1: Architecture and Basic Multilingual Plane, ISO/IEC 10646:1993 (E), International Organization for Standardization, 1993.

JIS TR X 0015

Japanese Industrial Standards Committee. XML Japanese Profile, JIS TR 0015:1999, Japanese Standards Association, May 1999.

JIS X 0201

Japanese Industrial Standards Committee. 7-bit and 8-bit coded character sets for information interchange, JIS X 0201:1997, Japanese Standards Association, 1997.

JIS X 0208

Japanese Industrial Standards Committee. 7-bit and 8-bit double byte coded KANJI sets for information interchange, JIS X 0208:1997, Japanese Standards Association, 1997.

JIS X 0212

Japanese Industrial Standards Committee. Code of the supplementary Japanese graphic character set for information interchange, JIS X0212:1990, Japanese Standards Association, 1990.

JIS X 0221

Japanese Industrial Standards Committee. Universal Multiple-Octet Coded Character Set (UCS) -- Part 1: Architecture and Basic Multilingual Plane, JIS X0221:1995, Japanese Standards Association, 1995.

Unicode 3.0

The Unicode Consortium. The Unicode Standard, Version 3.0. Reading, MA, Addison-Wesley Developers Press, 2000. ISBN 0-201-61633-5.

US-ASCII

American National Standards Institute, Coded character set ? 7-bit American National Standard Code for Information Interchange, ANSI X3.4-1986.

XML

W3C (Worlde Wide Web Consortium). XML (Extensible Markup Language) Recommendation, http://www.w3.org/TR/REC-xml , 1998.

3. Definitions

4. Coded Character Set

5. Character Encoding Schemes

6. Charset Names

Content-Type: text/xml; charset=utf-8

7. XML Constituents in Files for Information Interchange

To store XML constituents in files for information interchange, they should be encoded in either UTF-16 or UTF-8.

In storing XML constituents to files, a BOM or encoding declaration must announce the CES, unless they are encoded in the default CES, i.e., UTF-8.

When encoded in UTF-16, no charset names need to be explicitly specified since the BOM announces UTF-16 as the CES.

Without any external information available for the CES, an XML processor determines the CES from the BOM or encoding declaration. This auto-detection of CESs is described in Appendix F of [XML].

When octet streams representing XML constituents serve as input/output of application programs, such streams should also follow the rules for information interchange files.

8. Delivering XML Constituents by HTTP 1.1

XML servers should use UTF-16 or UTF-8 in delivering XML constituents.

XML documents delivered via HTTP 1.1 ([IETF RFC 2616]) with the media type "text/xml" or "application/xml" should have the correct charset parameter. In the case of "application/xml", the charset parameter may be omitted if the CES is indicated by the BOM or encoding declaration.

The client determines the CES from the charset parameter. The default of the charset parameter for "text/xml" is US-ASCII. When the charset parameter of the media type "application/xml" is omitted, the auto-detection procedure for information interchange files is applied.

9. Delivering XML Constituents via EMAIL or NEWS

To deliver an XML constituent as the EMAIL body, they should be encoded in UTF-16 or UTF-8.

If the charset is UTF-16, use "application/xml" as the media type; otherwise, use "text/xml". In both cases, the charset parameter is STRONGLY RECOMMENDED.

Apply a content transfer encoding (Base64 etc.), if required.

10. Avoiding Conversion Ambiguities

As an example of ambiguous conversion, consider a "Shift_JIS" XML document as below:

<?xml version="1.0" encoding="shift_jis"?>
<doc>{1}{2}{3}{4}{5}
{6}{7}{8}{9}{10}</doc>

where {1} through {10} are used to represent octets as below:

1. 0x5C(YEN SIGN),
2. 0x7E(OVERLINE),
3. 0x815C(FULLWIDTH EM DASH),
4. 0x815F(REVERSE SOLIDUS),
5. 0x8160(WAVE DASH),
6. 0x8161(DOUBLEVERTICAL LINE),
7. 0x817C(MINUS SIGN),
8. 0x8191(CENT SIGN),
9. 0x8192(POUND SIGN), and
10. 0x81CA(NOT SIGN).

Different conversion tables interpret this document differently.

Case 1: x-sjis-jdk1.1.7

Case 2: x-sjis-unicode-0.9 (the third character is HORIZONTAL BAR)

Case 3: x-sjis-jisx0221-1995 (the third character is EM DASH)

Case 4: x-sjis-cp932

We can avoid this ambiguity by using character refererences. For example, the document after normalization in case 3 can be explicitly specified as below:

<?xml version="1.0" encoding="shift_jis"?>
<doc>&#x005C;&#x007E;&#x2015;&#x005C;&#x301C;
&#x2016;&#x2212;&#x00A2;&#x00A3;&#x00AC;</doc>

Alternatively, one can declare and use parsed entities for these character references. The following document captures all Unicode characters ambiguously represented by Shift_JIS.

<?xml version="1.0?>
<!DOCTYPE doc [
  <!ENTITY revsol  "&#x5C;">
  <!ENTITY tilde   "&#x7E;">
  <!ENTITY cent    "&#xA2;">
  <!ENTITY pound   "&#xA3;">
  <!ENTITY yen     "&#xA5;">
  <!ENTITY not     "&#xAC;">
  <!ENTITY mdash   "&#x2014;">
  <!ENTITY horbar  "&#x2015;">
  <!ENTITY dvline  "&#x2016;">
  <!ENTITY fparato "&#x2225;">
  <!ENTITY wdash   "&#x301C;">
  <!ENTITY fminus  "&#xFF0D;">
  <!ENTITY fbsol   "&#xFF3C;">
  <!ENTITY ftilde  "&#xFF5E;">
  <!ENTITY fcent   "&#xFFE0;">
  <!ENTITY fpoun   "&#xFFE1;">
  <!ENTITY fnot    "&#xFFE2;">
  <!ENTITY fmacron "&#xFFE3;">
  <!ENTITY fbrvbar "&#xFFE4;">
  <!ENTITY fyen    "&#xFFE5;">
]>
<doc>&revsol;
&tilde;
&cent;
&pound;
&yen;
&not;
&mdash;
&horbar;
&dvline;
&fparato;
&wdash;
&fminus;
&fbsol;
&ftild;
&fcent;
&fpoun;
&fnot;
&fmacron;
&fbrvbar;
&fyen;</doc>

11. The xml:lang Attribute

Use the xml:lang attribute to indicate the use of the Japanese language for an XML document in whole or in part.

Specify a language code for Japanese ( ja, for example) as the value of this attribute. Values of this attribute are case-insensitive.

An XML document containing a Japanese paragraph and an English paragraph is shown below:

<?xml version="1.0" encoding="utf-8"?>
<document>
  <para xml:lang="ja">これは段落です。</para>
  <para xml:lang="en">This is a paragraph.</para>
</document>