From: http://www.ietf.org/internet-drafts/draft-phillips-langtags-02.txt
Title: Tags for Identifying Languages
Reference: IETF Network Working Group, Internet Draft 'draft-phillips-langtags-02'
Date: April 8, 2004
See also: http://xml.coverpages.org/languageIdentifiers.html#PhillipsLangTag02
http://xml.coverpages.org/draft-phillips-langtags-02a.pdf
--------------------------------------------------------------------------------
Network Working Group A. Phillips, Ed.
Internet-Draft webMethods, Inc.
Expires: October 7, 2004 M. Davis
IBM
April 8, 2004
Tags for Identifying Languages
draft-phillips-langtags-02
Status of this Memo
This document is an Internet-Draft and is in full conformance with
all provisions of Section 10 of RFC2026.
Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF), its areas, and its working groups. Note that other
groups may also distribute working documents as Internet-Drafts.
Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress."
The list of current Internet-Drafts can be accessed at http://
www.ietf.org/ietf/1id-abstracts.txt.
The list of Internet-Draft Shadow Directories can be accessed at
http://www.ietf.org/shadow.html.
This Internet-Draft will expire on October 7, 2004.
Copyright Notice
Copyright (C) The Internet Society (2004). All Rights Reserved.
Abstract
This document describes a language tag for use in cases where it is
desired to indicate the language used in an information object, how
to register values for use in this language tag, and a construct for
matching such language tags, including user defined extensions for
private interchange.
Phillips & Davis Expires October 7, 2004 [Page 1]
Internet-Draft langtags April 2004
Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3
2. The Language Tag . . . . . . . . . . . . . . . . . . . . . . 4
2.1 Syntax . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
2.2 Language Tag Sources . . . . . . . . . . . . . . . . . . . . 5
2.2.1 Pre-Existing RFC3066 Registrations . . . . . . . . . . . . . 10
2.2.2 Possibilities for Registration . . . . . . . . . . . . . . . 10
2.3 Choice of Language Tag . . . . . . . . . . . . . . . . . . . 12
2.4 Meaning of the Language Tag . . . . . . . . . . . . . . . . 15
2.4.1 Language Range . . . . . . . . . . . . . . . . . . . . . . . 16
2.4.2 Matching Language Tags . . . . . . . . . . . . . . . . . . . 16
3. IANA Considerations . . . . . . . . . . . . . . . . . . . . 18
3.1 Registration Procedure for Subtags . . . . . . . . . . . . . 18
4. Security Considerations . . . . . . . . . . . . . . . . . . 21
5. Character Set Considerations . . . . . . . . . . . . . . . . 22
6. Changes from RFC3066 . . . . . . . . . . . . . . . . . . . . 23
References . . . . . . . . . . . . . . . . . . . . . . . . . 25
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . 26
A. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 27
B. Examples of Language Tags (Informative) . . . . . . . . . . 28
Intellectual Property and Copyright Statements . . . . . . . 30
Phillips & Davis Expires October 7, 2004 [Page 2]
Internet-Draft langtags April 2004
1. Introduction
Human beings on our planet have, past and present, used a number of
languages. There are many reasons why one would want to identify the
language used when presenting or requesting information.
Information about a user's language preferences commonly needs to be
identified so that appropriate processing can be applied. For
example, the user's language preferences in a brower can be used to
select web pages appropriately. A choice of language preference can
also be used to select among tools (such as dictionaries) to assist
in the processing or understanding of content in different langauges.
In addition, knowledge about the particular language used by some
piece of information content may be useful or even required by some
types of information processing; for example spell-checking,
computer-synthesized speech, Braille transcription, or high-quality
print renderings.
One means of indicating the language used is by labeling the
information content with a language identifier. These identifiers can
also be used to specify user preferences when selecting information
content, or for labeling additional attributes of content and
associated resources.
These identifiers can also be used to indicate additional attributes
of content that are closely related to the language. In particular,
it is often necessary to indicate specific information about the
dialect, writing system, or orthography used in a document or
resource, as these attributes may be important for the user to obtain
information in a form that they can understand, or important in
selecting appropriate processing resources for the given content.
This document specifies an identifier mechanism, a registration
function for values to be used with that identifier mechanism, and a
construct for matching against those values. It also defines a
mechanism for private use extension and how private use, registered
values, and matching interact.
The keywords "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
document are to be interpreted as described in [RFC 2119] [10].
Phillips & Davis Expires October 7, 2004 [Page 3]
Internet-Draft langtags April 2004
2. The Language Tag
2.1 Syntax
The language tag is composed of one or more parts: A primary language
subtag and a (possibly empty) series of subsequent subtags. Subtags
are distinguished by their length and content, so that each type of
subtag can be recognized solely by these features. This makes it
possible to construct a parser that can extract and assign some
semantic information to the subtags, even if specific subtag values
are not recognized.
The syntax of this tag in ABNF [RFC 2234] [11] is:
= lang *["-s-" extlang] ["-" script] ["-" region] *["-" variant] ["-x" extensions]
=/ "x" extensions ; private use tag
=/ grandfathered-registrations
lang = 2*3 ALPHA ; shortest ISO 639 tag
=/ registered-lang
extlang = 2*15 ALPHA ; additional language subtag
script = 4 ALPHA ; ISO 15924 tag
region = 2 ALPHA ; ISO 3166 tag
=/ 3 DIGIT ; UN country number
variant = 5*15 alphanum
extensions = 1* ("-" value) ; private use extensions
value = 1*32 alphanum
registered-lang = 5*15 alphanum
grandfathered-registrations = ALPHA * (alphanumdash)
alphanum = (ALPHA / DIGIT)
alphanumdash = (alphanum / "-")
Figure 1: Language Tag ABNF
The character "-" is HYPHEN-MINUS (ABNF: %x2D).
The tags and their subtags, including private use extensions, are to
be treated as case insensitive: there exist conventions for the
capitalization of some of them, but these should not be taken to
carry meaning. For instance, [ISO 3166] [4] recommends that country
codes be capitalized (MN Mongolia), while [ISO 639] [3] recommends
that language codes be written in lower case (mn Mongolian). In the
language tags defined by this document, however, the tag 'mn-MN' is
not distinct from 'MN-mn' or 'mN-Mn' (or any other combination) and
each of these variations conveys the same meaning.
For examples of language tags, see Appendix B at the end of this
document.
Phillips & Davis Expires October 7, 2004 [Page 4]
Internet-Draft langtags April 2004
2.2 Language Tag Sources
The namespace of language tags and their subtags is administered by
the Internet Assigned Numbers Authority (IANA) [15] according to the
rules in Section 3.1.
Terminology in this section:
o Tag or tags refers to a complete language tag, such as
'fr-Latn-CA'
o Subtag refers to a specific section of a tag, separated by hyphen,
such as the subtag 'Latn' in 'fr-Latn-CA'
o Code or codes refers to tags defined in external standards (and
which are used as subtags in this document). For example, 'Latn'
is an ISO 15924 [2] script code (which can be used as a script
subtag in a language tag)
The rules in this section apply to the various subtags within the
language tags defined in this document, excepting those
"grandfathered" tags defined in Section 2.2.1.
Note that registered subtags can only appear in specific positions in
a tag. Specifically, they can only occur as primary (language)
subtags, as extended language subtags delmited with the single letter
subtag "s" or as variant subtags.
In addition, private use subtags may only occur at the end of the
sequence of subtags and will not be interspersed with subtags defined
in the rules that follow.
Each subtag type has unique length and content restrictions that make
identification of the subtag's type possible, even if the content of
the subtag itself is unrecognized. This allows tags to be parsed and
processed without reference to the latest version of the underlying
standards or the IANA registry and it makes the associated exception
handling when parsing tags simpler.
Single letter and digit subtags are reserved for current or future
use. These include the following current uses:
o The single letter subtag "i" is reserved for use with
grandfathered IANA registrations that begin with that tag, such as
"i-hakka". Language tags or subtags with the "i" subtag prefix
MUST NOT be registered in the future. That is, the list of tags or
subtags identified by the subtag "i-" will not change in the
future.
Phillips & Davis Expires October 7, 2004 [Page 5]
Internet-Draft langtags April 2004
o The single letter subtag "x" is reserved to introduce a sequence
of private use (or "extension") subtags. No subtags defined by the
rules in this section or in any standard or registry defined in
this document follow the "x" subtag.
o The single letter subtag "s" is reserved to introduce a language
extension subtag. Language extension subtags are currently
reserved for future standardization.
o All other single-character subtags are reserved and many only be
used by revision of this document.
The primary subtag is the first subtag in a language tag and cannot
be empty. Except as noted, the primary subtag is the "language"
subtag. The following rules apply to the assignment and
interpretation of the primary subtag:
o All 2-character subtags are interpreted according to assignments
found in ISO standard 639, "Code for the representation of names
of languages" [ISO639-1] [3], or assignments subsequently made by
the ISO 639 Part 1 maintenance agency or governing standardization
bodies.
o All 3-character subtags are interpreted according to assignments
found in ISO 639 part 2, "Codes for the representation of names of
languages -- Part 2: Alpha-3 code [ISO 639-2] [1]", or assignments
subsequently made by the ISO 639 part 2 maintenance agency or
governing standardization bodies, or assignments of 3-character
disambiguation registrations according to Rule 7a. Ambiguity in
Section 2.3 of this document.
o ISO639-2 reserves for private use codes the range 'qaa' through
'qtz'. These codes should be used for non-registered language
subtags.
o Subtags of 5 to 15 characters may be registered with IANA,
according to the rules in Section 3.1 of this document. (Note that
previously, in rfc3066 [16], the IANA registry contained whole tag
registrations such as 'de-CH-1994', whereas this document refers
to the registration of subtags such as 'tsolyani')
o The single character subtag "x" as the primary subtag indicates
that the whole language tag is a private use tag. The value and
semantic meaning of such a tag as a whole and of the subtags used
within such as tag are undefined by this document.
o The single character subtag "i" as the primary subtag indicates
one of the grandfathered IANA registered tags starting with "i",
Phillips & Davis Expires October 7, 2004 [Page 6]
Internet-Draft langtags April 2004
such as "i-tsolyani"
o Other values shall not be assigned to the primary subtag except by
revision of this document.
The following rules apply to the extended language subtags:
o Each extended language subtag must be prefixed with the single
letter subtag "s". If there are two or more extended language
subtags, each such subtag must be preceded by the subtag "s".
o Extended language subtags are currently reserved for future
standardization.
o Extended language subtags must follow the primary subtag, precede
any other subtags, and there may be more than one extended
language subtag.
o Note: The order of the extended language subtags is important in
some interpretations of language tags. See the section Section
2.4.
o Example: 'zh-s-min-s-nan' would represent the subdialect 'nan' of
the Chinese dialect 'min'.
The following rules apply to the script subtags:
o All 4-character subtags are interpreted as ISO 15924 alpha-4
script codes from [2], or subsequently assigned by the ISO 15924
maintenance agency or governing standardization bodies, denoting
the script or writing system used in conjunction with this
language. These alpha4 tags may only occur following the primary
language subtag and any extended language subtags and must occur
before any other type of subtag described below.
o Example: 'de-Latn' represents German written using the Latin
script.
o ISO 15924 reserves the codes Qaaa-Qacz for private use values.
These codes should be used for non-registered script values.
o Script subtags MUST NOT be registered using the process in Section
3.1 of this document. Variant subtags may be considered for
registration for that purpose.
The following rules apply to the region subtags:
Phillips & Davis Expires October 7, 2004 [Page 7]
Internet-Draft langtags April 2004
o The region subtag must follow any language, extended language, or
script subtags and must precede all other subtags.
o All 2-character subtags following the primary subtag denote the
region or area to which this language variant relates, and are
interpreted according to assignments found in ISO 3166 alpha-2
country codes from [4], assignments subsequently made by the ISO
3166 maintenance agency, or governing standardization bodies.
o All 3-character codes consisting of digit (numeric) characters
denote the region or area to which this language tag relates, and
are interpreted according to the assignments found in UN Standard
Country or Area Codes for Statistical Use [5] or assignments made
thereto by the governing standards body.
o The ISO3166 code MUST be used to form the subtag except for
countries with ambiguous ISO 3166 alpha-2 codes as defined in Rule
7a in Section 2.3.
o If the ISO 3166 alpha-2 code for a country is ambiguous as defined
in Rule 7a, then the UN numeric code MUST be used to form the
region subtag.
o UN numeric codes assigned to 'macro-geographical (continental)' or
sub-regions not associated with an assigned ISO3166 alpha-2 code
MAY be used to form the region subtag.
o UN numeric codes for 'economic groupings' or 'other groupings'
MUST NOT be used to form language tags.
o Note: Generally there will be an informative IANA registration for
valid UN numeric country codes. Continental codes will not be
listed in the IANA registry, but may be used.
o Note: the alphanumeric codes in Appendix X of the UN document must
not be used. (At the time this document was created these values
match the ISO 3166 alpha-2 codes.)
o ISO 3166 reserves the country codes AA, QM-QZ, XA-XZ and ZZ as
user-assigned codes. These codes should be used for private use
region subtags.
o Region subtags MUST NOT be registered using the process in Section
3.1 of this document. Variant subtags may be considered for
registration for this purpose.
o Region subtags must occur after any script subtags and before any
variant subtags or extensions.
Phillips & Davis Expires October 7, 2004 [Page 8]
Internet-Draft langtags April 2004
o Example: 'de-Latn-CH' represents German written using Latin script
for Switzerland.
o Example: 'sr-Latn-891' represents Serbian written using Latin
script for Serbia and Montenegro, whose ISO3166 alpha2 code is
ambiguous.
o Example: 'es-419' represents Spanish as spoken in the UN-defined
'Latin America and Caribbean' region.
The following rules apply to the variant subtags:
o Variant subtags must follow all of the other defined subtags, but
precede any private use extensions.
o Alphanumeric subtags of 5 to 15 characters may be registered with
IANA, according to the rules in Section 3.1 of this document.
o Registered subtags MUST NOT begin with the character 'x', which is
reserved for private use subtags. (Note that previously, in
rfc3066, the IANA registry contained whole tag registrations such
as 'en-boont', whereas this document refers to the registration of
subtags such as 'boont')
o Alphanumeric subtags of 5 to 15 characters starting with 'x' are
reserved for private use. The semantics of these subtags must be
defined by the end users of such subtags and the semantic meaning
should be considered external to this document.
The following rules apply to private use extensions:
o Private Use Extensions subtags are separated from the other
subtags defined in this document by the reserved single-character
subtag "x".
o Private Use Extensions must follow all language, extended
language, script, region, and variant subtags.
o The reserved subtag "x" must be followed by at least one subtag.
o Private Use Extension subtags must consist of one to thirty-two
alphanumeric characters, with each subtag separated by a single
"-".
o No source is defined for private use extensions. Use or
standardization of the private use extension subtags is by private
agreement and should not be considered part of this document.
Phillips & Davis Expires October 7, 2004 [Page 9]
Internet-Draft langtags April 2004
For example: Users who wished to utilize SIL Ethonologue for
identification might agree to exchange tags such as
'az-Arab-x-AZE-derbend'. This example contains two extension subtags.
The first is "AZE" and the second is "derbend".
2.2.1 Pre-Existing RFC3066 Registrations
Existing IANA registered language tags from RFC1766/RFC3066 that are
not defined by additions to this document maintain their validity.
IANA will maintain these tags, adding a notation that they are
"grandfathered from RFC 3066".
The rules governing existing RFC 1766 and RFC 3066 registered tags
are:
o If the formerly registered tag would now be defined by this
document, then the existing tag is marked as superseded by this
document and no subtag will be registered as a result. For
example, 'zh-Hans' is now defined by the addition of ISO 15924
script codes.
o If the registered tag contained one or more subtags that follow
the guidelines for registered language or variant subtags, and all
of the subtags are either now defined by this document or would be
valid to register, then each subtag not already covered by this
document will be registered automatically by IANA without further
review and the existing tag marked as superseded by this document.
For example: the tag 'en-boont' fits the pattern for a registered
variant. The variant subtag "boont" will be registered
automatically and 'en-boont' marked as superseded.
o If the registered tag contains any subtags that are not otherwise
valid for registration according to the rules in this document,
then the tag as a whole is maintained as an exceptional case (that
is, it is "grandfathered"). This includes special cases of Sign
Language tags. For example, the tag 'i-enochian' is not covered by
any addition and is grandfathered, as is 'sgn-BE-fr' (Belgian
French Sign Language).
Users of tags that are grandfathered should consider registering
appropriate subtags using the new format (but are not required to).
2.2.2 Possibilities for Registration
Possibilities for registration of subtags include:
o Languages not listed in ISO 639 that are not variants of any
listed language, can be registered, such as i-mingo. Before
Phillips & Davis Expires October 7, 2004 [Page 10]
Internet-Draft langtags April 2004
attempting to register a language subtag, there should be a good
faith attempt to register the language with ISO 639. No language
subtags will be registered for codes that exist in ISO 639-1 or
ISO 639-2.
o Dialect or other divisions or variations within a language, its
orthography, writing system, regional variation, or historical
usage, such as the "scouse" subtag (the Scouse dialect of
English).
This document leaves the decision on what subtags are appropriate or
not to the registration process described in Section 3.1.
ISO 639 defines a maintenance agency for additions to and changes in
the list of languages in ISO 639. This agency is:
International Information Centre for Terminology (Infoterm)
Aichholzgasse 6/12, AT-1120
Wien, Austria
Phone: +43 1 26 75 35 Ext. 312 Fax: +43 1 216 32 72
ISO 639-2 defines a maintenance agency for additions to and changes
in the list of languages in ISO 639-2. This agency is:
Library of Congress
Network Development and MARC Standards Office
Washington, D.C. 20540 USA
Phone: +1 202 707 6237 Fax: +1 202 707 0115
URL: http://www.loc.gov/standards/iso639
The maintenance agency for ISO 3166 (country codes) is:
ISO 3166 Maintenance Agency
c/o International Organization for Standardization
Case postale 56
CH-1211 Geneva 20 Switzerland
Phone: +41 22 749 72 33 Fax: +41 22 749 73 49
URL: http://www.iso.org/iso/en/prods-services/iso3166ma/index.html
The registration authority for ISO 15924 (script codes) is:
Unicode Consortium Box 391476
Mountain View, CA 94039-1476, USA
URL: http://www.unicode.org/iso15924
The Statistics Division of the United Nations Secretariat maintains
the Standard Country or Area Codes for Statistical Use and can be
reached at:
Phillips & Davis Expires October 7, 2004 [Page 11]
Internet-Draft langtags April 2004
Statistical Services Branch
Statistics Division
United Nations, Room DC2-1620
New York, NY 10017, USA
Fax: +1-212-963-0623
E-mail: statistics@un.org
URL: http://unstats.un.org/unsd/methods/m49/m49alpha.htm
2.3 Choice of Language Tag
One may occasionally be faced with several possible tags for the same
body of text.
Interoperability is best served if all users send the same tag, and
use the same tag for the same language for all documents. If an
application has requirements that make the rules here inapplicable,
the application protocol specification MUST specify how the procedure
varies from the one given here.
The text below is based on the set of tags known to the tagging
entity.
1. Use as precise a tag as possible, but no more specific than is
justified. For example, 'de' might suffice for tagging an email
written in German, while 'de-CH-1996' is probably unnecessarily
precise for such a task.
2. Avoid using subtags that add no distinguishing information about
the content. For example, the script subtag in 'en-Latn-US' is
generally unnecessary, since nearly all English texts are written
in the Latin script.
3. When a language has both an ISO 639-1 2-character code and an ISO
639-2 3-character code, you MUST use the ISO 639-1 2-character
code.
4. When a language has no ISO 639-1 2-character code, and the ISO
639- 2/T (Terminology) code and the ISO 639-2/B (Bibliographic)
codes differ, you MUST use the Terminology code. NOTE: At present
all languages that have both kinds of 3-character code also are
assigned a 2-character code, and the displeasure of developers
about the existence of two different code sets has been
adequately communicated to ISO. So this situation will hopefully
not arise.
5. You SHOULD NOT use the UND (Undetermined) code unless the
protocol in use forces you to give a value for the language tag,
Phillips & Davis Expires October 7, 2004 [Page 12]
Internet-Draft langtags April 2004
even if the language is unknown. Omitting the tag is preferred.
6. You SHOULD NOT use the MUL (Multiple) tag if the protocol allows
you to use multiple languages, as is the case for the
Content-Language header in HTTP.
NOTE: In order to avoid versioning difficulties in
applications such as that experienced in RFC 1766 [8], the ISO
639 Registration Authority Joint Advisory Committee (RA-JAC)
has agreed on the following policy statement:
"After the publication of ISO/DIS 639-1 as an International
Standard, no new 2-letter code shall be added to ISO 639-1
unless a 3-letter code is also added at the same time to ISO
639-2. In addition, no language with a 3-letter code available
at the time of publication of ISO 639-1 which at that time had
no 2-letter code shall be subsequently given a 2-letter code."
This will ensure that, for example, a user who implements
"haw" (Hawaiian), which currently has no 2-character code,
will not find his or her data invalidated by eventual addition
of a 2-character code for that language."
7. To maintain backwards compatibility, there are two provisions to
account for potential instability in ISO 639, 3166, and 15924
codes.
a) Ambiguity. Beginning with these standards as of 1 January,
2003, in the event that ISO639, ISO3166, or ISO15924 assigns a
code a new meaning or reassigns a deprecated code, the new use
of the code is not permitted in language tags defined by this
document.
In the event that either ISO 639 or ISO 15924 assigns a new
meaning to an existing code, the language subtag reviewer, as
described in Section 3, shall prepare a proposal for entering
in the IANA registry as soon as practical a variant or
registered language subtag as a surrogate value for the new
code. The form of the registered language subtag or variant
subtag will be at the discretion of the language subtag
reviewer and must conform to other restrictions on language or
variant subtags in this document.
In the event ISO 3166 assigns a new meaning to an existing
code, then the language subtag reviewer, as described in
Section 3, shall prepare a proposal for entering the
Phillips & Davis Expires October 7, 2004 [Page 13]
Internet-Draft langtags April 2004
appropriate numeric UN country code as an informative entry in
the IANA registry.
The normal registration process described in Section 3.1 of
this document applies to the review and registration of the
registered subtags described above. Note that these subtags
should never be used in combination with the subtag type for
which they are a surrogate. For example, a "region" variant
subtag should not be used with a region subtag.
For example:
cs-CS (Czech for Czechoslovakia)
sr-891 (Serbian for Serbia and Montenegro, using the UN
country code)
qx-Latn (hypothetical reassigned value 'qx')
qx2003-Latn (hypothetical registered language subtag)
b) Stability. All other ISO codes are valid, even if they have
been deprecated. Some examples, current at the time this
document was drafted, are listed below. Where a new equivalent
code has been defined (given below on the right side after a
tilde), implementations should treat these tags as identical.
For example, some deprecated ISO 639 codes:
iw ~ he
in ~ id
ji ~ yi
sh
For example, some deprecated ISO 3166 codes:
FX
TP ~ TL
YU
Phillips & Davis Expires October 7, 2004 [Page 14]
Internet-Draft langtags April 2004
2.4 Meaning of the Language Tag
The language tag always defines a language as spoken (or written,
signed or otherwise signaled) by human beings for communication of
information to other human beings. Computer languages such as
programming languages are explicitly excluded.
If a language tag B contains language tag A as a prefix, then B is
typically "narrower" or "more specific" than A. For example,
'zh-Hant-TW' is more specific than 'zh-Hant'.
This relationship is not guaranteed in all cases: specifically,
languages that begin with the same sequence of subtags are NOT
guaranteed to be mutually intelligible, although they may be. For
example, the tag 'az' shares a prefix with both 'az-Latn'
(Azerbaijani written using the Latin script) and 'az-Cyrl'
(Azerbaijani written using the Cyrillic script). A person fluent in
one script may not be able to read the other, even though the text
might be identical. Content tagged as 'az' most probably is written
in just one script and thus might not be intelligible to a reader
familiar with the other script.
The relationship between the tag and the information it relates to is
defined by the standard describing the context in which it appears.
Accordingly, this section can only give possible examples of its
usage.
o For a single information object, it could be taken as the set of
languages that is required for a complete comprehension of the
complete object. Example: Plain text documents.
o For an aggregation of information objects, it should be taken as
the set of languages used inside components of that aggregation.
Examples: Document stores and libraries.
o For information objects whose purpose is to provide alternatives,
the set of tags associated with it should be regarded as a hint
that the content is provided in several languages, and that one
has to inspect each of the alternatives in order to find its
language or languages. In this case, a tag with multiple languages
does not mean that one needs to be multi-lingual to get complete
understanding of the document. Example: MIME multipart/
alternative.
o In markup languages, such as HTML and XML, language information
can be added to each part of the document identified by the markup
structure (including the whole document itself). For example, one
could write C'est la vie. inside a
Phillips & Davis Expires October 7, 2004 [Page 15]
Internet-Draft langtags April 2004
Norwegian document; the Norwegian-speaking user could then access
a French-Norwegian dictionary to find out what the marked section
meant. If the user were listening to that document through a
speech synthesis interface, this formation could be used to signal
the synthesizer to appropriately apply French text-to-speech
pronunciation rules to that span of text, instead of misapplying
the Norwegian rules.
2.4.1 Language Range
A Language Range is a set of languages whose tags all begin with the
same sequence of subtags. The following definition of language-range
is derived from HTTP/1.1 [14].
language-range = language-tag / "*"
That is, a language-range has the same syntax as a language-tag, or
is the single character "*" and implicitly assumes that there is a
semantic relationship between tags that share the same subtag
prefixes.
A language-range matches a language-tag if it exactly equals the tag,
or if it exactly equals a prefix of the tag such that the first
character following the prefix is "-".
The special range "*" matches any tag. A protocol which uses language
ranges may specify additional rules about the semantics of "*"; for
instance, HTTP/1.1 specifies that the range "*" matches only
languages not matched by any other range within an "Accept-Language:"
header.
As noted above, not all languages or content denoted by a specific
language-range may be mutually intelligible and this use of a prefix
matching rule does not imply that language tags are assigned to
languages in such a way that it is always true that if a user
understands a language with a certain tag, then this user will also
understand all languages with tags for which this tag is a prefix.
The prefix rule simply allows the use of prefix tags if this is the
case.
2.4.2 Matching Language Tags
Implementations that are searching for content or otherwise matching
language tags to a language-range [Section 2.4.1] may choose to
assume that there is a semantic relationship between two tags that
share common prefixes. This is called 'language tag fallback'. The
most common implementations follow this pattern:
Phillips & Davis Expires October 7, 2004 [Page 16]
Internet-Draft langtags April 2004
1. When searching for content using language tag fallback, the
language tag is progressively truncated from the end until a
match is located. For example, starting with the tag
'en-US-boont', searchs or matches would first be performed with
the whole tag, then with 'en-US', and finally with 'en'. This
allows some flexibility in finding content in accordance with
Rules 1 and 2 in Section 2.3; allows better maintenance; and
usually provides better results when data is not available at a
specific level of tag granularity or is sparsely populated (than
if the default language for the system or content were used). Any
implementation that uses this technique should ensure that
appropriate data is available on each level.
Tag to match: en-US-boont
1. en-US-boont
2. en-US
3. en
Figure 2: Default Fallback Pattern Example
2. Private Use Extensions are orthogonal to language tag fallback.
By default, implementations should ignore private use extensions
and follow the default fallback pattern (above). Thus matching
the tag "en-US-boont-x-traditional" would be exactly the same as
the example above.
3. Implementations that choose to interpret one or more private use
extension subtags can choose a different fallback pattern or use
the private use extensions to interpret content in a different
fashion.
Phillips & Davis Expires October 7, 2004 [Page 17]
Internet-Draft langtags April 2004
3. IANA Considerations
This section deals with the registration of subtags for use in
language tags defined by this document, in accordance with the
requirements of RFC2434 [13].
3.1 Registration Procedure for Subtags
The procedure given here MUST be used by anyone who wants to use a
subtag not given an interpretation in Section 2.2 of this document or
previously registered with IANA.
This procedure MAY also be used to register information with the IANA
about a tag or subtag defined by this document, for instance if one
wishes to make publicly available a reference to the definition for a
language such as sgn-US (American Sign Language), or additional
information about a registration previously made via this procedure.
Variant subtags MUST NOT be registered using the pattern 2 ALPHA *
DIGIT to accommodate the provisions in Section 2.3, rule 7a of this
document. That is, the subtag yx1234 can NOT be registered except
under the aforementioned provisions.
Extended language subtags that are registered MUST be at least 4
characters long.
Subtags MUST NOT be registered that start with the letter 'x', since
this prefix is reserved for Private Use subtags.
The process starts by filling out the registration form reproduced
below.
LANGUAGE SUBTAG REGISTRATION FORM
Name of requester:
E-mail address of requester:
Subtag to be registered:
Type of Subtag:
[ ] language
[ ] extended language
[ ] variant
[ ] region (informative; for use by language subtag reviewer only)
Full English name of subtag:
Intended meaning of the subtag:
If variant subtag, the intended prefix(es) of subtag:
If extended language subtag, the intended prefix(es) of subtag:
Native name of language (transcribed into ASCII):
Reference to published description of the language (book or article):
Any other relevant information:
Phillips & Davis Expires October 7, 2004 [Page 18]
Internet-Draft langtags April 2004
Figure 3
The subtag registration form MUST be sent to
for a two week review period before it can
be submitted to IANA. (This is an open list. Requests to be added
should be sent to .)
Variant subtags are generally registered for use with a particular
prefix or set of prefixes. For example, the subtag 'boont' is
intended for use with the prefix 'en-', since Boontling is a dialect
of English. This information MUST be provided in the registration
form.
Any registered subtag MAY be incorporated into a variety of language
tags, according to the rules of Section 2.1. This makes validation
simpler and thus more uniform across implementations, and does not
require new registrations for different intended prefixes.
However, the intended prefixes for a given registered subtag will be
maintained in the IANA registry as a guide to usage. If it is
necessary to add an additional intended prefix to that list for an
existing language tag, that can be done by filing an additional
registration form. In that form, the "Any other relevant information:
" field should indicate that it is the addition of an additional
intended prefix.
When the two week period has passed, the subtag reviewer, who is
appointed by the IETF Applications Area Director, either forwards the
request to IANA@IANA.ORG, or rejects it because of significant
objections raised on the list. Note that the reviewer can raise
objections on the list himself, if he or she so desires. The
important thing is that the objection must be made publicly.
The applicant is free to modify a rejected application with
additional information and submit it again; this restarts the two
week comment period.
Decisions made by the reviewer may be appealed to the IESG [RFC 2028]
[9] under the same rules as other IETF decisions [RFC 2026] [18]. All
registered forms are available online in the directory http://
www.iana.org/numbers.html under "languages".
Updates of registrations follow the same procedure as registrations.
The subtag reviewer decides whether to allow a new registrant to
update a registration made by someone else; normally objections by
the original registrant would carry extra weight in such a decision.
Registrations are permanent and stable. When some registered subtag
Phillips & Davis Expires October 7, 2004 [Page 19]
Internet-Draft langtags April 2004
should not be used any more, for instance because a corresponding ISO
639 code has been created, the registration should be amended by
adding a remark like "DEPRECATED: use instead" to the
"other relevant information" section.
Note: The purpose of the "published description" is intended as an
aid to people trying to verify whether a language is registered, or
what language a particular subtag refers to. In most cases, reference
to an authoritative grammar or dictionary of that language will be
useful; in cases where no such work exists, other well known works
describing that language or in that language may be appropriate. The
subtag reviewer decides what constitutes "good enough" reference
material.
Phillips & Davis Expires October 7, 2004 [Page 20]
Internet-Draft langtags April 2004
4. Security Considerations
The only security issue that has been raised with language tags since
the publication of RFC 1766, which stated that "Security issues are
believed to be irrelevant to this memo", is a concern with language
ranges used in content negotiation - that they may be used to infer
the nationality of the sender, and thus identify potential targets
for surveillance.
This is a special case of the general problem that anything you send
is visible to the receiving party. It is useful to be aware that such
concerns can exist in some cases.
The evaluation of the exact magnitude of the threat, and any possible
countermeasures, is left to each application protocol.
Phillips & Davis Expires October 7, 2004 [Page 21]
Internet-Draft langtags April 2004
5. Character Set Considerations
Language tags may always be presented using the characters A-Z, a-z,
0-9, and HYPHEN-MINUS, which are present in most character sets, so
presentation of language tags should not have any character set
issues.
The issue of deciding upon the rendering of a character set based on
the language tag is not addressed in this memo; however, it is
thought impossible to make such a decision correctly for all cases
unless means of switching language in the middle of a text are
defined (for example, a rendering engine that decides font based on
Japanese or Chinese language may produce sub-optimal output when a
mixed Japanese- Chinese text is encountered)
Phillips & Davis Expires October 7, 2004 [Page 22]
Internet-Draft langtags April 2004
6. Changes from RFC3066
The main goals were to maintain backward compatibility (so that all
previous codes would remain valid); reduce the need for large numbers
of registrations; to provide a more formal structure to allow parsing
into subtags even where software does not have the latest
registrations; to provide stability in the face of potential
instability in ISO 639, 3166, and 15924 codes (*demonstrated
instability* in the case of ISO 3166); and to allow for external
extension mechanisms.
o Allows ISO15924 script code subtags and allows them to be used
generatively.
o Adds the concept of a variant subtag and allows variants to be
used generatively.
o Adds an extension mechanism which does not require registration to
use.
o Defines the private use tags in ISO639, ISO15924, and ISO3166 as
the mechanism for creating private use language, script, and
region subtags respectively
o Defines a syntax for private use variant subtags which can be used
without registration.
o Defines a process for handling reuse of values by ISO639,
ISO15924, and ISO3166 in the event that they register a previously
used value for a new purpose.
o Changes the IANA language tag registry to a language subtag
registry
Substantive changes between draft-01 and this version are:
Added a reference to the most recent version of the UN country IDs
to the address information in section 2.2.2.
Removed references to the 'i-klingon' tags (previously used as
examples) since that tag is now deprecated (due to the addition of
the ISO639-2 tag 'tlh').
Made the choice of UN or ISO3166 codes explicit in Section 2.2 and
modified the text of Rule 7a. It also sets a start date for
ambiguity resolution.
Prohibited future registration of "i-" prefixed tags or subtags.
Phillips & Davis Expires October 7, 2004 [Page 23]
Internet-Draft langtags April 2004
Extensive non-substantive edits were made to the text to clarify
positioning and make the rules for subtag assignment clearer.
Phillips & Davis Expires October 7, 2004 [Page 24]
Internet-Draft langtags April 2004
References
[1] International Organization for Standardization, "ISO 639-2:1998
- Codes for the representation of names of languages -- Part 2:
Alpha-3 code - edition 1", August 1988.
[2] ISO TC46/WG3 and M. Everson, Ed., "ISO 15924:2003 (E/F) - Codes
for the representation of names of scripts", March 2003.
[3] International Organization for Standardization, "Code for the
representation of names of languages, 1st edition", ISO
Standard 639, 1988.
[4] International Organization for Standardization, "Codes for the
representation of names of countries, 3rd edition", ISO
Standard 3166, August 1988.
[5] Statistical Division, United Nations, "Standard Country or Area
Codes for Statistical Use", UN Standard Country or Area Codes
for Statistical Use, Revision 4 (United Nations publication,
Sales No. 98.XVII.9, June 1999.
[6] Hardcastle-Kille, S., "Mapping between X.400(1988) / ISO 10021
and RFC 822", RFC 1327, May 1992.
[7] Borenstein, N. and N. Freed, "MIME (Multipurpose Internet Mail
Extensions) Part One: Mechanisms for Specifying and Describing
the Format of Internet Message Bodies", RFC 1521, September
1993.
[8] Alvestrand, H., "Tags for the Identification of Languages", RFC
1766, March 1995.
[9] Hovey, R. and S. Bradner, "The Organizations Involved in the
IETF Standards Process", BCP 11, RFC 2028, October 1996.
[10] Bradner, S., "Key words for use in RFCs to Indicate Requirement
Levels", BCP 14, RFC 2119, March 1997.
[11] Crocker, D. and P. Overell, "Augmented BNF for Syntax
Specifications: ABNF", RFC 2234, November 1997.
[12] Berners-Lee, T., Fielding, R. and L. Masinter, "Uniform
Resource Identifiers (URI): Generic Syntax", RFC 2396, August
1998.
[13] Narten, T. and H. Alvestrand, "Guidelines for Writing an IANA
Considerations Section in RFCs", BCP 26, RFC 2434, October
Phillips & Davis Expires October 7, 2004 [Page 25]
Internet-Draft langtags April 2004
1998.
[14] Fielding, R., Gettys, J., Mogul, J., Frystyk, H., Masinter, L.,
Leach, P. and T. Berners-Lee, "Hypertext Transfer Protocol --
HTTP/1.1", RFC 2616, June 1999.
[15] Carpenter, B., Baker, F. and M. Roberts, "Memorandum of
Understanding Concerning the Technical Work of the Internet
Assigned Numbers Authority", RFC 2860, June 2000.
[16] Alvestrand, H., "Tags for the Identification of Languages", BCP
47, RFC 3066, January 2001.
[17] Yergeau, F., "UTF-8, a transformation format of ISO 10646", STD
63, RFC 3629, November 2003.
[18]
Authors' Addresses
Addison Phillips (editor)
webMethods, Inc.
432 Lakeside Drive
Sunnyvale, CA 94088
US
EMail: aphillips@webmethods.com
Mark Davis
IBM
EMail: mark.davis@us.ibm.com
Phillips & Davis Expires October 7, 2004 [Page 26]
Internet-Draft langtags April 2004
Appendix A. Acknowledgements
Any list of contributors is bound to be incomplete; please regard the
following as only a selection from the group of people who have
contributed to make this document what it is today.
The contributors to RFC 3066 and RFC 1766, the precursors of this
document, made enormous contributions directly or indirectly to this
document and are generally responsible for the success of language
tags.
The following people (in alphabetical order) contributed to this
document or to RFCs 1766 and 3066:
Glenn Adams, Harald Tveit Alvestrand, Tim Berners-Lee, Marc Blanchet,
Nathaniel Borenstein, Eric Brunner, Sean M. Burke, Jeremy Carroll,
John Clews, Jim Conklin, Peter Constable, John Cowan, Mark Crispin,
Dave Crocker, Martin Duerst, Michael Everson, Doug Ewell, Ned Freed,
Tim Goodwin, Dirk-Willem van Gulik, Marion Gunn, Paul Hoffman,
Richard Ishida, Olle Jarnefors, Kent Karlsson, John Klensin, Alain
LaBonte, Eric Mader, Keith Moore, Chris Newman, Masataka Ohta, George
Rhoten, Markus Scherer, Keld Jorn Simonsen, Thierry Sourbier, Otto
Stolz, Tex Texin, Rhys Weatherley, Misha Wolf, Francois Yergeau and
many, many others.
Very special thanks must go to Harald Tveit Alvestrand, who
originated RFCs 1766 and 3066, and without whom this document would
not have been possible. Special thanks must go to Michael Everson,
who has served as language tag reviewer for almost the complete
period since the publication of RFC 1766.
Phillips & Davis Expires October 7, 2004 [Page 27]
Internet-Draft langtags April 2004
Appendix B. Examples of Language Tags (Informative)
Simple language code:
de (German)
fr (French)
ja (Japanese)
Language code plus Script code :
zh-Hant (Traditional Chinese)
en-Latn (English written in Latin script)
sr-Cyrl (Serbian written with Cyrillic script)
Language-Script-Region:
zh-Hans-CN (Simplified Chinese for the PRC)
sr-Latn-891 (Serbian, Latin script, Serbia and Montenegro)
Language-Script-Region-Variant:
en-Latn-US-boont (Boontling dialect of English)
Language-Region:
de-DE (German for Germany)
zh-SG (Chinese for Singapore)
cs-CS (Czech for Czechoslovakia)
sr-891 (Serbian for Serbia and Montenegro, UN country code, see 7a
in Section 2.3
Other Mixtures:
zh-CN (Chinese for the PRC)
en-boont (Boontling dialect of English)
Extension mechanism:
Phillips & Davis Expires October 7, 2004 [Page 28]
Internet-Draft langtags April 2004
de-CH-x-phonebook
az-Arab-x-AZE-derbend
Extended language subtags:
zh-s-min
zh-s-min-s-nan-Hant-CN
Private Use tags:
qaa-Qaaa-QM-xsouthern (all private tags)
de-Qaaa (German, with a private script)
de-Latn-QM (German, Latin-script, private region)
de-Qaaa-DE (German, private script, for Germany)
Some Invalid Tags:
de-891-DE (two region tags)
a-DE (use of a single character tag)
zh-xsouthern-DE (private use variant followed by another tag)
Phillips & Davis Expires October 7, 2004 [Page 29]
Internet-Draft langtags April 2004
Intellectual Property Statement
The IETF takes no position regarding the validity or scope of any
intellectual property or other rights that might be claimed to
pertain to the implementation or use of the technology described in
this document or the extent to which any license under such rights
might or might not be available; neither does it represent that it
has made any effort to identify any such rights. Information on the
IETF's procedures with respect to rights in standards-track and
standards-related documentation can be found in BCP-11. Copies of
claims of rights made available for publication and any assurances of
licenses to be made available, or the result of an attempt made to
obtain a general license or permission for the use of such
proprietary rights by implementors or users of this specification can
be obtained from the IETF Secretariat.
The IETF invites any interested party to bring to its attention any
copyrights, patents or patent applications, or other proprietary
rights which may cover technology that may be required to practice
this standard. Please address the information to the IETF Executive
Director.
Full Copyright Statement
Copyright (C) The Internet Society (2004). All Rights Reserved.
This document and translations of it may be copied and furnished to
others, and derivative works that comment on or otherwise explain it
or assist in its implementation may be prepared, copied, published
and distributed, in whole or in part, without restriction of any
kind, provided that the above copyright notice and this paragraph are
included on all such copies and derivative works. However, this
document itself may not be modified in any way, such as by removing
the copyright notice or references to the Internet Society or other
Internet organizations, except as needed for the purpose of
developing Internet standards in which case the procedures for
copyrights defined in the Internet Standards process must be
followed, or as required to translate it into languages other than
English.
The limited permissions granted above are perpetual and will not be
revoked by the Internet Society or its successors or assignees.
This document and the information contained herein is provided on an
"AS IS" basis and THE INTERNET SOCIETY AND THE INTERNET ENGINEERING
TASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING
BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION
Phillips & Davis Expires October 7, 2004 [Page 30]
Internet-Draft langtags April 2004
HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF
MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.
Acknowledgment
Funding for the RFC Editor function is currently provided by the
Internet Society.
Phillips & Davis Expires October 7, 2004 [Page 31]