TMX Format

Specifications

Version 1.3 - August-29-2001

Summary of purpose

This document describes the TMX file format. TMX stands for Translation Memory eXchange. OSCAR (Open Standards for Container/Content Allowing Re-use) is the LISA Special Interest Group responsible for its definition.

Contents

• 1. Overview
• 2. Specifications
  • 2.1. SGML/XML Compliance
  • 2.2. Code Sets
  • 2.3. Element Definitions
  • 2.4. Attribute Definitions
  • 2.5. Recommended Values for Attributes
  • 2.6. Content Mark-Up
• 3. TMX Sample File
• 4. Glossary
• 5. References

See also:

• Document Type Definition for TMX version 1.3
• TMX Implementation Notes
  
  For reference:
• Document Type Definition for TMX version 1.2
• Document Type Definition for TMX version 1.1

1. Overview

The purpose of the TMX format is to provide a standard method to describe translation memory data that is being exchanged among tools and/or translation vendors, while introducing little or no loss of critical data during the process.

TMX is defined in two parts:

• A specification of the format of the container (the higher-level elements that provide information about the file as a whole and about entries). In TMX, an entry consisting of aligned segments of text in two or more languages is called a Translation Unit (the <tu> element).
• A specification of a low-level meta-markup format for the content of a segment of translation-memory text. In TMX, an individual segment of translation-memory text in a particular language is denoted by a <seg> element. See the section Content Markup for more details.

TMX can be implemented on three levels:

• Level 1 (Plain Text Only) - Support for the container only. The data inside each <seg> element is plain text, without Content Markup. This level is enough when the data do not have in-line codes, for example software messages. It is usually not sufficient for documentation-type formats
• Level 2 (Meta-Markup) - Support for both container and content. The application uses the TMX Content Markup, but ignores the native codes inside the in-line tags. This level is usually enough for most purposes.
• Level 3 - (Native-Markup) Support for both container and content. The application is able to use the native codes inside the Content Markup elements.

2. Specifications

2.1. SGML/XML Compliance

TMX is XML-compliant. It also uses various ISO standards for date/time, language codes, and country codes. (See References section.)

TMX files are intended to be created automatically by export routines and processed automatically by import routines. TMX files are "well-formed" XML documents that can be processed without explicit reference to the TMX DTD. However, a "valid" TMX file must conform to the TMX DTD, and any suspicious TMX file should be verified against the TMX DTD using a validating XML parser.

Since XML syntax is case sensitive, any XML application must define casing conventions. All elements and attributes names of TMX are defined in lower-case.

The TMX namespace is defined as http://www.lisa.org/tmx. For example, if you want to use TMX fragments in another XML document you document will look something like:

<?xml version="1.0" ?>
<myformat>
 <data>
 <tmx xmlns="http://www.lisa.org/tmx">
 ... TMX data
 </tmx> 
 </data>
</myformat>

2.2. Code Sets

TMX files are always in Unicode. They can use either of three encoding methods: UCS-2 (16-bit files), UTF-8 (8-bit files) or ISO-646 [US-ASCII] (7-bit files). In both cases only the following five character entities are allowed: &amp; (&), &lt; (<), &gt; (>), &apos; ('), and &quot; ("). For 7-bit files, extended (non-ASCII) characters are always represented by numeric character references using the Unicode hexadecimal values (e.g. &#x0396; for a GREEK CAPITAL LETTER DELTA).

Since all XML processors must accept the UTF-8 and UTF-16 encodings and since US-ASCII and UCS-2 encoding methods are, respectively, sub-sets of UTF-8 and UTF-16, a TMX document can omit the encoding declaration in the XML declaration.

Note that UCS-2 files always start with a Unicode byte-order-mark value, the ZERO WIDTH NO-BREAK SPACE 0xFEFF.

2.3. Element Definitions

The following table lists the different elements of a TMX document (Container):

The following table lists the different elements of a TMX document (Content):

2.4. Attribute Definitions

The following table lists the different attributes used in the elements of a TMX document. The same attribute may be used with multiple elements, but will be either mandatory or optional depending on the specific occurrence.

2.5. Recommended Values for Attributes

By using standard values for attributes, the TMX format can minimize the amount of data lost during the exchange process. However, this dynamic nature of the diverse collection of data that needs to be captured doesn't lend itself to being part of the TMX format specification. By specifying the recommended values for attributes in the accompanying Implementation Notes, developers of translation memory tools can update this information on an on-going basis without initiating a revision of the TMX format specification itself.

The TMX specification strongly recommends that developers of TMX-aware tools use only recommended attribute values when writing TMX data in order to ensure full TMX compliance. The Implementation Notes document specifies recommended values for the datatype and type attributes.

2.6. Content Mark-Up

Each TM system uses a different method of marking up the formatting. Formats are constantly evolving, and new formats will be introduced on a regular basis. Attempting to collect, interpret, disseminate and maintain finite descriptions of each formatting tag used at any given time by any of the TM systems is not possible.

The best way to deal with these native codes is to delimit them by a specific set of elements that convey where they begin and end, and possibly additional information about what they are (bold, italic, footnote, etc.).

Native codes can be grouped into four categories:

1. Codes that either begin or end an instruction, and whose beginning and ending functions both appear within a single segment. For example, an instruction to begin embolden for a range of words which is then followed in the same segment by an instruction to end bold formatting.
2. Codes that either begin or end an instruction, but whose beginning and ending functions are not both contained within a single segment. For example, an instruction to embolden text may apply to the first three sentences in a paragraph, but the instruction to turn off bolding may only appear at the end of the third sentence. Its beginning instruction is present in the first segment, while its closing tag is present in the third segment.
3. Codes that represent self-contained functions that don't require explicit ending instructions. An image or cross-reference token are examples of these standalone codes.
4. Codes that have unknown behavior.

Respectively, the TMX vocabulary provides elements to mark up each category of native code sequences:

1. The <bpt> and <ept> elements demark paired sequences of native code which begin and end in the same <seg> element.
2. The <it> element demarks a paired native code that is isolated from its partner, possibly due to segmentation.
3. The <ph> element demarks a stand-alone native code.
4. The <ut> element demarks a native code that cannot be identified by the TMX processor.

An additional element (<sub>) is provided to delimit sub-flow text within a sequence of native codes. For example, if the text content of a footnote is defined within the footnote marker code, it may be demarked with the <sub> element.

For example:

Without Content mark-up tags:
<seg>Text in {\i italics}.</seg>

With Content mark-up tags (content markup in bold red):
<seg>Text in <bpt type="italic">{\i </bpt>italics<ept>}</ept>.</seg>

Such a mechanism allows tools to perform matching at several levels:

1. Ignoring the codes. Since the code parts of the data stored in the segment are well delimited with the TMX elements, the tools can simply ignore them. It is equivalent to working at a plain text level. This solution may be the least efficient, but it does allow better matching results between units than when the codes get in the way.
   
   
2. Matching the content mark-up tags. A second and more efficient way to proceed is to recognize the TMX content mark-up elements and use them as part of the matching criteria. This additional step gives you better accuracy.
   
   
3. Matching the native codes inside the content mark-up tags. If the tool is sophisticated enough, it can go to a deeper and more accurate matching algorithm by also looking at the data the content mark-up elements delimit, and therefore get complete coverage for its matching algorithm.

For example, here are four segments differing only by the formatting codes:

Plain text:  Special text   
    RTF v1:  {\b Special} text
    RTF v2:  {\cf7 Special} text
      HTML:  <B>Special</B> text

The same samples with the TMX content mark-up tags:

Plain text:  <seg>Special text</seg>   
    RTF v1:  <seg><bpt type="bold">{\b </bpt>Special<ept>}</ept> text</seg>
    RTF v2:  <seg><bpt>{\cf7 </bpt>Special<ept>}</ept> text</seg>
      HTML:  <seg><bpt type="bold">&lt;B></bpt>Special<ept>&lt;/B></ept> text</seg>

• Native codes (RTF, HTML, etc.) need not be parsed before using the segments; the TMX elements allow you to make the distinction between code and text.
• Comparisons across file formats can be achieved, and if there is enough information (like for the previous bolding example) leverage of native code of source into target can be performed. This not only leverages the translated text but also the correct formatting, even if it was originally in a different source file format.

The datatype attribute is used to specify the kind of native code the data contains.

Matching codes between <tuv> elements (x attribute)

TMX implements a mechanism to help you match codes between source and target text. The x attribute in the <bpt>, <it>, <ut> and <ph> elements allows you to pair codes between two <tuv> elements (even if they are not in the same order any more because of the translation syntax). For example:

<seg>The <bpt x="1">{\b </bpt>
black<ept>}</ept><bpt x="2">{\i </bpt> cat<ept>}</ept> sleeps.</seg>

<seg>Le<bpt x="2">{\i </bpt> chat<ept>}</ept> <bpt x="1">{\b </bpt>
noir<ept>}</ept> dort.</seg>

Overlapping codes (i attribute)

TMX provides a way to deal with overlapping tags. Such constructions are not used often, however several formats allow them. For example, the following HTML segment, even if not strictly legal, is accepted by some HTML editors and usually interpreted correctly by the browsers.

HTML:
<B>Bold,
<I>Bold+Italic</B>,
Italic</I>

TMX (without content mark-up):
<seg>&lt;B>Bold,
&lt;I>Bold+Italic&lt;/B>,
Italic&lt;/I></seg>

With the TMX content mark-up, since the <ept> element does not necessarily have a type, it can be difficult to know which sequence of codes it closes as illustrated by the following segment:

TMX (with basic content mark-up):
<seg><bpt>&lt;B></bpt>Bold,
<bpt>&lt;I></bpt>Bold+Italic<ept>&lt;/B></ept>,
Italic<ept>&lt;/I></ept></seg>

The attribute i is used to specify which <ept> is closing which <bpt>.

TMX (with correct content mark-up):
<seg><bpt i="1">&lt;B></bpt>Bold,
<bpt i="2">&lt;I></bpt>Bold+Italic<ept ="1">&lt;/B></ept>,
Italic<ept i="2">&lt;/I></ept></seg>

See the Implementation Notes for more details.

3. TMX Sample File

Notational conventions: The restrictions on the number of occurrences of each element and whether an attribute is mandatory within an element are indicated by:

• BOLD for the items that are mandatory.
• ITALIC for the items that can be specified zero or one times.
• NORMAL for the items that can be specified zero, one or more times.

This is an example of a TMX file. (The indentations are only there for ease of reading). Different types of notation are mixed to illustrate the various possibilities.

<?xml version="1.0" ?>
<!DOCTYPE tmx SYSTEM "tmx13.dtd">
<!-- Example of TMX document -->
<tmx version="1.3">
   <header
      creationtool="XYZTool"
      creationtoolversion="1.01-023"
      datatype="PlainText"
      segtype="sentence"
      adminlang="en-us"
      srclang="EN"
      o-tmf="ABCTransMem"
      creationdate="19970101T163812Z"
      creationid="ThomasJ"
      changedate="19970314T023401Z"
      changeid="Amity"
      o-encoding="iso-8859-1"
   >
      <note>This is a note at document level.</note>
      <prop type="RTFPreamble">{\rtf1\ansi\tag etc...{\fonttbl}</prop>
      <ude name="MacRoman" base="Macintosh">
         <map unicode="#xF8FF" code="#xF0" ent="Apple_logo" subst="[Apple]"/>
      </ude>
   </header>

   <body>
      <tu
         tuid="0001"
         datatype="Text"
         usagecount="2"
         lastusagedate="19970314T023401Z"
      >
         <note>Text of a note at the TU level.</note>
         <prop type="x-Domain">Computing</prop>
         <prop type="x-Project">P&#x00E6;gasus</prop>
         <tuv
            xml:lang="EN"
            creationdate="19970212T153400Z"
            creationid="BobW"
         >
            <seg>data (with a non-standard character: &#xF8FF;).</seg>
         </tuv>
         <tuv
            xml:lang="FR-CA"
            creationdate="19970309T021145Z"
            creationid="BobW"
            changedate="19970314T023401Z"
            changeid="ManonD"
         >
            <prop type="Origin">MT</prop>
            <seg>donn&#xE9;es (avec un caract&#x00E8;re non standard: &#xF8FF;).</seg>
         </tuv>
      </tu>
      <tu tuid="0002" srclang="*all*">
         <prop type="Domain">Cooking</prop>
         <tuv xml:lang="EN">
            <seg>menu</seg>
         </tuv>
         <tuv xml:lang="FR-CA">
            <seg>menu</seg>
         </tuv>
         <tuv xml:lang="FR-FR">
            <seg>menu</seg>
         </tuv>
      </tu>
   </body>

</tmx>

4. Glossary

DTD

An SGML document has an associated Document Type Definition (DTD) that specifies the rules for the structure of the document. Several industries have standardized on various DTDs for the different types of documents that they share.

SGML

SGML stands for Standard Generalized Markup Language. An ISO standard (ISO-8879) allows the definition of structured formats. SGML is not a format by itself, but a set of rules to define formats. SGML mark-up systems are defined in Document Type Definition files (DTDs).

XML

XML stands for Extensible Markup Language. XML is a simplified and restricted subset of SGML.

UCS-2

UCS-2 is a 16-bit fixed-length encoding scheme of the Unicode character set.

UTC

UTC stands for Coordinated Universal Time.

5. References

The following links are useful references for implementing TMX, SGML and XML-related applications.

• ISO 639:1988 -- Code for the representation of names of languages
• ISO 3166:1993 -- Code for the representation of names of countries
• RFC 3066 -- Tags for Identification of Languages
• ISO 646:1991 -- Information Technology -- ISO 7-bit coded character set for information interchange (ASCII)
• ISO 8601:1988 -- Data elements and interchange formats - Information interchange - Representation of dates and times
• ISO 8879:1986 -- Information Processing - Text and Office Systems - Standard Generalized Markup Language (SGML)
• ISO 10646-1:1993 -- Information Technology - Universal Multiple-Octet Coded Character Set (UCS) - Part 1: Architecture and Basic Multilingual Plane
• IANA Code set names -- Code sets naming conventions

• XML specifications repository (at W3C)
• ISO Site
• LISA Site
• Unicode Consortium Site
• W3C Site

The most up-to-date version of this document can be obtained on the LISA Web site at http://www.lisa.org/tmx/.