Ian Lancashire's paper

Standard Generalized Markup Language (SGML) encodes medieval and Renaissance manuscripts and printed books with difficulty. This computer language is an ISO standard, but one acknowledged more in the breach than in the observance. Here I argue that the humanities should follow the originators of the World Wide Web, who made HTML (Hypertext Markup Language), an encoding standard using SGML syntax but serving purposes alien to the intentions of SGML's creators. The Text Encoding Initiative (TEI) SGML document-type definition is unusable for my kind of scholarly editing, and for the editing of early texts generally. However, the TEI Guidelines is an excellent discussion of tagging, principles and practice, and its system of over 400 tags is the starting point for anyone interested in text encoding.

I attended the founding meeting of TEI at Vassar College in 1987 and served on two of its committees (Literary Studies and the Advisory Committee, co-representing the Modern Language Association). The TEI editors knew the views I express below before they issued the current version of the TEI Guidelines. My objections helped me shape the encoding guidelines of Renaissance Electronic Texts (RET). Published on the World Wide Web at

these give practical advice and examples for applying two encoding schemes, SGML and COCOA to many textual situations. Text-Analysis Computing Tools (TACT) adopts and extends the second scheme, COCOA, from the Oxford Concordance Program. I use SGML to encode Representative Poetry, a textbook edited by members of the Department of English at Toronto and published by the University of Toronto Press from 1912 to 1967, and two volumes of RET, the 1623 edition of the Elizabethan homilies and the 1609 quarto of Shakespeare's sonnets. I use COCOA to encode Representative Poetry and many other English literature texts forthcoming from the Modern Language Association TACT manual. These texts run from Beowulf to H. G. Wells' The Time Machine. In general I use COCOA for analysis and SGML for interchange.

Many routine difficulties arise in using SGML today. The manuals and books that explain its syntax are generally written for technical experts. Michael Sperberg-McQueen, co-editor of the TEI Guidelines, describes SGML as "a formal computer language for representing text in electronic form, defined by International Standard ISO 8879" (Sperberg-McQueen 1995: 248). Every SGML document must be accompanied by a document-type definition (DTD), a structured data-file identifying tags and their relationships, but SGML has no tags. Users must invent all of them. The TEI Guidelines creates a wonderful tagset but runs to well over 1000 pages, so that the CD-ROM edition is essential to navigate the Guidelines. TEI also requires users to adopt the TEI2.DTD document-type definition (p. 49) or else modify it, a technically troublesome task, comparable to revising someone's else's software. Although widely advertised as software- independent, SGML-encoded texts can neither be verified nor browsed without using an SGML parser-browser. Such programs are not easy to acquire or to use.

SGML and TEI use entity references for characters not in ASCII. Such entities derive from official ISO character sets, but they do not include many characters found in pre-modern texts. For example, consider Figure 2, my sample edited text of the start of the late medieval interlude Lucidus and Dubius. Four of the seven special characters used on this page have no ISO entity references. Figure 3b lists all characters appearing on this page. Further, it is frustrating to make entity references for these characters because no SGML software can display them. Thus if we use undisplayable entity references, no one will be able to read the texts. The entities interfere with word-recognition. For this reason, the RET series encloses non-ASCII characters within braces and puts special marks of abbreviation (brevigraphs) within vertical bars. RET guidelines offer a table of special characters and their codes. Neither braces nor vertical bars interfere seriously with reading the edited texts, even the diplomatic transcription shown in Figure 2. It is unlikely that the ISO will create a special character set for medieval texts and thus that SGML software will develop techniques to display obsolete characters unless medievalists act in concert to express their collective will.

I believe that SGML and TEI make anachronistic assumptions about text that fly in the face of the cumulative scholarship of the humanities.

What are the principal purposes of SGML? First, it creates general textual markup that enables publishers to translate electronic texts quickly into products. Rather than having to maintain separate programs to convert WordPerfect, Word, and other proprietary word- processing schemes into local typesetting software and house style, a publisher who uses only SGML-encoded texts can devise one translation scheme. Second, SGML serves commercial text-retrieval software searching large databases and the World Wide Web. If everyone uses SGML (or HTML), everyone can search everyone else's texts. Standards are tools for the interchange of texts. For interchange purposes, SGML is essential.

However, the TEI Guidelines are inspired by a greater ambition than to do what HTML does (that is, online typesetting, which translates rendition in a text onto the screen). Encoding, the TEI Guidelines say, is "any means of making explicit an interpretation of a text." It is "a process of making explicit what is conjectural or implicit." Consequently, the TEI tagset embodies interpretation of text on hundreds of points of logical structure. It erects "content models" on the dubious assumption "that there is a common core of textual features shared by virtually all texts and virtually all serious work on texts" (7). TEI did not heed the advice of its own Literary Studies subcommittee, chaired by Paul Fortier and with a membership of scholars including myself. In October 1990 Fortier wrote in his critique of TEI P1:

My perspective is that coding (inputting or converting text) is not the same as interpreting. Descriptive coding as presented in the Guidelines is squarely in the domain of interpretation. Scholars do not want interpreted texts; they expect to do that job themselves. When possible scholars hire assistants to input texts, and do not expect these assistants to do the interpretation. This whole aspect needs to be brought into conformity with scholarly practice, otherwise the TEI standards will not be respected.

SGML was devised originally for technical writers to encode texts with their own interpretation of its parts. The author of a text interprets readily, but editors of other people's texts cannot. The interpretation of old texts, in particular, asks for a skeptical mind and a careful critical vocabulary. Thus a hidden assumption of SGML and TEI is that the encoder has the authoritative knowledge of the author. The TEI's own Literary Studies committee -- and a survey of humanities scholars then on-line used by it -- disagreed, but our objections ran counter to a principle, that tags should not describe displayable textual features.

SGML and TEI also assume that all textual features of physical layout, specific typeface, material font, and script, etc., need not be encoded in themselves. Their interpretation must be encoded instead. For example, italics should be tagged according to the purpose the encoder believes that they have. The TEI Guidelines thus do not encode the most immediate textual elements that an editor of pre-modern electronic texts faces:

We repeat the advice given at the beginning of this chapter, that these recommendations are not intended to meet every transcriptional circumstance ever likely to be faced by any scholar. They are intended rather as a base to enable encoding of the most common phenomena found in the course of scholarly transcription of primary source materials. These guidelines particularly do not address the encoding of physical description of textual witnesses: the materials of the carrier, the medium of the inscribing implement, the layout of the inscription upon the material, the organisation of the carrier materials themselves (as quiring, collation, etc.), authorial instructions or scribal markup, etc. (p. 557)

Some "carrier materials" were recognized by TEI, but only in ways that limit their usefulness in the study of early texts. I will give two examples.

The five basic attributes that every TEI element must have (pp. 45-47) includes rend (the way in which the text governed by the tag is rendered or presented), but making font or script an attribute of every other tag -- rather than an element in itself -- causes problems for editors of early texts. Change in font or script almost always is significant in itself. It may indicate emphasis, a proper name, a quoted phrase, an authoritative text of some sort, a language, or combinations of these things. The use of spaced letters in a word (called monumental or lapidary style) also is substantive. For example, the word "GOD" in the Elizabethan homilies, which I edited as Renaissance Electronic Texts 1, always appears in lapidary form. However, showing, by means of tags, the significance for each aspect of rendition makes the speculative seem more factual than it is.

My second example arises from the TEI "formework" tag, which is used to describe textual features for which the scribe and the printer are responsible. The tag marks many features of physical books (catchwords, signatures, running titles, foliation) but does so inside TEI structural tagging. tags are incidental and floating. TEI ignores bibliographical structures such as W. W. Greg, Fredson Bowers, and Thomas Tanselle have catalogued. All features of a bibliographical nature fall inside the TEI textual structure of , and tags, although bibliographical structure in fact subsumes all textual structures. Diplomatic, old-spelling editions of early works thus cannot be TEI-conformant documents unless they throw out basic TEI divisional tags and the TEI philosophy of text. Renaissance Electronic Texts accordingly proposes a two-fold parallel set of structures, one textual (content-oriented), the other bibliographical (oriented to carrier materials). See Figure 3(d) for the tagset.

At present, TEI treats sections of a text belonging to the printer or publisher as if they belonged to the author. The title-page contains the name and address of those responsible for making the book into formes and for putting signatures at the bottom of pages. It labels the physical book. The table of contents correlates the author's logical structure with widely separated page numbers, which are the printer's responsibility. The errata page is also bibliographical, not textual. It is a list of instructions from the printer: bibliographical conventions (page and line numbers, often) matched with changes in text. Scholarly editions recognize that most texts result from the efforts of at least two people, a scribe/printer and an author who may have worked on the text at different times. The textual structure of any book or manuscript is nested within a bibliographical structure; and the latter has a pervasive impact on their former. Consequently, a tagging system must be able to distinguish the work of these different agents.

Multiple responsibility and the attendant uncertainties in ascertaining its boundaries thus complicate the content-model textual structures proposed by TEI for a DTD. What happens if the two systems, bibliographical and textual, conflict, as occurs when two compositors set different parts of the same work and use different spelling systems? The two men who set Shake-speares Sonnets (1609), the forthcoming RET volume edited by Hardy Cook and myself, each interfered with the manuscript spelling -- conceivably Shakespeare's -- but did so differently. The printing house divided pages arbitrarily between the two compositors, with the result that changes of responsibility often occur in the middle of poems. Because poems are a textual division in the TEI model, bibliographical structure cuts across the textual structure at hard-to-establish points and affects the nature of the text.

SGML does not allow one to retrieve a given string according to its place in more than one structure. The so-called SGML "concur" function allows editors to encode two structures in a single text but not to retrieve strings under the two simultaneously. If the text itself has two structures (e.g., act-scene-line and classical scene-speech) in addition to the bibliographical structure, the editor cannot even represent the three structures in an SGML edition. The "concur" function only allows two structures at once. SGML has a built-in limitation. An easy solution -- within SGML -- is to declare and to employ the structural tags in the text but without specifying their relationships in the DTD.

SGML elements and their attributes belong to a metalanguage, like the markup in any tagset. They are words about words, that is, metawords, signs pointing to signs, and thus depend on a theory of the meaning of text. The theory accepted by TEI is anachronistic when applied to early texts. For example, SGML structural tags like ,, and (Charles F. Goldfarb uses these in his definitive manual to SGML) impose a visual, anthropomorphic metaphor. It is like a person viewed from the front, from inside, and from the back. Yet does any medievalist believe texts necessarily or generally have this structure? Do texts have three parts, comparable to the printed book's title-page and preliminaries, the things pointed to by the table of contents, and the closing pages, which may contain an advertisement or a postscript?

TEI could have acted to remedy this fault by drawing on humanities scholarship. Then TEI would have recognized that the -- metaphor obscures the basic structures endowed by multiple author- agents (the printer/scribe and the author) or further structural ambiguities existing in works like the Winchester interludes that belong to two or more genres -- in this case drama and verse. The bibliography at the end of the TEI Guidelines does not refer to any standard humanities works on bibliographical and textual structures. This neglect of humanities scholarship is pervasive in the Guidelines.

For example, the TEI definition of the core tagset includes basic tags for poems. It asserts that "The fundamental unit of a verse text is the verse line rather than the paragraph ..." and uses the following hierarchy to represent poetic structure: (verse line), nested beneath (line group), which in turn is nested under divisional

, , etc. TEI cites no authorities for this model. Neither does it qualify its assertion. Some literary critics (Roche 1988: 3) say that the line is the basic unit of verse (the word "verse" under some circumstances means "line"). However, most believe that the metrical foot is the basic unit (Barnet and others 1960: 90). Metrical feet nest within the line sometimes, but arguably within the rhythmic unit as often; and a rhythmic unit may be part of a line, or may cross over lines in so-called "run-on lines." It is true that early poetry, especially, is highly formal, but the basis for its form is not the line. Many manuscripts, from the Beowulf onward, do not use lineation to mark poetic form. Verse lines are run together seamlessly as if they were prose. The two TEI tags, and their rationale, do not do justice to the reality of poems. Beginning in the 19th century, when fixed metrical form itself begins to disappear as a part of the model, the TEI model totally collapses. As Bernard Dupriez says, "Each poem nowadays possesses its own structure" (1991: 346). If so, developing generalizable content models is impossible. A unique divisional structure must be assigned to many modern poems.

See Figure 2 for another illustration of the problem. What is the correct verse lineation of Lucidus and Dubius? It is not the manuscript's lineation, encoded with . I have added a tag to order text lines differently, that is, by meter. Braces determine where this different verse order breaks from the manuscript lineation, braces are encoded with and tags. Such braces obviously link rhyming lines. Thus meter is the fundamental unit of verse structure, not the kind of lineation represented by the TEI tag, which in fact misleads the reader in this case.

TEI also treats the relations of the core elements for verse to other tags oddly. Figure 4 lists the elements that the verse line tag may include, and that may include it. Why does the tag occur within cast-lists, descriptions of dramatic settings, and line-groups but not within stanzas or refrains? A TEI-er would say that a playwright might choose to put verse in a cast-list and that stanzas are types of textual divisions and so could be attributes of tags like and . Would a literary historian, however, even concede that stanzas or refrains should invariably be called divisions or line-groups? What about the poem with only one stanza, like a limerick? Can a poem with only one thing at the top level be said to be divided in terms of stanzas? does it make any sense to say that an entire poem constitutes a line-group or that one division comprises everything? Sir Gawain and the Green Knight has fits, the The Faerie Queene has books and cantos, but what does Beowulf have? When I was a graduate student, I understood it to have digressions. Are they divisions?

Now consider the things that a TEI verse line tag may contain. Can it in any world of text familiar to us hold the TEI tag , which "contains a fully-structured bibliographic citation, in which all components of the TEI file description are present" (p. 874)? or the TEI tag , which "describes a particular camera angle or viewpoint in a screen play" (p. 882)? or the TEI tag , "a reference to the orthographic form(s) of the headword" (of a dictionary)? or the TEI tag , which is exactly what it seems to be, "text displayed in tabular form, in rows and columns" (p. 1175)? In what textual universe does it make sense to define a verse line as potentially containing a one-row table? Now look at what verse-lines do not contain. There is no tag for metrical foot. The only encoding that makes it possible to encode meter is a general tag to be placed in the TEI header to a document. The TEI Guidelines, developed by three organizations, two of which were about computing in the humanities, appear to misinterpret verse structure.
6. The Trouble with TEI 2: A Forgotten Mandate
Section 1.3 of the Guidelines describes the historical background of the TEI. Nine principles were agreed on by thirty people who met to plan the TEI in November 1987, myself among them. The authors admit readily that TEI did not achieve a number of the important ends of that conference. Principle 3, in particular, was not achieved. It was that
The guidelines should define a recommended syntax for the format, define a metalanguage for the description of text-encoding schemes, describe the new format and representative existing schemes both in that metalanguage and in prose. (p. 10)
TEI did not do these things. "The only metalanguage used ... is that of SGML, and no formal definitions are given of other common encoding schemes" (p. 11). TEI took over what ISO had already developed for technical writing and publishing and tried to accommodate scholarly needs within its framework. TEI also ignored existing encoding schemes, although they had been developed ground-up by humanities researchers since the late 1960s (e.g., the encoding of the TLG, ARTFL, ADMYTE, COCOA, the Helsinki and London-Lund corpora, shareware like TACT, and commercial software such as WordCruncher). TEI was administered largely by computer professionals. Their goal appeared to shift from developing an interchange format for humanities texts, based on the features of those texts, to imposing an existing encoding format, SGML, on the humanities.
Principle 4 also states that "The guidelines should propose sets of encoding conventions suited for various applications" (p. 10). TEI admits it does not do so, "since consensus on suitable conventions for different applications proved elusive; this remains a goal for future work" (p. 11).
7. Where do we Go from here?
Editors should begin by resolving to re-assess the tagging of electronic texts from first principles, without necessarily being constrained by any known system, SGML, or TEI, or RET. The different tasks of tags should be affirmed. They certainly can instantiate the editorial apparatus of a new medium (and hence are interpretative), but they also can describe literal and visible features of text. The scholarly community should assert its traditional academic freedom in the tagging of texts. No one editorial standard has emerged after 3,000 years of work, and I am skeptical that one will emerge. A scholar should have the right to publish texts encoded in the way he or she believes best suits the purpose of the analysis.
Scholars must revisit the purposes of creating encoded electronic texts: are they just for text retrieval? If so, I think we are wasting much of our time, because although researchers need tools to find references, they are not themselves working on text-retrieval systems. They are trying to discover new things about texts. And so the question has to be put, what new things can we do with electronic texts, and then what kind of tagging do we need to accomplish those things? Analysis, not retrieval, is the important issue.
Many at this conference have already spent years thinking about these issues. For these people, more practical issues must dominate.
First, we do not yet have an inventory of codes for early characters. Only about a third of the non-ASCII characters RET editions need so far have ISO entity names. Ideally we should have the choice of not looking at either codes or standard displays of these characters -- we might want to see images of the originals -- but we will still need some way of classifying what we see. We need to define logically the early character set, or we are truly building electronic texts on sand. We have to be able to name characters so that they can be retrieved and discussed. By developing and insisting upon our own specialized character set, preferably keyed to images of the characters we are describing, we can exert pressure on publishers to deal with the issue of text representation.
Second, we need a basic tagging grammar and tagset for representing features of what the TEI Guidelines call the carrier materials. The World Wide Web uses a DTD to handle basic on-line typesetting. We need an extension of HTML to name all the literal textual phenomena we see in early books. After all, what else do we have to build an interpretation of early texts on but an accurate representation of the features of the early manuscript and print culture? Defining even apparently simple things, such as the hierarchy of pages within sheets within gatherings, is not easy. As we know, pages from the same side of the sheet (or from the same form) alternate when the sheet is folded in text order. How does one tag the form so that the appropriate pages all nest within the proper form? Try it.
SGML is a text-translation tool. It exists to turn texts encoded by scholars for their own research purposes into an interchange format suitable for publication, whether in printed books or during on-line retrieval. This format must be able to handle all character sets and all literal, uninterpreted textual phenomena. The DTD we adopt should impose few interpretative structures on our tags but should make the implementation of a variety of such structures possible. TEI did not create such a format, but it did point us in the right direction. The success of HTML shows that SGML document-type definitions can be made that defy the original intentions of SGML itself in respect of procedural tagging. I hope that this conference will take up again the spirit of the 1987 Vassar College meeting to begin to create, in a cooperative way, a metalanguage and encoding method faithful to humanities texts so that, in time, an interchange format that meets those scholarly objectives can be devised. We can work with the TEI community (for example, the Center for Electronic Texts in the Humanities under Susan Hockey's directorship) to remove the tagging anomalies in the TEI DTD and to propose the character set and the descriptive markup we need.
FIGURES
The figures are still being formated for html; they will make more semse soon. 1. Winchester Ms 33, fol. 54v

2. Winchester Ms 33, fol. 54v (transcription of opening of "Lucidus and Dubius" on basis of RET guidelines) <bkdv2 type="foliation" n="54v"> 
<plydv1 type="interlude" id="Lucidus and Dubius"> <bkdv3 n="6"> {leaf} <stage><lang type="l"><s type="af_initial">I<s type="af_normal">n|t'|roga p|_r|es tuos |&| dic|_e|t |t^i^|<lang type="e"></stage> {leaf}
<bkdv3 n="7"><plydv2 type="speech" n="1"><speaker id="Lucidus"><sppfx><s type="ha_large">lucidus</sppfx> <bkdv3 n="8"><plydv3 n="1"><s type="ha_initial">T<s type="ha_normal">he fo{r2}my{s}t fade|r'| |{th}^t^| fo{r2}med {y}ou a{ll}<xref type="brace" target="1"> <bkdv3 n="9"><plydv3 n="2">the wytt |&| {th}^e^ wy{s}dom of his {s6}|_o|ne dere<xref type="brace" target="2"> <bkdv3 n="10"><plydv3 n="3">{th}^e^ lo{r2}e of {th}^e^ holy go{s}te which y ca{ll}<target xref="1"> <bkdv3 n="11"><plydv3 n="4">to marke {th}^e^ mate|r'| to {y}|o^r^| mynde which {y}e {s6}hul hee|r'|<target xref="2"> <bkdv3 n="12"><plydv3 n="5">My name is doct|o^r^| lucidus A doct|o^r^| of {th}^e^ lawe<xref type="brace" target="3"> <bkdv3 n="13"><plydv3 n="6">to {y}ou hide|r'| y come the {s6}othe fo{r2} to {s6}ay<xref type="brace" target="4"> <bkdv3 n="14"><plydv3 n="7">que{s}tio|_n|s to A{s}oyle / |{th}^t^| be to men vnknawe<target xref="3"> <bkdv3 n="15"><plydv3 n="8">herkene |{th}'|fo{r2}e good {s6}ires fey|r'| y {y}ou pray<target xref="4"> <bkdv3 n="16"><plydv3 n="9">A doct|o^r^| of dyuynete ful wyde Am y knowe<xref type="brace" target="5"> <bkdv3 n="17"><plydv3 n="10">clerely kan y An{s}we|r'| to que{s}tio|_n|s of dowte<xref type="brace" target="6"> <bkdv3 n="18"><plydv3 n="11">|{th}^t^| beth now on {th}^e^ peple wonderly {s}owe<target xref="5"> <bkdv3 n="19"><plydv3 n="12">|&| mevith hem at hi|r'| ma|_n|gery wh|_a| {th}ei ben on a rowte<target xref="6"> <bkdv3 n="20"><plydv3 n="13">But what what / <bkdv3 n="21"><plydv3 n="14">Y know hem e|u'|ychon<xref type="brace" target="7"> <bkdv3 n="22"><plydv3 n="15">Y dowte neue|r'| oon<target xref="7"> <bkdv3 n="21.5"><plydv3 n="16">{S6}ome man may preue<xref type="brace" target="8"> <bkdv3 n="23"><plydv3 n="17">The|r'| fo{r2}e wolde y<xref type="brace" target="9"> <bkdv3 n="24"><plydv3 n="18">{th}at {s6}ome man boldely<target xref="9"> <bkdv3 n="23.5"><plydv3 n="19">A que{s}tio|_n| wolde meve<target xref="8">
<bkdv3 n="25"><plydv2 type="speech" n="2"><speaker id="Dubius"><sppfx>{rule}dubius</sppfx> <bkdv3 n="26"><plydv3 n="20">A god the {s6}afe doct|o^r^| gent<xref type="brace" target="10"> <bkdv3 n="27"><plydv3 n="21">Myn owen mai{s}|t'| verament<target xref="10"> <bkdv3 n="26.5"><plydv3 n="22">Y know {y}ou by {th}^e^ speche<xref type="brace" target="11">
<bkdv3 n="28"><plydv2 type="speech" n="3"><speaker id="Lucidus"><sppfx>{rule}lucidus</sppfx> <bkdv3 n="29"><plydv3 n="23">whi {s6}o childe what is {th}i nAme<xref type="brace" target="12">
<bkdv3 n="30"><plydv2 type="speech" n="4"><speaker id="Dubius"><sppfx>{rule}dubi|9|</sppfx> <bkdv3 n="31"><plydv3 n="24>Dubius {s6}i|r'| A{ll} in gAme<target xref="12"> <plydv3 n="25>|{th}^t^| {y}e we|r'| wonte to teche<target xref="11">
<bkdv3 n="32"><plydv2 type="speech" n="5"><speaker id="Lucidus"><sppfx>{rule}lucidus</sppfx> <bkdv3 n="33"><plydv3 n="26>what is {th}i nAme dubius <bkdv3 n="34"><plydv2 type="speech" n="4"><speaker id="Dubius"><sppfx>{rule}dubius</sppfx>
3. Character Set (fol. 54v)
(a) Initial alphabetic I T (c) Normal alphabetic

(c) Abbreviations

(d) Tagset

4. TEI and its Relations
MAY OCCUR WITHIN: add argument body castList corr div div0 div1 div2 div3 div4 div5 div6 div7 epigraph epilogue equiv item lg lg1 lg2 lg3 lg4 lg5 metDecl note performance prologue q quote remarks set sic sp stage view
MAY CONTAIN: #PCDATA abbr add address anchor att bibl biblFull biblStruct c caesura camera caption castList cit cl corr date dateRange dateStruct del distinct emph expan figure foreign formula gap gi gloss handShift hi label lang link list listBibl m measure mentioned move name note num orig oRef oVar phr ptr pRef pVar q quote ref reg rs s seg sic sound soCalled stage table tag tech term text time timeRange timeStruct title val view w xptr xref
Works Cited

Early Books, RET Encoding Guidelines, and the Trouble with SGML

Early Books, RET Encoding Guidelines,
and the Trouble with SGML