LARGE-SCALE ELECTRONIC DOCUMENT DISTRIBUTION AT PACIFIC BELL

Server

Figure 3: Server Software Architecture

The server-side of the software architecture of our system is illustrated in figure 3. The workhorse of this piece of our delivery system is Open Text Inc.'s LiveLink Search (LLS) product, a search and retrieval engine which runs as a CGI process under the HTTP demon. The LLS engine provides the function of retrieving entire documents or selected regions of documents according to a "region query" passed to it as a parameter (see Region Queries). The documents themselves are stored in the server's file system along with indices which are rebuilt periodically. Indices and the documents themselves may be distributed across multiple servers for load distribution and/or redundancy purposes; LLS' "Parallel Execution Monitor" is used to broadcast incoming queries to these multiple servers to realize these goals.

Documents are added to the delivery system as the final step in a document loading and preparation process. While a complete explanation of this process is beyond the scope of this paper, the following highlights are offered. Documents arrive at the start of the preparation process either by being posted to the server using an HTML-form-based file upload tool, or as a side-effect of them being posted into a document management system employed within Pacific Bell. Once entered into this process, documents are subjected to one or more translation steps. In cases where a document doesn't arrive in SGML or HTML format, translation into one or the other of these formats is first performed. A translation application built using the Omnimark translation tool (Omnimark Technologies Inc.) is then used to extract the document structure data needed to implement the navigable table of contents (TOC); and, depending on the format of the source document, may also perform some optimization of the document content's markup.

When a document has been loaded into the system, its TOC elements are ready for use in the TOC database, and its contents are loaded into the LLS and indexed. When a document or document region is requested and retrieved by the LLS system, it is piped through a Results Processor, a program which performs any on-the-fly transformations required. Currently the Results Processor is a Perl program we have developed which highlights user-entered search terms in the text and also adds Previous and Next Region hyperlinks at the beginning and end of the text. We have also prototyped use of the Results Processor to convert documents from their source format, e.g., SGML, into the delivery format, HTML, however in our current implementation we perform such transformations off-line as described above.

The server-side of a distributed TOC delivery, presentation and navigation subsystem is built using a lightweight database program which contains the TOC for each document in the system, and which executes as a CGI program on the server. The TOC, consisting of the set of "nodes" in the documents structure hierarchy, can consist of a significant amount of data. Our distributed design eliminates the need to deliver all of this data to the client when only a small portion of it is employed in a particular view of the TOC the user will see. The input into this program is a document identifier, the current state of the TOC on the client, and the reader's latest request. A request only goes to the server when the reader requests a portion of the TOC which is not present on the client. The program returns the previous state of the TOC and the new section requested by reader, as well as any peer nodes at the most recently opened level of the TOC hierarchy and, in order to take advantage of the principle of locality, some adjacent nodes as permitted by size constraints which are set based on experience with the network load (optimum size to be delivered to the client) and the performance of the (JavaScript-based) client-side of the TOC subsystem.

Client

The client-side of our delivery system is built on the web browser provided by our standard desktop environment, currently Netscape Navigator. The special features of our system are implemented through the use of frames, HTML forms, and JavaScript.

Figure 4: Sketch of the User Interface.

The interface used by users of the system is sketched in figure 4. ^* Three frames are presented to the reader. The main frame, on the right hand side, holds the document proper. The left hand side is for locating information; one of the two frames contains an HTML form from which the reader can send full-text, structural, and domain-restricted queries and the other frame is the navigator. Three different views can be presented in the navigation frame; library, book TOC, and search results, depending upon the system's state.

The document frame contains the document region last selected by the reader from either the search results or the book TOC. In the former case hits will be highlighted with hyperlinked arrow icons which allow the user to jump between occurrences of the hits within the current region. There are also previous and next icons at the top and bottom of the document (region) which permit the reader to go to adjacent regions. While our system is not page-oriented this feature provides a comfortable equivalent to "turning the page". Previous and next , like all document retrieval operations, are implemented as queries which are passed to the server and there to the LLS and Results Processor.

The search frame is a simplified variant of the standard LLS query interface, but with specific knowledge about the organization of the document repository "built-in". Specifically, relevance and precision may be increased by limiting the domain of the search. For example, we have organized a repository along the lines of the SuperBook model with Libraries containing Shelves and Shelves containing Books. The search may be restricted to specific books, shelves, or libraries or may be repository-wide. The reader can also limit the scope of the information returned to hierarchically defined containers within the documents such as "chapter" or "section" or to content or structure defined containers such as "examples", "warnings", "tables", or "figure captions". The organization of the repository and the definitions of hierarchical, content, or structural containers varies among different groups within Pacific Bell. The search frame may be customized to accommodate these differences.

Library Navigation and TOC Subsystem

An expandable-contractable library/TOC presentation and navigation capability is built using an implementation distributed across client and server platforms. On the client-side, both library-level and book TOC-level presentation and navigation offer an expandable and collapsible tree view, using the same JavaScript-based client-side program. At start-up, or in response to user navigation, TOC data is fetched from the server-side TOC database and carried to the client within the dataspace of this program. When this program runs it draws the currently visible library or TOC hierarchy with certain tree nodes "expanded" according to the current state as developed during prior reader interaction with this structure.

A subtree of a library or document TOC hierarchy is expanded when the user clicks on a plus sign to the left of a node's description label (e.g., section title), and is collapsed by a click on a minus sign. The JavaScript program will modify the state of the TOC and immediately do a redraw if all needed portions of the hierarchy have previously been downloaded to the client; otherwise the mouse click generates a request to the server and CGI TOC database program on the server. In response to this request an updated TOC is downloaded. We have discovered in our testing that significant delays may occur on the client if the JavaScript program must process a large number of TOC entries, so this limitation must be balanced against the cost of extra server requests and downloads. Alternatively, a reimplementation in Java could reduce this delay be increasing processing performance.

Clicking on a leaf node in the library navigator causes a book TOC to be downloaded into the same frame. A hyperlink which causes toggling back to the library navigator is provided in the upper frame. Clicking on any node description text (rather than on the plus or minus sign which may proceed it) in the book TOC causes all document content located beneath the subtree headed by this node to be downloaded into the text frame. The actual download request takes the form of a HTTP GET or POST message to the server which results in a CGI query to LLS for the appropriate region of the document.

When a user performs a search, the search results list also overwrites the navigator frame. Presentation of the search results list is customizable with respect to ranking criteria and meta-data offered such as the name of the container in which the hit occurs (e.g., book name, chapter title, section title) and the display of the proximate text.

State

Our system maintains a small amount of state information and maintains it entirely on the client; in cookies, embedded in interface forms, and in document content navigation links. The tree view program used in the navigator frame stores the state of each node (expanded or not-expanded) in the cookie. The search form contains a hidden field of information with the current context. The next and previous regions adjacent to the currently visible document region are stored as hyperlinks in document content held in the the view window.

Documents: SGML + HTML

Document markup and thus document preparation are just as critical to the systems capabilities as the other components of the delivery system. The very large number of documents which will be made available through our delivery system requires that the production process be streamlined for large-scale cost efficiency while meeting the specific requirements of a distribution system designed for a large-scale readership.

The ability to search and retrieve documents based on the hierarchy or content comes from the documents being hierarchically composed and marked for content in the first place. The technology for accomplishing this is Standard Generalized Markup Language (SGML). SGML defines a formal way to create document markup languages. The formal expression of the rules of the markup language can be employed in a variety of tools to create and check documents using those rules and to transform documents by manipulating the tokens delineated by the markup language. HTML, in fact, is a markup language defined by SGML. Unfortunately, HTML writers and HTML tools tend to honor SGML in the breach, impeding the application of SGML-based tools to the betterment of the World Wide Web.

The SGML markup in our system creates, as Douglas Engelbart expressed it in [2], "Explicitly Structured Documents -- where the objects comprising a document are arranged in an explicit hierarchical structure, and compound-object substructures may be explicitly addressed for access or manipulation of the structural relationships." Open Text's LLS provides us with the ability to index these hierarchical objects and to retrieve them using "region expressions", the general principle of which is described in [1].

Our documents originate from a variety of sources including Microsoft Word, native HTML, and SGML. In all three cases we require the authors or providers to, at a minimum, encode hierarchical divisions into the documents. Those hierarchical divisions are used to generate the TOC database and thus become basic units of content search and retrieval. The authors may also identify other units, or containers, of content (by use of tags or styles depending upon their authoring tools), and when they do, these become additional units of search and retrieval in the delivery system. These additional containers may be smaller (finer-grained), or larger than the hierarchical units, and may be based in traditional document technology, "example" or "paragraph, for example, or may contain units meaningful within an application or knowledge domain.

Use of Domain Specific Containers, such as for example, "product" and "question" in a FAQ document collection, can be used to improve precision in search and conciseness of returned content in recall. Containers (objects) such as these are defined within the knowledge domain of a particular application; for example, "order entry instructions" might be a meaningful object for sales representatives. Conversion scripts have been developed to pair optimal authoring environments in these application areas, to SGML markup useful to navigation and search device construction, and HTML markup which is useful in delivery.

Our current SGML markup encompasses HTML; that is, both HTML and non-HTML tags can exist in a document at the same time. The non-HTML tags provide container and the addressing information while the HTML markup describes content formatting. Currently, we are able to deliver these "SGML+HTML" documents directly to the Netscape client as it, and other current generation browsers, simply ignores the unrecognized SGML markup. We have it at least informally from W3C that the HTML standard will include a formalization of the current treatment of unrecognized markup but we recognize that this may be an unsolid foundation. If future web-browsers are built to be less lenient toward unrecognized markup or collisions in tag name space occur we can either make creative use of the existing HTML element set or we will need to strip the non-HTML tags from the content before delivering it to the client. We could do the latter either during preprocessing or on-the-fly in the results processor. A third option rapidly developing into a distinct possiblity is to use our current element set with a Web browser capable of handling the simplified form of SGML called XML (eXtended Markup Language) being developed, under the auspices of W3C, for general Web use and as an alternative to HTML.

Open Systems

Each component in our delivery system provides a standard or widely-accepted interface to other components in the system. The availability of open, modular components has enabled us to architect the best overall strategy for our environment and to further customize our approach for our sub-organizations. We have not had to build any of the base technologies (document encoding, search, server, presentation) from scratch effect.

Vendor Independence

Scalability, as a function over time¹ benefits greatly from the ability to take advantage of improvements in technology as scale increases. Maintaining vendor independence gives the maximum freedom to adopt best of the breed technologies as they appear and also to keep the pressure on vendors to keep technologies competitive. Document production, delivery, and presentation are decoupled and we continue to evaluate and integrate multiple tools in the areas of authoring (Word processors, DTP, and SGML), search, and browsers (HTML, PDF, SGML, and others).

Zero Added Cost Client Administration

The maintenance of common set of computer desktop tools and capabilities in a large corporation with a heterogeneous body of users and equipment is a mind-boggling undertaking and perhaps the greatest single challenge in enterprise client-server computing. ² Our thin client built on the Web browser achieves zero added cost roll-out and administration for our Netscape-equipped Standard Desktop while still distributing computing, being network-efficient, and meeting our user interface objectives. Strictly speaking, we should perhaps distribute the cost of maintaining the Web browser among all the applications that take advantage of its generic interface, but in any event it is clear that the client administration overhead of our document distribution system is drastically reduced over that of a thick client system like SuperBook. The number of workstations with access to the document distribution system can always be immediately scaled to the number of workstations equipped with our Standard Desktop. From the point of view of client administration there is no limit within our enterprise to the scalability of the thin client solution.

Balance the Distribution of Computation

Server load is reduced and controlled by pushing some of the computation which could take place at delivery time off-line. The use of region indices, computed at off-hours, greatly reduces the overhead for retrieval. This in turn was enabled by design of the SGML+HTML documents, again generated off-line, which can both be region-indexed and delivered to the browser without on-the-fly transformation. TOC processing is done on the client, while database retrieval operations on large TOCs are on the server. The size and frequency of TOC downloads can be adjusted and balanced by the library administrator.

Structured Documents

Our insights into the natural and traditionally implicit structure of documents came out of our experience with SGML. We require our authors to make that structure explicit using a variety of authoring tools and techniques from word processor stylesheets to full structural (SGML) editors and use that structure to chunk information and level system demand on delivery. Unlike pure SGML systems correct stylesheet usage is not checked when the document is created but we perform a series of validations during subsequent procesing of the document. One of our greatest challenges in creating a large-scale volume of information is to retrain authors accustomed to crafting printed pages using WYSIWYG software. (See [8] for an excellent critique of the problem of moving from WYSIWYG systems to structured document authoring. Engelbert says WYSIWYG will give way to WYSIWYN[eed], that is, authoring tools will provide "different options for how you'd view selected portions of the document space". [1]) Change overnight in a large organization like ours is not feasible so the emphasis has been on incremental improvement. In lieu of having authors produce highly and accurately structured documents, a lot of our effort is currently being put into creating special purpose, custom filters which can take loosely structured word processing documents and recognize implicit structure from sequential, textual, and visual patterns. In the SGML world this is part and parcel of the highly developed art of legacy conversion as described in numerous papers such as [7].

Write Once/Read Many³ Benefit Analysis

One of the primary objections to structured authoring is that it requires more upfront effort. Whether or not that is true, the simple fact that effort applied at the authoring side can be repaid many times over by the effectiveness of the delivery of information to a large number of clients is often overlooked. Sometimes the authoring team does not have a sense of their documents as a product which adds value to the company when delivered effectively or sometimes there is a lack of appreciation for the economics of large-scale document distribution. Our project has been very effective in motivating interest in structured authoring because the benefits are clear.

Deliver Only What the Reader Wants

The reader is not forced to think about downloading the optimum-sized piece of information as the TOC and region delivery directs him or her to the size component needed. A reader can choose, however, between downloading a chapter at once in its entirety or section by section. On average the size and frequency of downloads will settle at the optimum balance as the readers learn to respond intelligently to system performance.

Detach Delivery from Authoring

We mentioned that some authors may object to additional effort imposed by structured documents. But that extra effort may be paid back not only by the efficiency of the delivery system but also by the effective life-span of the documents they create. Structured authoring leads to more tool-independent and reusable, multi-purpose documents. We also mentioned that scalability is a function of time; major changes in tools and proprietary formats every two years was standard but now we faced with Web "dog-years" and six month cycles. For example, we anticipate significant changes in HTML before we have rolled out our system to the majority of users at Pacific Bell⁴. We will be able to take advantage of those changes. The intrinsic properties of information are eternal but the format du jour is not.

Automation and High-Volume Document Production

WYSIWYG DTP tools have enabled and required the technical writer to take over many of the tasks in the publication production workflow which were formerly performed by specialists in those areas - e.g., graphics designers, typesetters, pasteup artists, illustrators. ⁸ While this approach has a number of benefits it may not be the most efficient production model for the creation of a large volume of corporate information. Structured authoring reintroduces the idea of a division of labor and also promotes the consolidation and automation of repetitive and repeated tasks. In a structured authoring environment the design of document structures is performed by a document architect, the mapping of document structure to a page layout is performed by a formatting expert and the mechanics of page composition is handled by a programmer. The authors primary concern becomes writing, just writing. A side-effect of this process which turns out to be a great advantage in automating the document production process is that the idiosyncrasies of individual authors working with WYSIWYG tools largely disappear. WYSIWYG, paper production tools, virtually gurantee that format inconsistencies which are inconsequential on the printed page will cause problems when the documents are processed by text processing software. We are working toward setting up high-volume production processes which minimizes the amount of manual "fix-up"⁵, an objective which is entirely complementary to our move toward structured authoring in order to enjoy the benefits of document structure for document distribution.

Summary

One of the things we hoped to get across in this paper is that there is an intimate relationship between scalability of technology and the way that human beings use that technology. We have found that we can neither count on the technology being used efficiently in the absence of a careful designed plan for scalability nor that we can obtain scalability simply by "educating users". The technology must be crafted so that the rewards ...and punishments... condition both content developers and readers to make the most efficient use of resources from a global point of view. The incentive for the reader to learn to navigate, search, and retrieve in the, network-wise, most efficient way is that that is the way the information sought is most quickly found and downloaded. The incentive for creating structured information is in lower long-term production costs, better longevity and therefore increased payback from development costs, and the higher intrinsic value through the increased efficiency of the reader and the system utilization. Engelbart in [2] discusses the relationship of explicitly structured documents to knowledge-domain interoperability, not scalability. One of the greatest upsides of our project is that we have been able to obtain scalability while also taking a large step toward enabling us, in the future, to optimize the use of our corporate information as an enhanced knowledge repository.

Bibliography

F. J. Burkowski, C. L. A. Clarke, and G. V. Cormack, "An Algebra for Structured Text Search and a Framework for its Implementation," The Computer Journal, 38(1):43-46, 1995. (electronic version @ ftp://cs-archive.uwaterloo.ca/cs-archive/CS-94-30)

D. C. Engelbart, "Knowledge-Domain Interoperability and an Open Hyperdocument System," Proc. of the Conf. on Computer-Supported Cooperative Work, Los Angeles, CA, October 7-10, 1990, pp. 143-156. Republished in Hypertext/Hypermedia Handbook, E. Berk and J. Devlin, eds., McGraw-Hill, 1991. pp. 397-413. (electronic version @ http://beluga.dc.isx.com/bootstrap/final/augment-132082.htm)

J. R. Remde, L. M. Gomez, and T.K. Landauer, "SuperBook: An Automatic Tool for Information Exploration - Hypertext?,", Hypertext `87 Papers.

D. E. Egan, J. R. Remde, L. M. Gomez, T. K. Landauer, J. Eberhardt, and C. C. Lochbaum, "Formative Design-Evaluation of SuperBook," ACM Transactions on Information Systems, Vol. 7, No. 1, January 1989, pp. 30-57.

D. E. Egan, J. R. Remde, T. K. Landauer, J. Eberhardt, C. C. Lochbaum, and L. M. Gomez, "Acquiring Information in Books and SuperBooks," Machine-Mediated Learning, Vol. 3, 1989, pp. 259-277.

D. E. Egan, J. R. Remde, T. K. Landauer, J. Eberhardt, C. C. Lochbaum, and L. M. Gomez, "Acquiring Information in Books and SuperBooks," Machine-Mediated Learning, Vol. 3, 1989, pp. 259-277.

C. Palowitch, D. Stewart, "Automating the Structural Markup Process in the Conversion of Printed Documents to Electronic Texts," Draft, possibly published in Digital Libraries '95 , March 1995. (electronic version @ http://sil.org/sgml/palowitchdl95paper.html)

C. Taylor, "What has WYSIWYG done to us?", The Seybold Report on Publishing Systems, Volume 26, Number 2, September 30, 1996. (electionic version @ http://www.datatext.co.uk/ideography/library/seybold/WYSIWYG.html )