Experience of EDI for Documents: The Role of SGML

Although format conversions of unstructured documents are possible, the set of target formats is often limited and, most of all, the result of the conversion is often not satisfactory. Also, such conversions should preferably occur on the desktop instead of on a server.

Anyone familiar with SGML and with earth gravity will easily understand why translations to SGML are called up-translation and translation from SGML down-translation . It is indeed difficult to automate the enrichment of document structures, and a fortiori to exploit unstructured documents. Converting correctly structured documents to other target formats is a lot easier.

Even in the case where up-translation to SGML sounds feasible on the server, it has the disadvantage of detecting mistakes very late. Using structured editors for producing SGML right at the source is better.

However, although SGML has technical advantages, the reality is that for many organizations (although not specifically for ESA) it is very difficult to impose SGML authoring tools on authors, for various reasons. This is especially so when the authors are employed by other organizations. Moreover, even if SGML is accepted, it is challenging to agree on a DTD when many partners are involved.

EDIDOC does not enforce the use of SGML for documents. At ESA, a radically different approach was even selected for sending calls for tenders to potential contractors. Complete flexibility is offered to authors, provided that they generate PDF files as output. Of course, the problem of having PDF as a source for distribution is that it then limits the flexibility for further conversions (only ASCII versions of documents are produced from PDF in our case). But this drawback is negligible in comparison with the problem that the set of authors would face if SGML were imposed.

However, authoring using SGML is still recommended in the following cases:

• when numerous target formats are required;
• when the documents must be protected against the future (the information will be used for a very long time, probably using new technologies);
• there is a limited set of authors ;
• the partners manipulating SGML documents are involved on a very regular basis ;
• document structure is needed by some specific electronic processing applications.

If few of those conditions are fulfilled, SGML should not be considered; it would be too much of a challenge for little benefit.

SGML for labelling EDIDOC envelopes

As EDIDOC messages can be transported by many different vehicles (for example email, X.400, and WWW), we had to define an envelope for the documents in order to specify the details of the exchange: originator, list of recipients, unique reference, subject, time stamps, document types and formats, security mechanisms, delivery options, groupware context, remote management options, error messages, and so on.

Here is a sample header for an EDIDOC message:

      <esh version="2"> 
      <secpack mode="clear">none</secpack>
      </esh>
      <eh version="2"> 
      <user level="process" notif="always" clearmsg="safemsg" clearnot="safenot" nondeliv="negack" destin="most">
      <sndreq>
      <from>phv@acse.be</from>
      <ref>phv@acse.be - 10/15/97 10:46:11</ref> 
      <subject>PDF version of EDIPress manual</subject>
      <docinfo> 
      <type>FLAT</type>
      <format>PDF</format> 
      </docinfo> 
      <to>sansari</to>
      </sndreq> 
      </user> 
      </eh>
      %PDF-1.2 document is included here...

These envelopes are generated by the producers of information to send requests to EDIDOC and are parsed by the recipients who do want to exploit the EDIDOC context. This means that people should be able to generate and process them on a lot of different platforms, using different tools. In fact we developed the following APIs and tools for parsing or producing the envelopes:

• specification
  • SGML DTDs à la XML (not even empty elements);
• developers' APIs
  • a Simple SGML-like Parser (named SSLP but that I would now call a light XML parser),
  • a C++ class corresponding to the data structured in the SGML envelopes, this class being used from C++ code for example to make the link with relational databases representations,
  • a Unix shell interface based on environment variables;

• end-user tools
  • graphical clients for Windows,
  • WWW CGI scipts and Java code,
  • server-side generation of pretty MIME email
  • ASCII pretty printing.

For those integrating their applications with EDIDOC, it was easier to generate and process the envelopes directly by looking at the simple SGML specification. When security algorithms are required, the API-level interfaces are preferred. Of course, the end-users just use tools, much as an Internet surfer would use a browser for viewing HTML.

Using valid XML files (called light normalized SGML at that time) for structuring data exchanged between loosely coupled applications has the following selling points:

• it is an relatively accessible format for the person who looks at it;
• it is standardized;
• it has the big advantage of offering automatic implementation of parsers by using SGML or even XML tools and loading them with a standardized meta-specification (DTD);
• it is so easy to generate and even to parse that you can deal with it if you simply have read/write access to files on your platform.

In the classic EDI context, EDIFACT has the big advantage of having well specified and standardized formats for many types of EDI messages, such as invoices and orders. The semantics of all the fields of those messages are then clearly specified.

However, we believe that SGML, and more specifically its little child XML, is more appropriate when no existing EDIFACT message fulfils the needs. Indeed, if a new message specification has to be defined, XML has the following advantages over EDIFACT:

• DTDs not only specify the structure of the document but also do it in a standard way and are processable by any SGML-compliant parser (the definition of EDIFACT messages also has to be modelled in the parser);
• valid XML is easy to parse, from many programming languages;
• SGML documents are more intelligible from a developer's point of view (the structure and the name of the elements are easier to understand).

One of the most famous examples is that Microsoft selected a light SGML specification, à la XML, for its Open Financial Connectivity, used in the electronic dialogues between Microsoft Money and banking institutions.

SGML for implementing convertors

In its generic way for configuring convertors, EDIDOC defined, for each type of document, a pivot format. Conversions (up or down) and conformance-checking scripts are defined relative to this format. Although EDIDOC does not impose its use, SGML is the obvious choice for being a pivot format. In fact SGML not only offers a way to specify and structure documents but it even offers a few mechanisms for processing the information.

The following features are particularly useful for processing non-SGML documents with full- blown SGML-aware toolkits.

• OMITAG allows for the automatic closing of open elements when some new elements are recognized. For example all open bullet lists, paragraphs, and subsections are automatically closed when a new section is encountered, simply because of the SGML specification of the document model being used at the heart of the converters.
• SHORTREF offers a nice mechanism for generating or preserving SGML entities or for recognizing special characters. With some non-standard extensions, SHORTREF can in fact even be used for adding sophisticated pattern-matching recognition in a similar way.
• LINK allows the context-dependent attachment of code to the occurrence of some elements, in a standard way.
• CONCUR is a must when processing documents that are formatted using various concurrent formatting styles. For example bold and italic markers can typically cross paragraph boundaries. Having a tool that can present the formatting possibilities as many different concurrent grammars is very useful. One of the convertors developed at ACSE uses 25 different grammars.

Glue for linking external applications

The EDIDOC server offers the possibility to plug-in customized application logics, such as a workflow scenario for supporting document review cycles. The server keeps track of the processing evolution, dispatches the various relevant events and, most importantly, offers transparent access to the various document formats, security mechanisms, and communication schemes.

Technically, so-called groupware scenarios can be plugged in EDIDOC by implementing external scripts called "event handlers".

We thought first about including a generic workflow engine in EDIDOC. This engine would have been processing the definition of finite-state machines or Petri networks. However, we realized that there were annoying limitations for the developer in each of those models (for example "What if a new reviewer is added in our scenario?") Also, we preferred to have loosely coupled external event handlers, interacting with EDIDOC through a clean interface.

This interface had to impose as few constraints as possible on the tools used for developing programs defining the customized behaviour of the server for a workflow scenario. Also, the interface between EDIDOC and the groupware event handlers has been defined as a simple SGML (XML-compliant) interface.

Those event handlers get various kinds of event from the server:

• groupware requests sent by a partner, generally including a document (whatever the network protocol, the security options, and the document formats, it will be transparent for the event handler);
• expired time-outs registered for the scenario instance;
• non-delivery notifications.

In response to those events, the external script can request the EDIDOC server to:

• register time-outs;
• update data to be stored in the server, usually for keeping track of the context;
• produce error notification messages;
• log some specific traces;
• send messages, using any of the options that EDIDOC offers.

All of those requests are passed between the script and the server as XML-formatted data flows that comply to specific DTDs: one for the standard intput, one for the standard output. This gives complete flexibility to the developers for the languages and tools to be used for implementing the external scripts; if they can control the standard input and output, they have all what is needed. Of course, using an SGML toolkit gives the advantage of having robust parsing with little effort.

Implementation of workflow scenarios

This is the most complicated part of the paper and covers a methodology that our Group is using on several projects. This methodology helps us to define, document, and implement applications based on multiple state diagrams. This is particularly useful for implementing workflow solutions.

State diagrams specification

We have been developing a meta-DTD for specifying classes of state diagrams. From this meta- DTD, the following documents are automatically generated:

• TeX documentation of the classes, state diagrams, and events.
• implementation-DTDs that define the implementation of state diagrams by means of standard SGML features.

In the meta-DTD, classes are defined as sets of finite state diagrams, including states and events.

SGML representation of events

In the generated implementation-DTD, each event is mapped to an SGML element. That element can have specific attributes and can include structured data as child elements; this allows for the passing complex structured information with each event.

Linking code to events

SGML link rules allows for the attachment of external information to SGML elements in a standard context-dependent way. We use this mechanism to attach application logic to SGML events. Thanks to the implementation-DTD, different link sets are used for each state in the diagram; they are automatically activated by standard SGML means when a new state is reached. Specific application code can then be linked to the occurence of the various events by the customization of the SGML link rules of the link set, for example for accessing a database and forwarding events to other processes. The code to execute when an event element occurs is selected by the standard use of SGML LINK.

Implementation of the state transitions

When a new event element is input, the SGML parser also knows the new state of the state diagram by standard SGML means. This is due to the appropriate definition of the SGML group models and the OMITAG feature in the generated implementation-DTD. No specific code has to be written for doing this.

The following example illustrates a definition of state elements similar to the ones generated in the sample DTD. When an event is encountered in the input stream, the SGML parser selects implicitly the new state.

      <!ELEMENT stateA O O	((event1,stateB)|(event2,stateA)> 
      <!ELEMENT stateB O O	(event3,stateC)|(event4,stateA)> 
      ...

Concurrent state-diagrams

At run-time, for each class of diagrams, a single SGML parsing process is in charge of managing all the state-diagrams and the associated logic; there is one concurrent parsing of the various SGML DTDs defined for each state diagram. The application language has concurrent access to the various contexts.

Benefits of the approach

Such an approach offers key benefits:

• the formalization of the application logic as state diagrams leads to cleaner, more robust code;
• it means that the code includes less explicit tests and is more concentrated on the implementation of specific processing;
• moreover, the code relative to the application logic, for example a workflow, is less subject to errors;
• if the workflow is slightly changed, for example by allowing a user action in a new state or by defining some new actions, the impact on the code is limited;
• there is an important productivity gain by using a common, consistent way to work across projects, if appropriate.

Conclusions

This paper has demonstrated where SGML is invaluable for system development and integration, based on the EDIDOC experience.

XML can be considered as a candidate of choice for structuring data in EDI applications, when no EDIFACT message fulfils the needs. It is also very useful as a way to structure interprocess communications when integrating distributed applications.

Full-blown SGML toolkits are a must for data processing. They are particularly useful for implementing data convertors, just by using standard features such as OMITAG, SHORTREF, LINK, and CONCUR. Finally, SGML and related development tools offer a nice way for addressing, at the same time, the definition, documentation, and implementation of workflow scenarios.

Thanks

I wish to thank Jordi Farrès, of the European Space Agency, for having defined the EDIDOC requirements and for the numerous open-mind technical discussions during the analysis and design phases.

My thanks are also due to Stéphane Bidoul for his explanations regarding the implementation of state-diagrams by the means of SGML technology.

References

• [Bidoul 96] Stéphane Bidoul; Object Orientation and SGML: Link Revealed. In Conference Proceedings of SGML'96 US , pp 485-98

Acronyms

CGI     Common Gateway Interface
DTD     Document Type Definition
EDI     Electronic Data Interchange
EDIDOC  Electronic Data Interchange for DOCuments
EDIFACT Electronic Data Interchange for Administration, Commerce and Transport
ESA     European Space Agency
FTP     File Transfer Protocol
HTML    HyperText Markup Language
MAC     Message Authentication Code
MIME    Multipurpose Internet Mail Extensions
PGP     Pretty Good Privacy
SGML    Standardized Generalized Markup Language
URL     Uniform Ressource Locator
WAN     Wide Area Network    
WWW     World Wide Web
XML     Extensible Markup Language

---

Please mail your comments to Philippe Vijghen at phv@acse.be

This paper was first published in the Conference Proceedings of SGML'97 US, December 1998, pp 213-218.

Copyright © 1997 The SGML Technologies Group