[This local archive copy is from the official and canonical URL, http://www.ozemail.com.au/~birchb/linuxml/linuxml.htm, 1999-02-26; please refer to the canonical source document if possible.]


Introduction | The "Vision" | How to contribute to the project | The Architecture | Design | Implementation

Welcome to the LINUXML project.

This project is devoted to changing the UNIX de facto standard for inter-process communication and storage from line-based ASCII records to XML. Linux is the reference platform.

1. Introduction

This project originated from my daily frustration with xterm. It annoyed me that xterm didn't allow me to click on the ASCII words and symbols with my mouse and get things to happen beyond simple text cut and paste. My original whinge on this subject was posted on Usenet back in 1994. (see below).

Since then Linux has come to us and all, and XML is now appearing. My original thinking on this has matured and I realise that a deeper change to UNIX can remove other frustrations I have with UNIX. For example, getting "sort" to work on arbitrary record formats always is a pain and remembering umpteen different "etc/conf" file formats is taxing.

By now you'll want to know some more details.

In essence the idea is simple. Instead of a program like "ls" outputting its data in ASCII records, it outputs the directory listing in XML. This allows all the downstream programs, like "sort" and "xterm" to actually understand what the data is and do more useful things with it. Here's an "ls" output:

[birch@redhat birch]$ ls -F -xml /

bin/ etc/ lost+found/ root/ u01/ u04/

boot/ home/ mnt/ sbin/ u02/ usr/

dev/ lib/ proc/ tmp/ u03/ var/

In particular xterm can be programmed that when you click on a directory such as "/home", it tells the shell to "cd /home".

Here is another "ls" example

[birch@redhat birch]$ ls -l -xml /

total 53

drwxr-xr-x 2 root root 2048 Jan 19 04:35 bin

drwxr-xr-x 2 root root 1024 Jan 30 23:30 boot

drwxr-xr-x 2 root root 21504 Jan 30 12:30 dev

drwxr-xr-x 25 root root 2048 Feb 2 12:59 etc

drwxr-xr-x 16 root root 1024 Feb 2 12:58 home

drwxr-xr-x 4 root root 2048 Jan 19 04:31 lib

drwxr-xr-x 2 root root 12288 Jan 19 04:08 lost+found

drwxr-xr-x 4 root root 1024 Jan 19 04:09 mnt

dr-xr-xr-x 5 root root 0 Jan 30 23:30 proc

drwxr-x--- 6 root root 1024 Feb 2 13:00 root

drwxr-xr-x 3 root root 2048 Jan 19 04:39 sbin

drwxr-xr-x 21 oracle dba 1024 Feb 7 04:25 tmp

Likewise "sort" can be re-programmed to sort on the size directly since the XML output of "ls" explicitly identifies the size. i.e. instead of "ls -l / | sort -n +4" you have:

[birch@redhat birch]$ ls -l -xml / | sort -size

dr-xr-xr-x 5 root root 0 Jan 30 23:30 proc

total 53

drwxr-x--- 6 root root 1024 Feb 2 13:00 root

drwxr-xr-x 2 root root 1024 Jan 30 23:30 boot

drwxr-xr-x 4 root root 1024 Jan 19 04:09 mnt

drwxr-xr-x 15 root root 1024 Jan 19 04:39 var

drwxr-xr-x 16 root root 1024 Feb 2 12:58 home

drwxr-xr-x 19 oracle dba 1024 Jan 19 04:24 usr

drwxr-xr-x 21 oracle dba 1024 Feb 7 04:25 tmp

drwxr-xr-x 2 root root 2048 Jan 19 04:35 bin

drwxr-xr-x 3 root root 2048 Jan 19 04:39 sbin

drwxr-xr-x 4 root root 2048 Jan 19 04:31 lib

drwxr-xr-x 25 root root 2048 Feb 2 12:59 etc

drwxr-xr-x 2 root root 12288 Jan 19 04:08 lost+found

drwxr-xr-x 2 root root 21504 Jan 30 12:30 dev

That's it! The rest of this web site is all about achieving this goal.

2. The "Vision"

I'm not a big fan of this word, it sounds pretentious but it seems to fit here.

2.1 Old Concept

Date: 24th April 1994

Subject: Do HyperTerminals Exist?

Summary: Or what do you get when you cross xterm, ksh and an OO-shell?

Synopsis: Requirement for a combined OO Shell and Terminal Emulator.


In this letter I describe a tool that I would really like to possess.

My question to you, dear reader, is does such a wonder exist? If it

does not I challenge the programming community to implement one for

public consumption and enjoyment.



You are all experts with ASCII terminals and command-line interfaces

such as the bourne shell. You are also conversant with hypertext and

object-oriented file managers. What I want is an amalgamation of all

these with a solid OO framework. A hyperTerminal session may proceed

as follows:

Imagine you are using X-windows, on your screen is an xterm-like

terminal emulator window running a shell. You type "ls" and a list of

files is printed, neatly arranged in columns. With the mouse, you

double-click on one of the filenames (foo.cc). An emacs window is

popped up open on the file. You notice that unseen hands have entered

the command "emacs foo.cc &", and the prompt has been re-printed.

Now you move the pointer over another file (bar.cc) and you

click-and-hold the mouse button. You drag the filename (which follows

the pointer) until it is over another filename (dir/). You release the

mouse button, and "bar.cc" disappears. You notice that unseen

hands have again been at work, having typed "mv bar.cc dir".

You double-click over the "dir/" text and the command "cd dir; ls" is

typed for you. Your prompt changes to show the new directory and a

directory listing is printed as before. You notice that "bar.cc"

appears among the entries.

What is a HyperTerminal?


A HyperTerminal provides all the functions of a standard xterm yet it

also understands the information it displays. HyperTerminal displays

a continuous stream views of underlying objects. It still manages to

preserve the familiar textual presentation of an ASCII terminal.

Since it is displaying views of objects, the objects can be directly

manipulated by the user with a mouse.

Commands such as "ls" are re-written or wrapped so that they output a

stream of view objects rather than a stream of characters. Hyperterm

can be programmed to understand all classes of objects ranging from

fragments of a text file, through to a UNIX process.

Users can browse the object classes with a graphical browser,

re-structuring inheritance relationships and defining shell scripts to

be executed for each method the object class supports.



Here are some of the things that a HyperTerminal could do:

* Fragments of ASCII files output by commands like "head", "grep" or

"cat" can be edited by positioning the cursor over the text, and

inserting or overtyping characters. The changes are committed to the


* Directories displayed with "ls -l" can be modified by positioning

the cursor over the relevant attributes and over-typing. Filenames,

permissions and dates can be modified by altering the corresponding

part of the output.

* Files can be moved between directories by drag-and-drop.

* File links can be established by drag&drop with the shift key held.

* Files can be deleted by selecting of a filename followed by hitting

the delete key. a confirmation dialogue may appear depending upon


* Programs such as compilers output references to fragments of ASCII

files. Double-clicking on an error message opens an editor on the

relevant file, positioned at the line in question. HyperTerminal

keyboard focus enables you so step between error messages with the

cursor keys.

* Job control and process management is simplified. For example,

selecting processes displayed with the "ps" command allows you to

kill the process with the delete key.

* Non-wrapped commands output straight text which HyperTerminal

treats identically to the ASCII terminal it emulates.

By now the general character of HyperTerminal is becoming clearer. It

allows you to use all the power of the scripted interface (and all

those familiar commands) yet providing object-oriented direct

manipulation of the results of those commands.



Because HyperTerminal belongs to that large family of windows-based

programs, it comes with the usual framework which includes:

* Pull-down menus from a menu-bar. These allow you to call up generic

methods shared by almost all objects. Some of the standard ones might be:










Save // saves the current session a la "script"







Class Browser




* The right mouse button would support a pop-up menu showing all the

actual methods supported by the selected object(s)

* Selection of many objects via the shift key, polymorphic method

execution over the selected objects.

* HyperTerminal would also provide users with many presentation

options for each class including: font, text style, colour and use of

icons. For example executable files might always be displayed in Red.

* Because HyperTerminal is pretending to be an ASCII terminal on X,

it could also be induced to display graphical representations of

objects. Thus icons could be used to display files in the output of an

"ls" command; tick boxes might replace permission switches.

Class Definitions


HyperTerminal is extensible allowing users to add object classes and

methods of those classes. It supports inheritance and polymorphism,

and maybe multiple inheritance. To make this easy, it comes with

a graphical class browser.

Dynamic Behaviour


Since HyperTerminal continues to know about the under-lying objects

it is displaying, it could update the display to reflect the execution

of subsequent commands or change to show the state of the objects.

For example, deleting a file may cause it's entry in previous "ls"

output to change into passive, black text.

Practical Considerations


Despite offering more functions than xterm, it would not consume much

more than "xterm" and "sh" combined. (The class browser would probably

be a separate process.) It would be fast and efficient, allowing users

to run dozens of HyperTerminal windows.

The source code of Hypterterminal would ideally be freely available

along with ports to most UNIX systems.

Hyperterminal would probably continue with the UNIX style of working

by interpreting a stream of characters on its standard input. A clever

arrangement of escape sequences would delimit the objects it displays

along with their class and identity. The sequences would be published

thus allowing users to write programs that output object views into




Ok so you have read my cursory description of the mythical

Hyperterminal. I hope it was reasonably clear :-). My questions are:

Does such an animal exist?

Would anybody else like one?

If there is one where is it, and how can I get one?

If there is nothing like Hyperterminal, I re-iterate my challenge to

the programming community, please, write one!


2.2 The New Vision

The new 'vision' for the project consists of combining the idea of 'object streams' and direct manipulation in the user interface. Rolling all the smarts into the user interface is an option, but we need to keep the magic of the "|" pipe alive. The original concept is still valid, but a more structured approach to getting there is needed. Remember this is 'vision' material so some of it will be beyond our immediate reach. There are some key parts which I believe we need.

2.2.1 A Standard for Defining Data

We need to be able to describe the format of objects in object-streams. The format of the data should itself be standard. Why have different output data for every program (so "find" and "ls" could share the sae output format). XML appears to provide us the solution.

Meta-data is also a requirement. Format specs have to be machine readable for programs that need it. An obvious candidate meta-data format is XML's DTD, although it does lack detail on atomic data types.

2.2.2 An Object Model for Common UNIX Objects

Objects like files in UNIX are very standard. It makes sense to create common object models (DTDs) for all standard UNIX objects. This way it won't hurt much when people hard code these into their programs. For example no-one should be penalised for hard-coding expectations about the XML syntax of standard objects. Also, having a standard for common things should allow a lot of code re-use.

Applications that don't use standard UNIX objects must also be able to use the same object and meta-data stream formats.

2.2.3 A Standard Way to Load Methods

It would be very cool indeed if programs like "sort" were able to dynamically load methods for objects they are working on. This would also apply to user-interface programs, which would need to dynamically discover the capabilities of an object class, and load the relevant methods.

2.2.4 Backward Compatibility

LINUXML needs to be fully backward compatible with UNIX & Linux of yore. For example new versions of "ls" would have the XML output as a new command-line switch, new versions of "xterm" must work with straight ASCII. Also the role of producers like "ls", filters like "sort" and terminals should remain the unchanged..

2.3 Examples

To be completed....

I would love to provide an electronic storyboard of the vision for you here. Either a movie file or an animation would be great.

3. How to contribute to the project

This is an Open Source project, and will be covered by the GNU license.

All forms of contribution will be welcomed. Currently the most useful contributions will be in the form of feedback on the vision and architecture. More concrete help in the forms of DTDs and source will be welcomed with open arms. Thinking an opinion on the issues list would also be helpful.

Email birchb@ozemail.com.au if you have some time.

4. The Architecture

To be completed.

A description of the relationship between the supplier programs like "ls", filters like "sort" , shells and user interfaces.

Current thinking places the user interface as being 'dumb'. It will know when objects have been clicked, and will convert user clicks into shell commands. eg xterm will transmit a message to a shell instructing it that the user had clicked on a directory. ("$ click /home") Currently, I see no reason to use XML on the input to a shell, mainly for backward compatibility reasons.

The shell will then deduce that a "cd" is appropriate and execute that operation. This places all the OO smarts in the shell (knowledge of object classes and methods). Allows early prototyping with existing shell languages. It could be that a Java shell will be the best?

4.1 Object Streams for UNIX

To be completed.

Description of an object stream. Differentiation between an object (eg a file) and a view on the object (eg output from "ls"). Incorporation of model-view pattern.

4.1.1 XML for Configuration & Data Files.

To be completed.

XML is ideal for structured config files. Combining a standard syntax with parsing libraries and XML structure editors will make system admin more pleasant.

4.2 Making Everything Click

To be Completed.

Definition of the role of terminal emulators. Definition of their external interfaces, including user events and escape syntax for incoming XML.

4.3 Why XML?

To be completed.

Detailed justification for XML. (It's a Standard. It allows terminal emulators to ignore tags since it is a mark-up language. It has a DTD for defining the syntax of object streams. There is much free XML stuff around. XML will also do for conf files. XML structure editors abound and will become a 'universal' editor for UNIX data streams. etc)

5. Design

To be completed.

Detailed specifications of interfaces (down to the bit).

5.1 DTDs

To be completed.

Definitive repository of standard UNIX DTDs for object streams.

6. Implementation

To be Completed.

Starting with a hacked-up prototype for initial play and usability testing, this will list and refer to downloadable LINUXML software and docs.

Bill Birch, Sunday, February 07, 1999