[This local archive copy is from the official and canonical URL, http://www.ozemail.com.au/~birchb/linuxml/linuxml.htm, 1999-02-26; please refer to the canonical source document if possible.]
Welcome to the LINUXML project.
This project is devoted to changing the UNIX de facto standard for inter-process communication and storage from line-based ASCII records to XML. Linux is the reference platform.
This project originated from my daily frustration with xterm. It annoyed me that xterm didn't allow me to click on the ASCII words and symbols with my mouse and get things to happen beyond simple text cut and paste. My original whinge on this subject was posted on Usenet back in 1994. (see below).
Since then Linux has come to us and all, and XML is now appearing. My original thinking on this has matured and I realise that a deeper change to UNIX can remove other frustrations I have with UNIX. For example, getting "sort" to work on arbitrary record formats always is a pain and remembering umpteen different "etc/conf" file formats is taxing.
By now you'll want to know some more details.
In essence the idea is simple. Instead of a program like "ls" outputting its data in ASCII records, it outputs the directory listing in XML. This allows all the downstream programs, like "sort" and "xterm" to actually understand what the data is and do more useful things with it. Here's an "ls" output:
[birch@redhat birch]$ ls -F -xml /
bin/ etc/ lost+found/ root/ u01/ u04/
boot/ home/ mnt/ sbin/ u02/ usr/
dev/ lib/ proc/ tmp/ u03/ var/
In particular xterm can be programmed that when you click on a directory such as "/home", it tells the shell to "cd /home".
Here is another "ls" example
[birch@redhat birch]$ ls -l -xml /
total 53
drwxr-xr-x 2 root root 2048 Jan 19 04:35 bin
drwxr-xr-x 2 root root 1024 Jan 30 23:30 boot
drwxr-xr-x 2 root root 21504 Jan 30 12:30 dev
drwxr-xr-x 25 root root 2048 Feb 2 12:59 etc
drwxr-xr-x 16 root root 1024 Feb 2 12:58 home
drwxr-xr-x 4 root root 2048 Jan 19 04:31 lib
drwxr-xr-x 2 root root 12288 Jan 19 04:08 lost+found
drwxr-xr-x 4 root root 1024 Jan 19 04:09 mnt
dr-xr-xr-x 5 root root 0 Jan 30 23:30 proc
drwxr-x--- 6 root root 1024 Feb 2 13:00 root
drwxr-xr-x 3 root root 2048 Jan 19 04:39 sbin
drwxr-xr-x 21 oracle dba 1024 Feb 7 04:25 tmp
Likewise "sort" can be re-programmed to sort on the size directly since the XML output of "ls" explicitly identifies the size. i.e. instead of "ls -l / | sort -n +4" you have:
[birch@redhat birch]$ ls -l -xml / | sort -size
dr-xr-xr-x 5 root root 0 Jan 30 23:30 proc
total 53
drwxr-x--- 6 root root 1024 Feb 2 13:00 root
drwxr-xr-x 2 root root 1024 Jan 30 23:30 boot
drwxr-xr-x 4 root root 1024 Jan 19 04:09 mnt
drwxr-xr-x 15 root root 1024 Jan 19 04:39 var
drwxr-xr-x 16 root root 1024 Feb 2 12:58 home
drwxr-xr-x 19 oracle dba 1024 Jan 19 04:24 usr
drwxr-xr-x 21 oracle dba 1024 Feb 7 04:25 tmp
drwxr-xr-x 2 root root 2048 Jan 19 04:35 bin
drwxr-xr-x 3 root root 2048 Jan 19 04:39 sbin
drwxr-xr-x 4 root root 2048 Jan 19 04:31 lib
drwxr-xr-x 25 root root 2048 Feb 2 12:59 etc
drwxr-xr-x 2 root root 12288 Jan 19 04:08 lost+found
drwxr-xr-x 2 root root 21504 Jan 30 12:30 dev
That's it! The rest of this web site is all about achieving this goal.
I'm not a big fan of this word, it sounds pretentious but it seems to fit here.
Date: 24th April 1994
Subject: Do HyperTerminals Exist?
Summary: Or what do you get when you cross xterm, ksh and an OO-shell?
Synopsis: Requirement for a combined OO Shell and Terminal Emulator.
Hi,
In this letter I describe a tool that I would really like to possess.
My question to you, dear reader, is does such a wonder exist? If it
does not I challenge the programming community to implement one for
public consumption and enjoyment.
Introduction
~~~~~~~~~~~~
You are all experts with ASCII terminals and command-line interfaces
such as the bourne shell. You are also conversant with hypertext and
object-oriented file managers. What I want is an amalgamation of all
these with a solid OO framework. A hyperTerminal session may proceed
as follows:
Imagine you are using X-windows, on your screen is an xterm-like
terminal emulator window running a shell. You type "ls" and a list of
files is printed, neatly arranged in columns. With the mouse, you
double-click on one of the filenames (foo.cc). An emacs window is
popped up open on the file. You notice that unseen hands have entered
the command "emacs foo.cc &", and the prompt has been re-printed.
Now you move the pointer over another file (bar.cc) and you
click-and-hold the mouse button. You drag the filename (which follows
the pointer) until it is over another filename (dir/). You release the
mouse button, and "bar.cc" disappears. You notice that unseen
hands have again been at work, having typed "mv bar.cc dir".
You double-click over the "dir/" text and the command "cd dir; ls" is
typed for you. Your prompt changes to show the new directory and a
directory listing is printed as before. You notice that "bar.cc"
appears among the entries.
What is a HyperTerminal?
~~~~~~~~~~~~~~~~~~~~~~~~~
A HyperTerminal provides all the functions of a standard xterm yet it
also understands the information it displays. HyperTerminal displays
a continuous stream views of underlying objects. It still manages to
preserve the familiar textual presentation of an ASCII terminal.
Since it is displaying views of objects, the objects can be directly
manipulated by the user with a mouse.
Commands such as "ls" are re-written or wrapped so that they output a
stream of view objects rather than a stream of characters. Hyperterm
can be programmed to understand all classes of objects ranging from
fragments of a text file, through to a UNIX process.
Users can browse the object classes with a graphical browser,
re-structuring inheritance relationships and defining shell scripts to
be executed for each method the object class supports.
Details
~~~~~~~
Here are some of the things that a HyperTerminal could do:
* Fragments of ASCII files output by commands like "head", "grep" or
"cat" can be edited by positioning the cursor over the text, and
inserting or overtyping characters. The changes are committed to the
files.
* Directories displayed with "ls -l" can be modified by positioning
the cursor over the relevant attributes and over-typing. Filenames,
permissions and dates can be modified by altering the corresponding
part of the output.
* Files can be moved between directories by drag-and-drop.
* File links can be established by drag&drop with the shift key held.
* Files can be deleted by selecting of a filename followed by hitting
the delete key. a confirmation dialogue may appear depending upon
options.
* Programs such as compilers output references to fragments of ASCII
files. Double-clicking on an error message opens an editor on the
relevant file, positioned at the line in question. HyperTerminal
keyboard focus enables you so step between error messages with the
cursor keys.
* Job control and process management is simplified. For example,
selecting processes displayed with the "ps" command allows you to
kill the process with the delete key.
* Non-wrapped commands output straight text which HyperTerminal
treats identically to the ASCII terminal it emulates.
By now the general character of HyperTerminal is becoming clearer. It
allows you to use all the power of the scripted interface (and all
those familiar commands) yet providing object-oriented direct
manipulation of the results of those commands.
Comforts
~~~~~~~~
Because HyperTerminal belongs to that large family of windows-based
programs, it comes with the usual framework which includes:
* Pull-down menus from a menu-bar. These allow you to call up generic
methods shared by almost all objects. Some of the standard ones might be:
File
New...
Open
Copy
Move
Delete
Print...
Properties...
Run
Save // saves the current session a la "script"
Edit
Cut
Copy
Paste
Options
Display
Class Browser
...
Window
Help
* The right mouse button would support a pop-up menu showing all the
actual methods supported by the selected object(s)
* Selection of many objects via the shift key, polymorphic method
execution over the selected objects.
* HyperTerminal would also provide users with many presentation
options for each class including: font, text style, colour and use of
icons. For example executable files might always be displayed in Red.
* Because HyperTerminal is pretending to be an ASCII terminal on X,
it could also be induced to display graphical representations of
objects. Thus icons could be used to display files in the output of an
"ls" command; tick boxes might replace permission switches.
Class Definitions
~~~~~~~~~~~~~~~~~
HyperTerminal is extensible allowing users to add object classes and
methods of those classes. It supports inheritance and polymorphism,
and maybe multiple inheritance. To make this easy, it comes with
a graphical class browser.
Dynamic Behaviour
~~~~~~~~~~~~~~~~~
Since HyperTerminal continues to know about the under-lying objects
it is displaying, it could update the display to reflect the execution
of subsequent commands or change to show the state of the objects.
For example, deleting a file may cause it's entry in previous "ls"
output to change into passive, black text.
Practical Considerations
~~~~~~~~~~~~~~~~~~~~~~~~
Despite offering more functions than xterm, it would not consume much
more than "xterm" and "sh" combined. (The class browser would probably
be a separate process.) It would be fast and efficient, allowing users
to run dozens of HyperTerminal windows.
The source code of Hypterterminal would ideally be freely available
along with ports to most UNIX systems.
Hyperterminal would probably continue with the UNIX style of working
by interpreting a stream of characters on its standard input. A clever
arrangement of escape sequences would delimit the objects it displays
along with their class and identity. The sequences would be published
thus allowing users to write programs that output object views into
Hyperterminal.
Conclusion
~~~~~~~~~~
Ok so you have read my cursory description of the mythical
Hyperterminal. I hope it was reasonably clear :-). My questions are:
Does such an animal exist?
Would anybody else like one?
If there is one where is it, and how can I get one?
If there is nothing like Hyperterminal, I re-iterate my challenge to
the programming community, please, write one!
Bill
The new 'vision' for the project consists of combining the idea of 'object streams' and direct manipulation in the user interface. Rolling all the smarts into the user interface is an option, but we need to keep the magic of the "|" pipe alive. The original concept is still valid, but a more structured approach to getting there is needed. Remember this is 'vision' material so some of it will be beyond our immediate reach. There are some key parts which I believe we need.
We need to be able to describe the format of objects in object-streams. The format of the data should itself be standard. Why have different output data for every program (so "find" and "ls" could share the sae output format). XML appears to provide us the solution.
Meta-data is also a requirement. Format specs have to be machine readable for programs that need it. An obvious candidate meta-data format is XML's DTD, although it does lack detail on atomic data types.
Objects like files in UNIX are very standard. It makes sense to create common object models (DTDs) for all standard UNIX objects. This way it won't hurt much when people hard code these into their programs. For example no-one should be penalised for hard-coding expectations about the XML syntax of standard objects. Also, having a standard for common things should allow a lot of code re-use.
Applications that don't use standard UNIX objects must also be able to use the same object and meta-data stream formats.
It would be very cool indeed if programs like "sort" were able to dynamically load methods for objects they are working on. This would also apply to user-interface programs, which would need to dynamically discover the capabilities of an object class, and load the relevant methods.
LINUXML needs to be fully backward compatible with UNIX & Linux of yore. For example new versions of "ls" would have the XML output as a new command-line switch, new versions of "xterm" must work with straight ASCII. Also the role of producers like "ls", filters like "sort" and terminals should remain the unchanged..
To be completed....
I would love to provide an electronic storyboard of the vision for you here. Either a movie file or an animation would be great.
This is an Open Source project, and will be covered by the GNU license.
All forms of contribution will be welcomed. Currently the most useful contributions will be in the form of feedback on the vision and architecture. More concrete help in the forms of DTDs and source will be welcomed with open arms. Thinking an opinion on the issues list would also be helpful.
Email birchb@ozemail.com.au if you have some time.
To be completed.
A description of the relationship between the supplier programs like "ls", filters like "sort" , shells and user interfaces.
Current thinking places the user interface as being 'dumb'. It will know when objects have been clicked, and will convert user clicks into shell commands. eg xterm will transmit a message to a shell instructing it that the user had clicked on a directory. ("$ click /home") Currently, I see no reason to use XML on the input to a shell, mainly for backward compatibility reasons.
The shell will then deduce that a "cd" is appropriate and execute that operation. This places all the OO smarts in the shell (knowledge of object classes and methods). Allows early prototyping with existing shell languages. It could be that a Java shell will be the best?
To be completed.
Description of an object stream. Differentiation between an object (eg a file) and a view on the object (eg output from "ls"). Incorporation of model-view pattern.
To be completed.
XML is ideal for structured config files. Combining a standard syntax with parsing libraries and XML structure editors will make system admin more pleasant.
To be Completed.
Definition of the role of terminal emulators. Definition of their external interfaces, including user events and escape syntax for incoming XML.
To be completed.
Detailed justification for XML. (It's a Standard. It allows terminal emulators to ignore tags since it is a mark-up language. It has a DTD for defining the syntax of object streams. There is much free XML stuff around. XML will also do for conf files. XML structure editors abound and will become a 'universal' editor for UNIX data streams. etc)
To be completed.
Detailed specifications of interfaces (down to the bit).
To be completed.
Definitive repository of standard UNIX DTDs for object streams.
To be Completed.
Starting with a hacked-up prototype for initial play and usability testing, this will list and refer to downloadable LINUXML software and docs.
Bill Birch, Sunday, February 07, 1999