CP RSS Channel
About Our Sponsors
Articles & Papers
Technology and Society
[January 14, 2000] "Conformance Testing." By Martha Gray, Alan Goldfine, Lynne Rosenthal, Lisa Carnahan (National Institute of Standards and Technology). January, 2000. "With any standard or specification, eventually the discussion turns to 'how will we know if an implementation or application conforms to our standard or specification?' The following discussion defines conformance and conformance testing as well as describes the components of a conformance testing program."
[January 15, 2000] "XML 1.0 Specification Errata." Updated "2000/01/14" or later. "This document records all known errors in the XML 1.0 specification, http://www.w3.org/TR/1998/REC-xml-19980210. The errata are numbered, classified as Substantive, Editorial or Clarification and listed in reverse chronological order of their date of publication. Early errata (1999-02-17 and before) are neither numbered, classified nor dated." [2000-01-15 local archive copy]
The notion of XML "conformance" is addressed directly in the XML specification (Version 1.0, REC-xml-19980210) section "5. Conformance," which includes a discussion on "5.1 Validating and Non-Validating Processors" and another on "5.2 Using XML Processors."
[March 28, 2001] W3C and NIST Coordinate to Develop the DOM Conformance Test Suites. A posting from Philippe Le Hégaret (W3C DOM Activity Lead) announces the intention of the W3C DOM Working Group and NIST to jointly develop the DOM Conformance Test Suites. He writes: "This development will be entirely public. It will produce a DOM Level 1 Test Suite (based on the NIST DOM Test Suite) and a DOM Level 2 Test Suite. The DOM Level 3 Test Suite will be addressed after the publication of the DOM Level 3 Candidate Recommendation. Dimitris Dimitriadis will be the contact for the W3C DOM Working Group, and Mary Brady will be the contact for NIST. Both will participate in the Workshop on Quality Assurance at W3C. We welcome the participation of interested parties in developing the DOM Conformance Test Suites." A public DOM TS mailing list for the development of the test suite was created on February 23, 2001, and is open for subscription; its archives are also public. Note that a document DOM Conformance Test Suites Process Document edited by Dimitris Dimitriadis is available from W3C. [Full context]
See the primary DOM Conformance reference list.
[April 02, 2001] Two New Test Suites for XML Document Object Model (DOM) Level 1. OASIS and NIST have announced the addition of "two new test suites for the XML Document Object Model (DOM) Level 1 to the growing collection of OASIS Conformance Tests for XML Technologies. DOM Level 1 provides a standard set of objects for representing HTML and XML documents, a standard model of how these objects can be combined and a standard interface for accessing and manipulating them. The two DOM Level 1 Conformance Test Suites were developed by NIST and submitted to OASIS for the benefit of the entire XML community. They include both ECMAScript and Java tests. The ECMAScript DOM Test Suite is interactive, providing instant pass/fail results. Developers may view the source code of the test, link to the test purposes (semantic requirements) and link to the appropriate section in the W3C DOM Recommendation. The suite includes more than 800 ECMAScript tests, covering all fundamental, extended and HTML interfaces. The Java DOM Test Suite also contains more than 200 tests, covering all fundamental and extended interfaces. The tests are organized into a set of classes, one for each interface. The suite includes a test driver that accepts implementations, interfaces and output files." NIST (National Institute of Standards and Technology) is also collaborating with W3C in the development of DOM Conformance Test Suites. [Full context]
[June 14, 2002] W3C and NIST Release XML 1.0 (Second Edition) Conformance Test Suite. W3C has announced the release of a new XML Conformance Test Suite that any developer can download free of cost and use to test an XML processor's conformance to the XML Recommendation. Built in cooperation with NIST and formerly hosted by OASIS, the XML 1.0 (Second Edition) errata 20020320, W3C Conformance Test Suite 20020606 contains over 2000 test files and an associated test report. The test report supplies background information on conformance testing for XML as well as test descriptions for each of the test files included in this release. This new XML test suite provides a set of metrics for determining conformance to the W3C XML 1.0 Second Edition Recommendation, and is intended to complement the W3C XML 1.0 (Second Edition) Recommendation dated 6-October-2000. All interpretations of this Recommendation are subject to confirmation by the W3C XML Core Working Group. Implementors are encouraged to write a harness around these tests to test their implementation for XML conformance." [Full context]
[August 17, 2001] W3C Launches Conformance and Quality Assurance Activity. W3C has announced a new Conformance and Quality Assurance Activity designed "to solidify and extend current W3C quality practices regarding specification editing, validation tools and test suites, and coordination efforts within W3C." The activity will be supervised by Karl Dubost, W3C Conformance Manager. A W3C QA Interest Group has been formed as well as a QA Working Group. The QA activity will "work on the quality of W3C specifications, promote the development of good validators, test tools, and harnesses for implementers, and think ahead to additional steps. The main objective of the QA Working Group is to foster the development of usable and useful test suites endorsed by the W3C, which share a common look and feel, and ensure that the validating tools of the W3C are fully operational, useful and educational. The working group will seek to (1) improve the quality of W3C specifications with respect to conformance statement, test assertions, tutorial/examples, formal representation of languages, etc.), by conducting reviews of specifications and producing guidelines for specification writers; optionally, the WG can work on specification improvement, but this is not a required deliverable; (2) develop a common framework/harness for developing and running tests and a process for maintaining/adding/removing tests from test suites; reviewing existing test tools (3) ensure coordination with W3C Working Groups developing specifications (formal channel, appeal); (4) coordinate works with internal W3C horizontal groups: WAI, I18N, TAG, and Communications Team." [Full context]
[May 04, 2001] "The Matrix of W3C Specifications." From the W3C "Conformance and Quality Assurance" resource section [Conformance and Quality Assurance Activity]. Maintained by Karl Dubost. For each named W3C specification, the table (matrix) includes information on the specification status (Rec, PR, CR, WD, etc) and URL for an [existing] QA Log, online Validator, Test Suites, W3C Notes / Tutorials, and Conformance section.
[February 07, 2001] W3C Workshop on Quality Assurance [and Quality Assurance Activity]. The World Wide Web Consortium (W3C) announced a 'Workshop on Quality Assurance' at W3C. Registration is open through 28-March-2001 for this W3C Workshop on Quality Assurance, to be hosted by NIST in Washington, D.C. USA, on April 3-4, 2001. Participants can share their understanding of Web QA tools, conformance activities at W3C, and discuss a potential new W3C QA Activity. Position papers should be submitted to the Workshop Chairs by March 16, 2001. The main objective of the workshop, identified in the call for participation, "is to have W3C, its membership and the Web community involved in QA at large to share their understanding of the state of affairs for Web QA tools, technical and business practices and conformance activities at W3C or related to W3C specifications. Furthermore, as we're planning the start of a new W3C activity, one of our goals is to get feedback on the best course of action within W3C that would improve the quality of W3C specifications implementation in the field over time (i.e. what will be in the charter of this activity)." See the Quality Assurance (QA) Activity Proposal (draft). "Besides the shape to give this new potential W3C QA activity, there are several areas of interest related to Quality Assurance and Conformance of W3C technologies that we would like to hear about at the workshop: (1) experience in validation of Web content and documents - e.g., is this CSS page valid?; (2) online testing conformance of user agents - is this multimedia player correctly implementing SMIL1.0?; (3) quality of W3C specifications themselves - wrt conformance statement, tutorial, etc.; (4) conformance testing methodology - e.g., is this CSS page valid?; (2) online testing conformance of user agents - is this multimedia playe test design and components of a test suite; (5) certification/labeling of content, products or services; (6) common framework/harness for running test; (7) coordination with W3C Working Group developing specifications; (8) IPR and funding model." See also the Workshop Agenda, the Background Reference Material, and the W3C mailing list 'email@example.com'.
W3C has also provided for the creation of a W3C XML Schema Test Collection, announced by Henry S. Thompson (University of Edinburgh and W3C; Oriol Carbo, University of Edinburgh and W3C). "Goals and Objectives: The W3C XML Schema Test Collection work aims at coordinating test suites for W3C XML Schema processors created by different developers." The main objectives as announced 2001-05-02 are: (1) to integrate existing tests for W3C XML Schema processors in a common environment so they can be accessed publicly and shared among developers; (2) to establish a standard approach to test material IPR which meets the needs of both contributors and users; (3) to collect and develop tools to automate the execution and presentation of the test suites; (4) to offer a standard description of tests related to W3C XML Schema processors: [...]; (5) [to provide test descriptions] understandable by a developer without the need to actually view the test file(s) themselves); (6) to offer a standard presentation of test results; (7) to design additional tests and add/regularise descriptions of the existing tests; (8) in due course, to provide an XSLT-based approach to comparing XML representations of the post schema-validation infoset as produced by different processors; we will shortly announce the availability of XML Schemas for both the ordinary Infoset and the PSVInfoset. "The W3C expects to author only a small part of the collection -- we are counting on Member organisations and others to contribute the majority. To offer materials for the collection, please send e-mail to firstname.lastname@example.org." Note from Henry Thompson: "...the email@example.com mailing address is not a mailing list; it's for potential contributors to use to initiate discussions about contributions. For discussions of testing, I don't think we need a new mailing list; I'd expect firstname.lastname@example.org to be used for discussing W3C XML Schema testing..." See the discussion for the XML Schema Recommendation release.
[August 31, 2000] "MSXML Conformance Update". By Chris Lovett. From XML.com. August 30, 2000. ['In the past, XML.com has tested Microsoft's MSXML parser for XML conformance with less than glorious results. In this article, Chris Lovett presents the significant improvements made by Microsoft in MSXML in recent months.]' "This article is an update to previous articles by David Brownell on the conformance of the Microsoft XML Parser (MSXML). The July 2000 MSXML 3.0 Beta Release has made a significant improvement in conformance against the OASIS XML conformance test suite. Besides reporting the test OASIS conformance results, this article also reports on the compliance of the new Visual Basic SAX interface included in MSXML 3.0. To run compliance testing on this component I developed a brand new test harness in Visual Basic, which is also included with this article... The OASIS XML conformance test suite is a published set of tests, collected over time from various sources, which measure the conformance of XML parsers against the W3C XML 1.0 specification. It does not include any tests from Microsoft at this time. For my test, I downloaded the updated test suite that David Brownell published in February. This updated test suite takes into account the W3C errata for the XML 1.0 specification. I made two modifications to this suite... see xmlconf-20000821.zip for my updated version of the test suite. . . I used the same ECMAScript test harness that David Brownell published, except for one minor modification. This modification stemmed from the issue of what to do with tests marked 'valid' that have no DTD (document type definition) at all. David's test harness treated this issue in a manner contrary to the design of MSXML. . . MSXML still has some issues to resolve relating to non-existent or malformed unused entities, attribute-value normalization, end-of-line handling, and reporting validity constraints when running in non-validating mode. However, you can see from the following table that MSXML is on a steady march towards 100% compliance."
OASIS (The Organization for the Advancement of Structured Information Standards, formerly 'SGML Open') has set up an XML Conformance Subcommittee under the OASIS technical program, currently chaired by Mary Brady (NIST). The subcommittee will be developing an XML Conformance Test Methodology.
[March 05, 2001] OASIS Releases Second Edition of the XML Conformance Test Suite. A joint announcement from OASIS and the National Institute of Standards and Technology (NIST) describes the public availability of the OASIS XML Conformance Test Suite, second edition. The conformance test suite has been updated to be in synchronization with XML 1.0 Second Edition, Recommendation. The test suite has been produced by the OASIS XML Conformance Technical Committee in conjunction with the W3C XML Core Working Group, resulting in the development of "comprehensive testing scenarios which now integrate over 1000 new tests contributed by IBM and modified the test suite to be consistent with changes to the XML recommendation." [Full context]
[July 30, 1999] In July 1999, the OASIS XML Conformance Subcommittee published an XML Conformance Text Suite (Working Draft 12-July-1999), as described in a public announcement.
[September 16, 1999] "Conformance Testing for XML Processors." By David Brownell. From XML.com (September 15, 1999). ['Not all XML parsers are alike, especially when it comes to how carefully they follow the XML 1.0 specification. Dave Brownell tests Java-based XML parsers and lets you know how closely they conform. This multi-part article evaluates the results of testing a dozen XML processors (XML parsers) against the OASIS Conformance Suite to see how well they follow the XML specification.'] - "One indication of XML's success is that a dozen or so implementations of an XML processor exist. These processors, spanning a variety of programming environments, are at the core of a new generation of web tools that are revolutionizing the dynamic generation of HTML and enabling new types of web applications, including business-to-business data messaging. But are all XML processors doing what they are supposed to do? Will the tools built with those processors create rivers of interoperable messages and documents? Will they create islands of data that can only be used with a single set of tools? This article looks at most of the XML processors available today for use in Java-based XML systems and evaluates how closely they follow the XML 1.0 specification. We will provide you with the hard data from our tests, so you can independently evaluate every claim made in this article and reproduce the results yourself. The results leverage the freely available OASIS XML Conformance Test Suite. While that suite has only been published relatively recently, many of its key components have been well known to the XML community for over a year and a half; they shouldn't come as a big surprise for any implementor. Many of these tests have been used for basic quality testing in a variety of XML processors." For related references, see the announcement for the test suite, "OASIS XML Conformance Test Suite Now Available. International Consortium Provides Set of 1,000 Tests for XML Parsers."
[May 12, 2000] "XML Conformance Update." By David Brownell. From XML.com (May 10, 2000). ['David Brownell has updated the OASIS test suite to take into account the most recent XML 1.0 specification errata, and run his tests against Ælfred 2, MSXML3, Oracle's parser, Sun's parser, Xerces-J and XP. The open source parsers delivered an excellent showing, leaving Microsoft and Oracle lagging behind.'] "This article is an update to my earlier articles from 1999 that tested the XML 1.0 conformance of parsers. Since then, there has been notable development of the parsers, and updates to the XML 1.0 specification itself. With respect to the dozen Java parsers reviewed last year, there have been a number of interesting changes, both in their conformance and in their open-source status. Microsoft's MSXML parser has also received a significant update, with the release of 'technology previews.'... There appears to be a pause in the OASIS work on XML infrastructure conformance. The ball is temporarily in the W3C's court to address bugs in the XML 1.0 specification that have been reported by its implementors and users. I'm not sure there's enough dialogue in that process: my personal preference would be for the W3C to adopt the IETF approach, and insist on interoperability and conformance testing processes with positive results (and multiple full implementations) before making recommendations."
[November 17, 1999] Test Driver, from David Brownell. Dated September 20, 1999 [or later]. Pre-Built Source and utilities. "This package is a driver for a set of SAX and XML conformance tests, using databases such as the OASIS/NIST XML Conformance Test Suite. The SAX API is used to access an XML processor (parser), and supports testing of every requirement in the XML 1.0 specification. The database of test cases is used to exercise those requirements. Try using this with any SAX processor, and see how conformant it is! (These use a slightly bugfixed version of the OASIS/NIST test suite. In that suite are almost 1100 tests.) See my conformance testing article at XML.com for more information about the results of this testing. Below are some slightly more up-to-date results..."
[December 03, 1999] DOM ECMAScript Test Suite. Mary Brady (NIST, Conformance Testing) posted an announcement regarding the update of the NIST DOM ECMAScript test suite, accessible via http://www.nist.gov/xml/. "Click on DOM Test Suite. This suite includes ~900 ECMAScript tests that exercise the DOM Level 1 Fundamental, Extended, and HTML interfaces. You can view the results using IE5 by clicking on first the category, and then the particular interface. Options are available for displaying the source code, semantic requirements (which are simply axioms we glean from the spec to organize our thoughts), and the actual specification. Please let me know if you find this useful. We are in the process of generating equivalent functionality for the java binding. We are just about finished with the fundamental interfaces, and expect to have a first set, including fundamental and extended available in early January."
NIST XML/DOM Conformance Testing. Organized through NIST's Software Diagnostics and Conformance Testing Division [SDTC], under Mary Brady. "In order to enable the next level of web applications, we need a flexible method of defining, interacting with, and exchanging data for a variety of domain-specific applications. The Extensible Markup Language (XML) provides a standards-based approach to universal methods for defining and exchanging data. Using XML, one can create customized markup languages for exchanging information within their own domain. In addition, the Document Object Model (DOM) defines ECMAScript and Java bindings for interacting with both XML and HTML data, permitting dynamic creation and manipulation of web pages defined using these metalanguages. Virtually all application domains are looking to use XML and DOM to manipulate and exchange structured information. In addition, XML processors and support for the DOM API are beginning to appear in beta versions of popular web browsers and associated applications. As such, conformance of these products to the W3C Recommendations will permit robust, interoperable solutions." [local archive copy]
[February 23, 1999] G. Ken Holman posted an announcement for the new public information page of the OASIS XML Conformance Technical Subcommittee.
The XML Conformance Subcommittee was chartered in August 1997, and is chaired by G. Ken Holman (Crane Softwrights Ltd.). According to a presentation by Ken Holman "Report on XML Conformance" at the Seattle XML Conference (XML Developers Day, March 27, 1998), the subcommittee is now "at the requirements collecting stage, [since] many members are currently occupied with XML releases of their own and are learning themselves what are important requirements in this area." See the link to the presentation slides, below. The subcommittee expects to:
- "set out guidelines for conformance development based on established requirements
- set out a list of deliverables and associated schedule for completion to meet the established requirements while following the guidelines
- work to produce the deliverables for release to the entire XML community upon completion
- maintain, as required, the evolution of XML conformance issues and set out new projects to meet new requirements."
[September 11, 1998] Ken Holman prepared a "Report on XML Conformance" for delivery at the XML Developers' Day, August 20-21, 1998. The presentation slides for this Report on XML Conformance are available in HTML format.
[June 04, 1998] A report on the OASIS conformance work was also given by G. Ken Holman at the OASIS Technical Committee meeting held after the SGML/XML Europe '98 Conference (Hotel Sofitel, Friday, May 22, 1998. 9:00 - 15:00). According to Ken's report in this update, current technical work in the conformance committee includes the goal of producing an XML property set (not now a part of the XML 1.0 specification), since the property set might play some role in development of an XML conformance test suite. It is not yet known whether XML conformance might best be expressed in terms of something like RAST (Reference Application for SGML Testing, as defined by ISO/IEC 13673:1995), or the XML property set (and groves), or something else. Work in the OASIS committee (or aligned with it) towards the creation of an XML property set includes contributions by NIST, Alex Milowski, Eliot Kimber, and Gavin Nicol. NIST, through its Software Diagnostics And Conformance Testing Division, is funding the work of an invited expert Tim Boland (email@example.com) to assist in the construction of the XML property set and a larger methodology for an XML test suite. The goals of NIST and OASIS in this work are said to be in close correspondence.
Ken Holman supervised several working sessions on "XML Conformance" at the OASIS Summer Workshop in San Diego. Current priorities in the OASIS effort include: "1) A simple binary test, whereby a processor is deemed either to conform or not; 2) A test of output, whereby the processor is determined to pass, or not, the appropriate information to the application; 3) A test of resulting properties, whereby the processor is found to produce, or not, a predictable value for the properties." See also: "Report on XML Conformance" at the XML Developers' Conference, Montréal, August 1998. A provisional test suite and a document describing the methodology is to be presented at the XML conference in Chicago, in November 1998.
- Contact: G. Ken Holman
[September 14, 2000] San Jose XML DevCon '2000 - OASIS XSLT/XPath Conformance Public Meeting Monday, November 13 21:00-22:30, San Jose Room. " With the generosity of Camelot Communications, the OASIS XSLT/XPath Conformance Technical Committee will be holding a public informational meeting at the XML DevCon San Jose '2000 conference The OASIS XSLT/XPath Conformance Technical Committee invites attendees to a public session to discuss the scope and content of our work. This interactive session gives attendees an opportunity to comment or to ask questions of committee members and the work of the committee. From G. Ken Holman (Crane Softwrights Ltd.).
OASIS XSLT/XPath Conformance Committee. The chair of the XSLT/XPath Conformance Subcommittee is G. Ken Holman of Crane Softwrights Ltd. "The XSLT Conformance Technical Committee was formed to collect, develop and document a suite of tests for XSLT and XPath processors. With major contributions from Lotus and NIST, and help from other member companies, the committee is addressing the text of the W3C XSLT and XPath 1.0 Recommendations, establishing tests useful to those measuring the conformance of processors to the published W3C documents."
[June 19, 2000] "OASIS and NIST Take the Lead in XML Conformance." - "The OASIS Interoperability and Conformance Technical Committee, led by representatives from the National Institute of Standards and Technology (NIST), today announced the expansion of their ongoing XML conformance work. The widely used OASIS/NIST XML Conformance Test Suite (http://www.oasis-open.org/committees/xmltest/testsuite.htm) is now being modified to reflect the W3C errata to the XML 1.0 Recommendation and is being extended to include an additional 1,000 tests contributed by IBM. The Consortium also revealed that a test suite for DOM Java is soon to be released and that work is underway to produce test suites for XML Schema, XSLT, XPath and DOM/SAX. An ECMAScript DOM 1.0 test suite (www.nist.gov/xml) released by NIST as well as NIST-developed Java DOM tests and Sun-developed Java and SAX tests will form the basis for the OASIS DOM/SAX test suite. OASIS and NIST bring two important attributes to the XML conformance work. They are both vendor-neutral organizations, separate from the development of the core XML family of standards such as XML, XSL, DOM, XML-Schema and others. 'OASIS and NIST provide unbiased test suites, uninfluenced by any one product or company,' explained Norbert Mikula (DataChannel), chief technical officer of OASIS. 'Additionally, by being separate from the development process, we bring a fresh, independent perspective on what the XML recommendation says'."
- "Report on XML Conformance." Slides from the presentation of Ken Holman for the OASIS XML Conformance Subcommittee. Delivered at XML '98 Developers' Day, Seattle, March 27, 1998.
- [June 19, 1998] Subcommittee update information, from Ken Holman
- [August 20, 1998] "XML Conformance." Report from Ken Holman. In OASIS@Work - Summer Workshop 1998 Newsletter. [local archive copy]
- Canonical XML - specification of a subset of XML for use in testing XML parsers. From James Clark's collection of XML Resources; [local archive copy, 980126]
- XML Test Cases - a collection of test cases for testing XML parsers, from James Clark. The valid test cases in the collection include the corresponding canonical XML. [local archive copy, 980210]
- XMLTest - A Java program from James Clark. Updated May 14, 1998. Uses David Megginson's SAX API, and can be used with any parser which has a SAX driver. "XMLTest uses SAX to generate canonical XML, allowing any Java XML parser with a SAX driver to be tested with any XML test suite that includes canonical XML output for its test cases."
- Validate/Check XML! - A document with links to several online XML validation services. These tools allow you to validate/check XML through the use of a Web browser.
- RUWF - Well-formedness checker from the XML.com Web site
- STG XML Validation Form [Beta version, 1998-08-12].
- XML Syntax Checker from Frontier 5. Use blox or the Frontier 5.1.3 built-in XML parser to check well-formedness of an XML document.
- WebTechs' Validation Service now supports 'XML' - (possibly using the SP xmlwf code).
- XML well-formedness checker from Richard Tobin. Based upon the RXP XML parser. See also RXP Output for OASIS Test Suite
- XML validation service from Techno 2000 Project Inc., part of a Korean XML site.
- James Clark's SP, which has enhanced XML support in version 1.3 for parsing XML.
- James Clark's Expat (XML Parser Toolkit) is "a library for XML parsing in C. This is the parser being used to add XML support to Netscape 5 and Perl." Contains an executable xmlwf for checking XML well-formedness. Predecessor was called xmltok.
- [February 10, 1998] [Historical reference now: 'Update for Clark's XMLTok, an XML parser in C: it includes 1) a low-level XML tokenizer; 2) a non-validating XML parser built on the tokenizer; this has an API designed for integration into Web browsers; 3) a simple application xmlwf for testing the parser, which can test XML entities for well-formedness and generate canonical XML."]
- xmlwf - XML well-formedness checker. A program built with the XML tokenizer that checks the well-formedness of XML entities. See expat above.
- Related: CSS1 Test Suite. See also the announcement: "The World Wide Web Consortium Releases CSS1 Test Suite."
|Receive daily news updates from Managing Editor, Robin Cover.|