[September 05, 2001] The Software Component Laboratory (SCL) "is a joint project between The University of Western Australia, Murdoch University, Software Engineering Australia (Western Australia) and a group of software developers in Perth, Western Australia. It was established with three aims: (1) to promote the use of component technology through the provision of courses and seminars for industry, (2) to provide a library from which reliable software components can be purchased. These components are contributed by developers throughout Australia and overseas: most of them are accompanied by test certificates which enable purchasers to rapidly verify that purchased components perform to specification in the target environment. (3) to develop the Component Test Bed -- a tool which assists component developers to quickly and easily generate and manage test data sets for components. Many of the test certificates accompanying components in the library were generated with the Component Test Bench."
The SCL Component Test Bed "is a suite of software tools which allow authors to package and test software subsystems as components which may be sold through the Software Component Server. The CTB may be used by software developers who have no formal training in software testing. It provides a check list that the developer may follow to ensure that a component is thoroughly tested prior to submission to the server. It also creates a test database, consisting of test patterns which may be bundled with the components that are sold. The benefits of such an approach are twofold: (1) This benefits the customer since they may purchase verifiably correct software components for inclusion within other projects. The component may be tested shortly after purchase and so the customer may be confident that the component can be safely included within another project without expensive side effects later surfacing. (2) It benefits the developer since it adds value to the software product which is sold. It also minimises ongoing support costs since much of the testing and verification can be performed prior to release. The CTB provides a user interface and a suite of tools which enforce a thorough testing methodology on components. It provides Wizard-like capabilities to guide novice testers through the test process. It also contains automated test pattern generation and recording capabilities. The outcome of the test process may then be saved as an XML compliant database which may be bundled with the software component when it is sold. Finally the CTB, in concert with the Software Component Server, adds serial numbers to each of the components that are sold. This makes it easier for vendors to track license abuse of components which are sold through the SCS..."
From the IEEE Computer article 2001-09: "... we designed a test specification that aims to be (1) standard and portable; (2) simple and easy to learn; (3) devoid of language-specific features; (4) equally able to work with object-oriented systems, simple functions, and complex components such as distributed objects or Enterprise JavaBeans; (5) efficient at handling the repetitive nature of many test sets; (6) capable of offering widely available and easily produced test-generation tools that do not require proprietary software; (7) free of proprietary-software requirements for interpreting and running the tests; and (8) able to support regression testing. We based our test pattern document format on the W3C's Extensible Markup Language, which satisfies most of our requirements. XML is a widely adopted general-purpose markup language for representing hierarchical data items. We have defined an XML grammar, specialized for representing test specifications, published in the form of a document type definition (DTD) that can be downloaded from our Web site. XML is well suited to representing test specifications because it adheres to a standard developed by an independent organization responsible for several other widely accepted standards. It has achieved broad acceptance across the industry, leading to the development of editors and parsers for a variety of platforms and operating systems. Further, XML's developers designed the language to provide structured documents, which support our test specifications well. XML documents -- laid out with some simple rules -- can be read and interpreted easily. Several readily available editors make understanding the language easier by highlighting its structure and providing various logical views. To keep the test specification simple and easy to use, we defined a minimal number of elements for it. Rather than adding elements to support high-level requirements, we allow testers to write helper classes in the language of the system they are testing. This approach gives testers all the power of a programming language they presumably already know and avoids forcing them to learn an additional language solely for testing...The specification uses the terminology of object-oriented designs and targets a class's individual methods. However, it can describe test sets for functions written using non-OO languages such as C or Ada equally well. As long as a well-defined interface exists, a tester can construct MethodCall elements..."
See the VeriLib Component Test Set Description DTD: "This DTD comprises the necessary data to describe both Test Specifications, Result Sets, and the old combined Component test specification. The DTD is separated into three parts, being the Test Set data, Result Set Data, and the addition glue from the Component test specification."
The CTB "Guide to Preparing Test Specifications" is set of notes to aid in the preparation of XML test specifications for running through the Test Pattern Verifier. It supplies an overview of the main elements used in the specification DTD: (1) <Component Name="specification_name"> - 'Component' is the top level element of the XML document. Use the Name attribute to name this test specification; (2) <Implementation> describes the implementation of the component: at this stage, only Java implementations can be tested. (3) <Interface TestSetName="test_set_name "> describes the interface - it assumes that multiple interfaces may be provided by the same component. test_set_name must match the Name attribute of a test set. (4) <TestSet Name="test_set_name"> - A test set should contain a logically related set of tests, eg tests for one component or class. Test sets may have any number of 'TestGroup' elements within them. There should only be only 'TestSet' per file. (5) <TestGroup> - A group of tests within a test set. Related tests should be organized into a hierarchy of test groups with appropriate names: the TPV will display the value of the Name attribute in its selection panel. The test group hierarchy may be used in any convenient way to group tests: one strategy might place all Operation's testing one method within a test group. (6) <Operation> - A set of constructors and method calls constituting a single test. An operation would normally represent a test of one equivalence class, i.e., you could have one operation to test 'standard' cases, one to test boundary cases etc."
References:
VeriLib Component Test Set Description DTD. This DTD defines the format for test specifiers which define component tests and can be executed by the Test Pattern Verifier. [cache]
[September 04, 2001] "Software Component Certification." By John Morris, Gareth Lee, Kris Parker, and Gary A. Bundell (University of Western Australia); Chiou Peng Lam (Murdoch University). In IEEE Computer Volume 34, Number 9 (September 2001), pages 30-36.
"A Software Component Verification Tool." By Gary A Bundell, Gareth Lee, John Morris, and Kris Parker (Centre for Intelligent Information Processing Systems, Department of Electrical and Electronic Engineering, The University of Western Australia; Email: {bundell,gareth,morris,kaypy}@ee.uwa.edu.au); Peng Lam (School of Engineering, Murdoch University, Australia). 10 pages. Abstract: "Component-Based Software Engineering depends on reliable, robust components, since it may omit a unit test phase wholly or partially from the development cycle. This paper describes a tool that allows a component developer to design and run verification tests. In developing components for our library, we found it necessary to provide multiple mechanisms for identifying and capturing tests to overcome the limitations of any single mechanism. Once specified, test specifications and test results are stored in XML documents, providing a standard, portable form of storing, retrieving and updating test histories. One module of our component test bench, the test pattern verifier, has been designed to be general, lightweight and portable, so that it can be packaged with a component and its test specifications. This allows a component user to verify a component's compliance with specifications in a target environment... Our CTB has been used to generate test specifications for several components available through our component server, VeriLib. Clients are able to download a package containing the component (compiled code and, optionally, source) as well as the XML test specification. The package includes a module from the CTB which runs the tests, compares output with the expected output and output from previous runs. Clients downloading components are able to immediately determine whether the component performs according to its specification in the target environment by running the test specification. They are also able to use the test specification to gain additional detailed information about the actual functions of the component as it is an XML document which can be read directly or -- in the case of large documents -- by using any one of a number of freely available XML editors to work through the structure selecting portions of interest... A major contribution of our work to this process is the coupling of components and test certificates -- enabling rapid verification that components do have the required capabilities and that upgrades have not affected correct functions. We define a test descriptor in XML -- a simple, portable doc-ument standard -- which permits test specifications to be read and interpreted on a wide variety of platforms. The tool we describe here contains a test pattern verifier - a light-weight, general and portable module which will run as a stand-alone package. Thus it can be packaged with a component and its test specification - enabling a component user to confirm correct functioning of the component in any target environment..." [cache PDF, Postscript]
"Component Certification." By John Morris, Gareth Lee, Kris Parker, Gary A. Bundell, and Peng Lam. 5 pages. Abstract: "Component-Based Software Engineering relies on sources of reliable components: developers need to be able to trust the components with which they build larger systems. This has led to suggestions that software certification laboratories should undertake the role of checking reliability of components offered for sale. This paper argues that is it more effective for developers to supply the test certificates in a standard, portable form, allowing purchasers to examine and run the test sets themselves and thus to form an opinion as to whether the component will meet their needs or not. By removing necessarily expensive third-party certification exercises, one of the major potential economic benefits of CBSE -- the ability to build a complex system from inexpensive, but trusted, components will be retained. Our specification is depicted in tree form [as] formally described by the document type definition (DTD). XML enables us to meet [our] requirement #1: (1) it has a standard developed by an independent organisation responsible for a number of other widely accepted standards; (2) it has achieved wide acceptance; (3) editors and parsers are available on a wide variety of hardware platforms and operating systems; (4) it was designed to provide structured documents and thus matches the requirements of our test specifications well. XML documents -- laid out with some simple rules -- are easy to read and interpret. Understanding is made easier by several readily available editors which highlight the structure and provide various logical views -- satisfying part of requirement 7. By defining a minimal number of elements in the test specification, we kept it simple and easy to use... Test Pattern Verifier: Component users must be able to run the tests described in a test specification. We have developed a lightweight, portable program -- the test pattern verifier (TPV) -- which reads XML test specifications, applies the tests to a component and checks results against those in 'Exp' elements or previously stored in 'ResultSet' documents by the TPV... Our approach which involves developers providing their owntest data to component purchasers has many advantages over the certification laboratory approach: (1) Costs are reduced: the incremental cost to developers is small. They have produced extensive tests as part of their own verification procedures; without them, they cannot make any claim for reliability. (2) There is no need for trust: purchasers are supplied with the test data, means to interpret it (the XML DTD can be used by most XML editors to display the test specification's structure and content) and a means to run the tests and verify that the developer's claims for correctness are sustainable. (3) Any reliability level claimed by the developer as a result of his testing can be confirmed by examination the tests for conformance with the purchaser's understanding of the specification and completeness. (4) The test specifications augment the -- usually natural language and therefore laden with potential forambiguity -- functional specifications. The test specifications and accompanying actual results provide a precise (if voluminous!) specification for the actual behaviour of the component..." [cache PDF, Postscript]
Component Technology White Paper. Part II, Technical Aspects. By Gary A. Bundell, Gareth Lee, John Morris, Stuart Hope, Shawn Parr, and R. Geoff Dromey. 21 pages. Background for the CTB project. Abstract: "Component software is widely recognised as the key to further improvements in software productivity, reliability and exibility. This paper reviews the state of the art in component technology and discusses the benefits of the component approach for developers, focussing on the technical benefits in increased productivity and reliability. In defining a software component, we require 're-use' and 'substitutability' and argue that these two capabilities capture the key properties of productive components. This definition enables us to build a taxonomy of components -- arranged in a hierarchy with increasing capabilities. The characteristics of each class in this taxonomy are discussed and a simple example -- a lookup table --is used to illustrate how each new capability may beintroduced into a simple component. The infrastructures needed to support distributed components are described. The development, verification and maintenance of components are also discussed..." [cache]
"Using Symbolic Execution to Guide Test Generation." By Gareth Lee, John Morris, Kris Parker, Gary A. Bundell, and Peng Lam. March 13, 2001. 9 pages. Abstract: "This paper examines the efficacy of symbolic execution as a method for automatic test pattern generation. We focus on the creation of test patterns for software components written in Java. A number of shortfalls of symbolic execution have been discussed in previous publications, such as dealing with loops, method calls, impossible paths and algebraic simplification. We present practical solutions to several of these problems which we have implemented in the form of a software component testing system. We also present a novel approach for dealing with aliasing of array elements and references within symbolic execution... We believe that the approach is particularly applicable to the semi-automated testing of software components, due to their limited size. Widespread reuse of components also allows the cost of testing to be amortized over a large user base. Our research aims to generalise symbolic execution to a point where it can be applied to any Java source code. Future work may allow symbolic execution to be used with other strongly typed languages. This work on symbolic execution has been part of a unified approach towards software component testing and marketing, which includes the development of a standard XML document format for the interchange of test pattern sets, such as those created by the CTB. The project also operates a software component trading web site, VeriLib. We encourage our contributors to test their software components using the CTB and the symbolic execution techniques that have been described here. We believe components that have been rigorously tested will be at a commercial premium over coming years. For further information regarding the future development of the CTB we encourage readers to periodically visit the VeriLib site..." [cache
