Proposed TC Charter: Testing and Monitoring Internet Exchanges

Proposed Charter for OASIS Testing and Monitoring Internet Exchanges Technical Committee

Subject:   Proposed Charter for OASIS Testing and Monitoring Internet Exchanges TC
From:      "Mary McRae" <mary.mcrae@oasis-open.org>
To:        <members@lists.oasis-open.org>,<tc-announce@lists.oasis-open.org>
Date:      Fri, 16 Nov 2007 18:00:21 -0500

To OASIS Members:

A draft TC charter has been submitted to establish the Testing and Monitoring Internet Exchanges (TaMIE) Technical Committee. In accordance with the OASIS TC Process Policy section 2.2: (http://www.oasis-open.org/committees/process.php#s2.2) the proposed charter is hereby submitted for comment. The comment period shall remain open until 11:45 pm ET on 30-November-2007.

OASIS maintains a mailing list for the purpose of submitting comments on proposed charters. Any OASIS member may post to this list by sending email to:

oasis-charter-discuss@lists.oasis-open.org

All messages will be publicly archived at:

http://lists.oasis-open.org/archives/oasis-charter-discuss/

Members who wish to receive emails must join the group by selecting "join group" on the group home page:

http://www.oasis-open.org/apps/org/workgroup/oasis-charter-discuss/

Employees of organizational members do not require primary representative approval to subscribe to the oasis-charter-discuss e-mail.

A telephone conference will be held among the Convener, the OASIS TC Administrator, and those proposers who wish to attend within four days of the close of the comment period. The announcement and call-in information will be noted on the OASIS Charter Discuss Group Calendar.

We encourage member comment and ask that you note the name of the proposed TC (TaMIE) in the subject line of your email message.

Regards,
Mary

Mary P McRae
Manager of TC Administration, OASIS
email: mary.mcrae@oasis-open.org
web: www.oasis-open.org
phone: 603.232.9090

Proposed Charter for Review and Comment

Charter for the "Testing and Monitoring Internet Exchanges" Technical Committee

1a. Name and Abbreviation

Testing and Monitoring Internet Exchanges (TaMIE) TC

1b. Purpose

Electronic collaborations over Internet between business partners (e-Business / e-Government) appear to be converging toward well-established types of message exchange patterns that involve both user-defined standards and infrastructure standards. At the same time, the notion of event is increasingly promoted for asynchronous communication and coordination in Event-Driven Architectures (EDA) that are considered as either complementary to or part of SOA systems. In both cases collaboration between partners or between components is achieved by the means of choreographed exchanges of discrete units of data — messages or events — over an Internet-based protocol. The TC will define an event-centric test case scripting markup and execution model for such systems.

In e-Business transactions as in EDAs, partners or components must agree on the use of a combination of standards in order to interoperate with each other. Typically, these standards can be classified into three layers:

Messaging infrastructure standards, ranging from transport level to higher-level messaging protocols and quality of service (QoS) including reliability and security, such as those defined as SOAP extensions, or REST (Representational State Transfer).
Multi-message exchange standards as manifested in business processes and choreographies. Included standards may conform to UMM business transaction patterns, ebXML Business Process Specification Schema (BPSS or ebBP), WS-Choreography or WS-BPEL.
Business document standards. These may be business content structure and semantics, taxonomies in use, code lists, semantic rules, or the XML schema modeling style. They are industry-specific (e.g., RosettaNet PIP schemas, AIAG Inventory Visibility and Interoperability schemas,), horizontal document standards (e.g., based on UN/CEFACT Core Components, OAGIS Business Object Documents schemas), or regional guidelines (e.g., KIEC's e-document modeling guideline).

There have been conformance and interoperability test suites (e.g., ebXML messaging V2 test suites, WS-I test assertions) and testing tools (e.g., ebXML IIC [OASIS ebXML Implementation Interoperability and Conformance (IIC) TC], NIST QOD, NIST Business Document Content testbed, KorBIT ebMS testbed, WS-I test tools) for each above layer individually. But the testing of integrations of standards has been ad-hoc (e.g., RosettaNet Self-Test Kit is tied to RosettaNet protocol), or limited mostly to standards in the messaging infrastructure (WS-I).

Although the need for testing some form of integration of standards has been well recognized for infrastructure standards (e.g., WS-I profiles), there has been little support for testing integrations that extend to the use of standards specific to a business — e.g., for documents or choreographies. Such integrations can be construed as user-defined profiles. For example, the level of QoS required for a business transaction may depend on the nature of business data being exchanged, or on some property defined by the related business process.

Testing and monitoring these infrastructure layers and their integration also requires that test cases access a combination of contracts — agreements or policies — represented by meta-level documents (WS-Policy, WSDL, ebXML CPPA, ebBP definitions, XML schemas).

This compliance objective goes beyond quality assurance for the messaging function: it requires the monitoring of live transactions in production environments, as well as verifying conformance of business endpoints in operation conditions. This calls for a flexible test execution model that can accommodate performance constraints as well as different timeliness constraints — e.g., where tests are either deferred over log data, or executed on live exchanges in a monitoring mode.

The output of a monitoring script also must provide more information than a report of the type pass / fail. Different ways of "passing" or "failing" must be reported on, as well as identifying the types of business transactions. This reporting must be easy to consolidate, for Service Level Agreements (SLA) and Business Activity Monitoring (BAM) purposes.

The TC will define a testing and monitoring model, as well as a test script markup, so that test cases or monitoring cases can be fully automated, and portable across test environments.

1c. Scope

The scope of activity for this TC must be within the following topics:

Use Cases and Requirements: Investigation of use cases that may combine standards — or specifications submitted to an SDO — in all areas concerned by message exchanges: protocols, QoS, choreography, documents and business content. Selection and prioritization of related requirements.
Test Execution Model: A model for executing test cases or monitoring cases addressing the previously described requirements will be defined. In particular, an event-centric approach such as the one described in the event-driven Test Scripting Language (see Input material) will be considered. This execution model also includes a representation and access model for log data.
Test Case Scripting: An XML markup that complies with the execution model will be designed. This may reuse or profile subsets of existing dialects that are already in use for domains that have similar functional requirements (if their IP status is compatible with the TC IPR mode), in which case the markup will act as a coordination or integration language, treating these dialects either as language extensions or as external resources used via adapters. In particular, openness to the reuse of XML processing dialects and tools based on recognized standards — such as XPath, XSLT, XQuery — is a requirement.

The design approach will favor:
1. simplicity of use: a small number of constructs that are easy to learn and to implement as opposed to feature-rich dialects
2. modular units of scripts that allows for functional reuse and extensibility
Test scripts will process run time materials (i.e., messages, events, signals), as well as configurative artifacts such as policies and agreements, schemas and interface definitions. Test cases or monitoring cases will also provide for more detailed outputs than pass/fail, and will strive to support SLA monitoring, BAM (Business Activity Monitoring) and Event-Driven Architectures (EDA). However, the objective will not be to specify a BAM system or an SLA enforcement system, but only the monitoring technology that can support these.
Evaluation and Investigation: of third-party markups and dialects, log formats and existing test practices, for possible reuse or interfacing in the test case scripting language. However no third-party specification or tool can be made essential or necessary to an implementation of the specification to be delivered, unless its licensing is free and unrestricted.
Methodology and Guidelines: In scope of this activity, are all forms of support for users, such as example scripts, best practices, primers and guideline documents, concerning all topics listed above.

Initial Input Material

The following documents are input material to this TC, which will deserve prime attention from the TC assuming their IP status is compatible with the IPR mode of the TC:

The Event-Driven Test Scripting Language (eTSL) Version 0.85 [at] http://www.oasis-open.org/committees/download.php/26036/eTSL-draft-085.pdf
Conformance Testing and Certification Framework (OASIS, Conformance TC, June 2001)
QA Framework: Specification Guidelines (W3C, November 2004)
Test Assertion Documents for WS-I Profiles (WS-I, 2003-2005)
OASIS IIC Test Framework 1.1

Other documents may be considered by the TC, subject to a TC decision.

Explicitly out-of-scope:

Defining or specifying detailed test case metadata or artifacts that would complement the above scripting — e.g., such as supported in ATML, for representing test environments, complete test suites, and/or their result.
Definition of particular test harnesses.
Use cases and requirements that refer to infrastructure specifications that are not submitted to an SDO, or that refer to business documents or content that are not approved by an organization or consortium representative of this business domain.

1d. Deliverables

The TC will produce the following deliverables:

A requirements document, which may include use cases for Internet exchanges, test assertions for related standards, references to existing test case dialects or existing logging formats or systems.
A specification defining a test case execution model and scripting, that supports both the testing and monitoring of message and business data exchanges.
A set of examples and use cases illustrating the use of above specification for testing or monitoring business exchanges based on (a) some business content standards, (b) some choreography standards, and (c) some specifications in the domain of SOAP messaging or others.
An implementation of the above specification used for proof of the proposed concept and principle.

Timeline

The TC will aim at a Public Review draft of the Test Case Execution Model and Scripting by the end of 2008.

1e. IPR Mode

Royalty-Free on Limited Terms

1f. Audience

The TC is welcoming any OASIS member with an interest and/or experience in testing and monitoring.

1g. Language

English

Part 2: Additional Information (Non-Normative)

2a. Similar Work

Identification of similar or applicable work that is being done in other OASIS TCs or by other organizations, why there is a need for another effort in this area and how this proposed TC will be different, and what level of liaison will be pursued with these other organizations

In general the following works fall short of addressing TaMIE charter, in that they target either:

a specific protocol stack
a particular domain of business

Their support for testing and monitoring the implementation of a combination of standards is either inexistent or limited, and they do not provide a versatile, extensible scripting of test cases. The requirements that are specific to B2B environments are not well addressed either: none is addressing both run-time monitoring of systems in production, and conformance testing prior to deployment. Only the latter is generally supported by existing work, e.g. neglecting real-time monitoring cases that may lead to dynamic notification or remedial actions, or ignoring real-time constraints about events throughput and recovery after shutdown time.

These test environments below — with the exception of WS-I tools — do not support the combined processing of electronic documents representing agreements or policies (metadata) and of run-time logs. Finally, none supports a general-enough notion of event, along with adequate time primitives and event correlation and search capability.

(a) The OASIS ebXML IIC Test Framework (TF v1.1)
http://www.oasis-open.org/committees/download.php/10896/IIC_ebXMLTestFramework.zip

This framework was designed specifically to test the conformance and interoperability of the software implementing the ebXML Messaging Specification. The framework defines components and harnesses (configurations) necessary to simulate and control the environment for conformance and interoperability tests.

A shortcoming of the IIC TF is that it has been designed for ebXML messaging 2.0, and provides weak extensibility options, both for external events and for specialized evaluations. The IIC TF does not support other suites of B2B integration standards such as other SOAP profiles integrating several Web Services standards. It also does not support ebMS 3.0 and electronic documents testing.

(b) RosettaNet Self-Test Kit (RNSTK)

RNSTK is a test system provided by the RosettaNet consortium. The system allows software solutions to perform self-tests for compliance to business collaboration scenarios and the RosettaNet Integration Framework (RNIF) specification. The RNSTK tests an integrated system implementing particular B2B collaboration scenarios, yet is tightly dependent on RNIF messaging. RNSTK test suites are encoded using the RNIF specification itself. Due to these reasons, the RNSTK cannot be used to test other suites of B2B standards including the Document Type Definition semantics. Another weakness is that its architecture only supports conformance test configurations.

[rcc addition:] See the Masters Thesis supervised by Per Johan Runeson: http://serg.telecom.lth.se/education/master_theses/docs/72_Kjellin_slutrapport.pdf = The RosettaNet Standard and Compliant Platforms. A study of the RosettaNet Business-To-Business communication standard and its role in business and technical integration platforms, by Albin Kjellin, Lund University Department of Communication Systems, 2005; 51 pages. Cache version.

These tools do not allow for the scripting of a test suite; instead they apply a battery of predefined tests to any Web service material provided as input. The objective is to verify the conformance of this material to one of the WS-I profiles. The tools only passively analyze message traffic and have no ability to control the systems under test. Nevertheless, this capture and analysis architecture is un-intrusive, simple, and supports deferred testing. The analyzer tool of WS-I cannot be used as a general test engine, being tied to profile definitions; but, it can be leveraged as a specialized module by a more general monitoring capability, when the conformance of message material to WS-I profiles is required. [Basic Profile 1.1 Second Edition]

(d) TTCN-3 (Testing and Test Control Notation Version 3, ETSI, June 2001)

TTCN-3 (http://www.ttcn-3.org/StandardSuite.htm) is a powerful, programmatically complete procedural language with specialized constructs for defining test procedures, test verdicts, matching mechanisms for evaluation, timer handling and distributed test components. Due to its original focus on telecommunication systems, it also has the ability to specify encoding information, to communicate both synchronously and asynchronously, and to perform passive monitoring. These capabilities are all essential for testing in message-based B2B integration. Relative to previous test frameworks, TTCN-3 is more powerful and flexible. However, TTCN-3 notation can be difficult for a business or domain expert to use. TTCN-3 has not been designed to delegate some functions to external modules and it is weak when it comes to validating business transaction events and XML electronic documents, which are essential for B2B integration testing.

(e) ATML (Automatic Test Markup Language, IEEE, December 2004)

ATML (http://grouper.ieee.org/groups/scc20/tii/ATML/Working%20Groups/Management/ATML%20Overview.doc) has been designed for exchange of diagnostic information, configuration data, and test instrument definitions between conforming software applications. It was not designed to support test scripting and execution. However, ATML representations could be complementary to executable test case scripts.

(f) General Software Test Environments and Scripting JXUnit

JXUnit is an XML-based test-scripting system built on top of the JUnit Java classes. It is a data-driven testing system in which input to the system under test and expected output are specified and then the actual output is evaluated. There is no built-in support for B2B testing, in particular the communication and the event-driven tests. However, as a general, test-scripting platform that relies on a common programming language, JXUnit and JUnit could be used as an implementation platform for essentially any test framework including the one proposed here.

[References:

JXUnit: Building Suites of Test Data with XML. "JXUnit is a directory-driven test scripting system which builds on JUnit."
JUnit. "JUnit is a simple framework to write repeatable tests. It is an instance of the xUnit architecture for unit testing frameworks."]

2b. First Meeting

Tentative date is January 30, 2008. The first meeting will be a 2-hour conference call. Fujitsu or KIEC will sponsor the call.

2c. Projected Meeting Schedule

A 90-minute meeting every two weeks will be scheduled, unless the TC decides upon a different frequency. Assuming the first meeting is a conference call, a face-to-face meeting will be set within 2 months of the first meeting once the work has actually started, to maximize the value of face time.

2d. Co-proposers

AIAG (Automotive Industry Action Group)
Mohammad Abidi, Mabidi@aiag.org

Axway
Dale Moberg, dmoberg@axway.com

Fujitsu
Jacques Durand, jdurand@us.fujitsu.com

KIEC (Korea Institute for Electronic Commerce)
Hyunbo Cho, hcho@postech.ac.kr

NIST (National Institute of Standards and Technology)
Nenad Ivezic, nivezic@nist.gov

OAGI (Open Applications Group, Inc.)
Dave Connelly, Dconnelly@openapplications.org

2e. Convenor

Jacques Durand, Fujitsu

2f. Related Member Section

N/A

2g. List of Contributions

The following work, already submitted in October 2006 to the OASIS ebXML Implementation, Interoperability and Conformance TC, is also contributed to this TC:

eTSL draft V0.85, (originally derived from the TestFramework 1.1 produced OASIS IIC, 2004-2005)

2h. Draft of F.A.Q.

Will be provided later

2i. Proposed Working Title for Deliverables

Deliverable #1: eTSM — Event-Driven Test Scripting and Model. A specification defining a test case execution model that supports both the testing and monitoring of message and business data exchanges, and the scripting language for defining and executing these test cases.

Deliverable #2: eTSM Use Cases Document — A set of examples and use cases illustrating the use of above specification for testing or monitoring business exchanges based on: (a) some business content standards, (b) some choreography standards, and (c) some specifications in the domain of SOAP messaging or others. This document may serve to illustrate requirements that have been addressed.

Deliverable #3: [eTSM Implementation] — An implementation of the above specification used for proof of the proposed concept and principle.

Prepared by Robin Cover for The XML Cover Pages archive. See the plain text source message in the list archive. Additional information on the related specifications is presented in the Cover Pages news story "OASIS Members Propose New TC for Testing and Monitoring Internet Exchanges."