SEARCH
Advanced Search
ABOUT
Site Map
CP RSS Channel
Contact Us
Sponsoring CP
About Our Sponsors
NEWS
Cover Stories
Articles & Papers
Press Releases
CORE STANDARDS
XML
SGML
Schemas
XSL/XSLT/XPath
XLink
XML Query
CSS
SVG
TECHNOLOGY REPORTS
XML Applications
General Apps
Government Apps
Academic Apps
EVENTS
LIBRARY
Introductions
FAQs
Bibliography
Technology and Society
Semantics
Tech Topics
Software
Related Standards
Historic
|
| News: Cover Stories |  |  |
| Web Services Test Forum (WSTF) Addresses End User Interoperability Scenarios. |
Contents
On December 8, 2008, several industry partners announced public access to a Web Services Test Forum (WSTF) designed as an open community to improve the quality of the Web services standards through interoperability testing. The Forum will provide a customer-centric focus driven by end user testing scenarios.
WSTF members identified in the initial announcement included Active Endpoints, AIAG, Axway, CISCO, eviware, FORD Motor Company, Fujitsu, Hitachi, IBM, Oracle, Red Hat, Software AG, Teamlog, and TIBCO Software Inc.
Through WSFT services, "members of the Web Service community can develop interop scenarios as well as test those scenarios against other Web Service implementations. WSTF also provides a common testbed of regression tests that the community can use during the developmen of their Web Service implementations."
The Web Services Test Forum has a lightweight organizational structure: no Board, no centralized authority, no dues, few barriers to participation, and no allowance for IPR encumbrances (royalty-free licensing is required). Individuals as well as corporate entities are welcome. Anyone with an interest in Web Services may join the group provided willingness to sign the WSTF Participation Agreement.
Documents on the WSTF web site are freely editable by members (similar to a Wiki). Testing scenarios "can be created from scratch, imported from some other organization (modulo any IP restrictions), or forked from an existing WSTF scenario." In this manner, WSTF represents "an attempt to apply some of the tenets of open source to the process of interoperability testing."
Three WSTF testing scenarios were published initially. SC002 'Notify-Echo' provides a baseline set of messages/operations that can be used in subsequent, more complex, scenarios. "Notify is a one-way operation, while Echo is a two-way (request/response) operation. While only support for SOAP (v1.1 and v1.2) will be required, implementations may also choose to expose WSDL describing their endpoints." SC003 'WS-Addressing' uses the basic Notify/Echo message exchanges of SC002, adding the use of WS-Addressing (1.0) to the exchanges. SC009 'Purchase Order Service' "illustrates how Web services can be used to solve the problems of asynchronous interaction, request/response correlation, status notification, with the ability to provide ongoing status on the transaction." It involves SOAP, WS-Addressing, WSDL 1.1, and WS-Policy 1.5.
As explained in the announcement, WSTF makes it "easy to introduce new interoperability scenarios and approve work through simple majority governance... Interoperability is a critical component to the success of Web services standards. WSTF members plan to work closely with their customers to develop scenarios and fill a void by providing access to an open forum to test or validate applications and services. Members can test their product implementations against other members' in an open environment. Providing access to continuously available services minimizes the need for each vendor or customer to set up multi-vendor test environments in house. This testing is intended to help organizations deliver higher quality products and open standards specifications, simplifying integration and improving interoperability for customers in heterogeneous environments."
The WSTF announcement declares that the Web Services Test Forum attempts to "fill a void by providing access to an open forum to test or validate applications and services... Using customer-based scenarios, interoperability is validated in a multi-vendor testing environment. Customers and vendors alike, independent of their geographic location, can dynamically test their applications against available implementations to ensure interoperability is preserved. As an open community, WSTF has made it easy to
introduce new interoperability scenarios and approve work through simple majority governance..." The motivation for formation of WSTF is clarified further in the following (excerpted) commentary from Gilbert Pilz and Doug Davis.
From Gilbert Pilz:
"Suppose you are a developer or an architect and you are trying to figure out how to integrate some systems, build out a system, etc.. Further suppose that you are planning on using web services in some parts of this project. Finally suppose that you know that these systems will involve two or more different web services implementations (for example, Apache Axis and Oracle WebLogic). Given this situation, the natural question to ask is "Will these different implementations interoperate across the technologies I intend use?"
The state of web services interoperability today is that, if you stick to the technologies covered by WS-I's Basic Profile (synchronous SOAP messaging, WSDL 1.1, etc.) and your target implementations comply with BP, it is unlikely that you will encounter any significant interoperability issues. That's the good news. Once you leave the safe harbor of these tried-and-true technologies, however, the picture is a bit murkier. What if you need to use some form of asynchronous communication, or need your messages to be integrity protected, etc.? The truth is that these technologies simply haven't received the necessary amount of real-world testing to flush out all their interoperability issues. This is particularly true when you "compose" two or more of these technologies in unique and interesting ways...
To be honest, the promise of web services interoperability has been somewhat oversold by the vendors (my present and former employers included). I say "somewhat" because there is a degree to which the promise of web services interoperability has always been more abstract and architectural than a statement of guaranteed, out of the box interoperability. To clarify, consider the example of JMS. JMS does not define a wire-level message format. This means that, although there are adapters between the various MOM systems with JMS APIs, you can't have standards-based interoperability between one JMS implementation and another; there is a crucial piece of the architecture that is simply not defined by any standard or specification...
The problem lies in the level of detailed testing required to make sure that different implementations of the same specifications really do interoperate. Keep in mind that standards are a necessary but insufficient pre-requisite for interoperability. Regardless of how hard or long the authors worked on a standard (and our industry is notoriously impatient when it comes to standards), it is inevitable that there will be areas that are unclear, contradictory, or under-specified. The majority of these areas probably won't impact your project, but some of them might. The only way to tell for sure is to test the implementations and technologies with constraints and configurations that match your project...
"If you have experience in the area of interoperability you know that these problems are never going to go away. As long as we continue to standardize new technologies there will be the inevitable gaps, misalignments, and misunderstandings that lie at the root of all interoperability problems. The WSTF is founded on the simple yet radical notion that the only effective way to discover and address these problems is to put the end-users in the drivers seat and have them direct the testing efforts. Only time will tell if this approach is workable..." [Pilz]
From Doug Davis:
"While Web service specifications sometimes do work as expected, if you use them on a regular basis, you have probably experienced far too many times when they don't... The goal of the WSTF is to bring together participants from across the Web services community and provide a forum for them to discuss and tackle the shared challenges they are facing related to the interoperability of specifications...
Three things that could contribute to the ease of use and interoperability of Web services, but that are missing from the current process, are: customer involvement, a long-term testing strategy, and a formal community to share ideas and concerns...
Customers fending for themselves: While specification writers may claim to be keeping customers and their scenarios in mind, something is clearly missing when a fair number of customers can't get even the most basic stuff to work without some amount of frustration. There are several aspects of meeting the customers needs to consider. The most basic is simply meeting customers' functional needs (for example, ensuring SOAP gets message X from point A to point B reliably). Beyond that, there are smaller and less tangible aspects to meeting customer needs. Examples of these other needs include things like customers having a place to ask more than one Web service vendor how things are supposed to work — mainly so that the questioner can get an unbiased answer. Or, being able to quickly see what several implementations do with common usage patterns, without having to call each vendor on the phone and ask them to devote days (or weeks) to set up a testing environment...
Test strategies missing the big picture: Another possible barrier to success is how the interoperability and verification process itself has been set up. Is the purpose of the testing to validate that what's in the specification can be sent and consumed by SOAP endpoints, or is it to validate that the specification composes with the rest of the Web service stack and actually meets the needs of customers? In the past there has been too much focus on the former and this is why there are usually very few interoperability issues found during the testing events. When the focus is simply on testing the XML that flows on the wire and not on the higher level business problem trying to be solved, it's too easy to miss the bigger picture. As a result, when customers go beyond the tightly scripted and tested scenarios, things tend to break down rather quickly — even if the usage they have in mind is a trivial one. Additionally, there are inherent problems with the point in time nature of the current interoperability testing...
Unbiased discussion: Something that the Web services community has been lacking is a place where all members in all various roles can come together to discuss testing and other concerns related to Web services. For example, there has been no place that a Web service vendor, an ISV, or a customer, could go to ask a broad community of people a question. Most other places have a natural bias. An inquiry posed to a particular vendor would naturally tend to result in an answer skewed towards that vendor's own product's capabilities..." [Davis]
Creators of the Web Services Test Forum freely acknowledge that the initiative is experimental, endeavoring to provide some of the "missing pieces" and "fill some of the gaps in the community" with respect to interoperability testing. WSTF thus complements work already being done in other fora and continues to work in synergy with standards bodies and specification authors. At the same time, WSTF is said to represent "a radical and innovative departure from business as usual in the interoperability space." A summary of intended benefits is provided in the WSTF FAQ document as follows:
There are several key benefits of the WSTF; they include:
- A focus on customer based scenarios and requirements rather than just low-level specification interoperability verification
- Long-lived endpoints that allow for continuous regression testing without the need for a formal 'interop event'
- Testing is accomplished using a "many-to-many" interoperability model
- A lightweight forum in which new scenarios can be started/tested without the bureaucracy found elsewhere; anyone can propose and work on a new scenario
- A forum in which specification at all stages of their lifecycle can be tested
- A forum where people can ask the WS community at large about some issue or concern without the need to worry of it being "out of scope"
- All Web Service related work can be examined — for example, this group is not limited to just SOAP-based Web Service testing
Other stated benefits include:
- interoperability testing independent of geographic location (e.g., vendor plugfest events often require travel to F2F meeting venues)
- provision of a shared testbed where the cost is spread out among the members by having each implementation host their own endpoints; removing the need for each to duplicate this effort themselves in-house
- independent testing opportunities: any test that requires two or more people from separate organizations to participate at the same time is inherently more expensive and complex than a test which can be conducted by one person alone; WSTF seeks to promote and support testing that does not require the active participation of all the parties involved in the test
- lower cost: end users get a way to test for interoperability problems in scenarios that they define without having to install any systems or write any code, and get the benefit of design advice on the use of various standards, where each scenario architecture can be regarded as a recommendation on how to best solve the business problem
- feedback to providers: vendors and other web services implementation providers get direct input from the end-users on the scenarios that interest them, allowing the providers to focus their testing efforts for maximum efficiency
The central vehicle for WSTF interoperability testing is the scenario.
According to the FAQ document, the two published scenarios Notify-Echo (SC002) and WS-Addressing (SC002) form the basis for most of the other scenarios. These two define just very basic operations that once implemented will allow users to focus the more interesting aspects of Web Services — composing in the other Web Service specifications. A third published scenario Purchase Order Service (009) "illustrates how Web services can be used to solve the problems of asynchronous interaction, request/response correlation, status notification, with the ability to provide ongoing status on the transaction. Use of long running web services provides an enterprise the ability to expose legacy systems to the internet / intranet and invoke interoperable services. The resultant value to the enterprise is increased flexibility and responsiveness to addressing business challenges."
Published scenario SC002 'Notify-Echo' exhibits a structure common to published scenarios SC003 and SC009: (1) Dependencies; (2) Scenario Description; (3) Scenario Details; (4) [Tests]; (5) WSDL; (6) [Schema]; (7) [Sample Messages]; (8) Findings; (9) Change History. Elements 4, 6, and 7 are variable.
SC002 "Notify-Echo" (produced 2008/11/03) "is meant to serve several purposes: [i] As the first scenario done by the WSTF, it will test the processes and infrastructure of the WSTF initiative; [ii] Test some very basic Web Service operations that can be used to build a set of regression tests for the WSTF set of scenarios; [iii] Provide a baseline set of messages/operations that can then be used in subsequent, more complex, scenarios. This scenario makes use of two operations: Notify and Echo. Notify will be a one-way operation, while Echo will be a two-way (request/response) operation. While only support for SOAP (v1.1 and v1.2) will be required, implementations may also choose to expose WSDL describing their endpoints. Sample WSDL is included... Other operations are defined as part of this scenario but are not necessary for the tests themselves. They are defined within this document to provide a full set of common operations that might be needed for future WSTF scenarios..."
What are scenarios? From the Wiki article and blog by Gilbert Pilz:
Test scenarios are made up of three parts:
- A high-level use case that describes the problem to be solved and the constraints on that solution
- An architecture that describes the services technologies and standards that will be used to address the problem and how they will be used
- A set of test cases along with the artifacts (WSDL, XML Schema, etc.) necessary to implement those test cases [Wiki]
The WSTF process is based around the concept of a scenario. A scenario is a number of things rolled into one. Scenarios are based on the business problems and use cases of interest to the members of the WSTF. The fundamental dilemma in web services interoperability testing is "Given a nearly infinite combination of technologies, options, constraints, etc., which ones should I test for interoperability?" The WSTF answer to this question is "The ones the end-users care about." Along with a description of the business problem, a scenario includes an architecture that describes the technologies and standards that will be used to address the problem and how they will be used. Finally the scenario includes a set of test cases along with the artifacts (WSDL, XML Schema, etc.) necessary to implement these test cases...
Here's a rough example: Suppose there is an end user organization that needs to share documents with its customers. The customer's systems will generally reside behind a NAT/firewall combination and will not be externally addressable. The documents must be integrity protected and authenticated. The system should be robust enough to handle temporary network outages and hiccups. The architecture portion of the scenario describes how to address this problem with a combination of WS-Addressing, WS-MakeConnection, WS-SecureConversation, and WS-ReliableMessaging (if you are curious about what this architecture looks like, join the WSTF and we can work it out). Finally there are a number of test cases accompanied by the XML Schema and WSDL definitions necessary to implement them...
Once a scenario has been defined, members of the WSTF may implement it using their products or open source projects. These implementations are deployed onto publicly available systems (maintained by the individual implementers) and testing is conducted in with other implementations in a crosswise fashion. Problems and issues are discussed on the WSTF mailing lists; the scenario may need to be clarified or re-factored during this process. If enough implementations of the scenario are produced and the implementers choose to do so, the scenario and its implementations can be made visible outside the WSTF by publishing it. Whether published or not, the endpoints that provide the scenario implementations are expected to be maintained indefinitely. This allows other members of the WSTF to perform regression testing, test new implementations, verify behavior, etc. without requiring the active participation of the implementer...
What about tangible results? Some of the things the WSTF produces:
- A yes/no indication of interoperability for a given scenario. Given "Does A interoperate with B using C?" — you get your answer.
- The endpoints that implement a scenario: The fact that these endpoints are long-lived means that you can re-check the interoperability results at any time. You also can create your own implementation of a scenario and check what you have done against any of the existing endpoints.
- A set of "findings" for the scenario: These are notes that discuss what was discovered during testing....
- A set of guidelines or best practices on how to address the business case using web services technologies.
Doug Davis provided further explanation about customer requirements in scenarios, functional testing, and forward looking scenarios:
"The goal of [WSTF] interoperability testing should not be to verify that scenarios can interoperate; it should be to verify that specifications actually meet customers' requirements in an interoperable fashion. Achieving this goal requires something that was missing from previous interoperability efforts: customers. The success of the WSTF will depend on whether true customer requirements are brought to the table in an objective way. Ideally, this would be done by having the customers themselves join the group and offer suggestions and guidance on the scenarios. As the WSTF is being announced, it already has several customers who are part of the group. The role of the customers is to offer ideas for scenarios and help keep the group focused on actual business requirements...
If viewed as a set of building blocks, functional testing is the foundation on which all other testing will happen. This means that while the WSTF is trying to test at a higher level than what has been done in the past, it must also continue to ensure that the foundation of the Web service specifications themselves are sound. In practical terms this means that most of the scenarios that have been tested in the past will gradually be brought into the WSTF. This will provide a transition from the previous testing efforts to WSTF. Vendors who participated in the previous interoperability events should be able to use the same code within the WSTF for these functional tests. This also will provide for regression testing...
[In addition to customer centric and specification centric testing, WSTF also provides a third view: forward looking scenarios. Forward looking scenarios try to use the specifications in ways that have not been done in the past. This serves as an attempt to stay one step ahead of where actual deployments are so that, vendors see problems before their customers do. Forward looking testing can be as simple as doing new compositional testing... Another type of forward looking scenario is one that exposes a hole in the current Web service architecture. These scenarios would most likely start by describing a business requirement. Then while going through the process of trying to figure out which Web service specifications are best suited to solve that problem, the community might realize that something new is needed... [Davis]"
A key goal of WSTF is to provide a forum in which WS-* specifications at all stages of their lifecycle can be tested.
Doug Davis, on 'Long Lived Endpoints':
"Previous interoperability testing efforts were point in time statements. If successful, the test proved that a certain set of implementations could get together and interoperate once. There was no guarantee that things would continue to work properly in the future, nor was there a way for new implementations to verify that they would interoperate with anyone else. Each implementation could (and probably does) set up other implementations in house to do this interoperability testing, but that is a very expensive and time consuming effort — an effort each vendor would need to duplicate. The WSTF tries to reduce this overhead by providing a shared, and long lived, set of testing endpoints that are available to the entire community.
The idea of long lived endpoints differentiates WSTF from other Web service testing efforts. Within the WSTF, each scenario will have a list of endpoints that support that scenario. By hosting and listing their endpoints, each implementation is agreeing not just that it supports the scenarios as described, but that it will try to keep those endpoints available for others to use for an indefinite amount of time. This means that the testing done within the WSTF is not a point in time statement; it's a continuous testing effort.
As vendors update products or develop new implementations, they can use the endpoints to verify that their new code works and ensure that bugs have not been introduced into existing implementations. While this could have been done in the past by each vendor hosting other vendors' implementations, the WSTF allows the true experts to host their own code — freeing developers to focus on their own implementations.
WSTF provides URLs that point to the endpoints themselves, not the actual implementations of the specifications. This means that vendors can test code that has not yet been shipped. They no longer need to wait until other vendors' products are publicly available before verifying that the latest versions have not introduced any regressions..."
"Unlike other interoperability efforts, the WSTF is not prevented from examining any Web Service specification at any stage of its development. Other groups are designed in a way that prevents them from leaving the confines of their tightly scoped charters. The WSTF does not have these kinds of limitations. It can test a specification during (or even before) it is in a standards body..."
Dan Toth, Manager, Enterprise Architecture at Ford: "[WSTF] will play a key role in accelerating interoperability for Web services standards by not waiting for Web services standards to be approved before initiating testing based on customer scenarios... It is very important that interoperability issues be identified as early as possible in order to eliminate obstacles to adopting Web services." [...] Providing access to continuously available services minimizes the need for
each vendor or customer to set up multi-vendor test environments in house. [WSTF Press release]
WSTF is presented as a forum designed for the Web Services community, for both individuals and corporate participants, where "people can ask the WS community at large about some issue or concern without the need to worry of it being 'out of scope'. The rules for participation are intentionally simple and lax, and there are no membership dues. The consensus model is favored.
"At its most basic level, the WSTF is a place for all members of the Web services community to discuss testing and any other concerns related to Web services." [Davis]
"A key provision of the WSTF's charter is that any member can create a new scenario or participate in the discussion and development of an existing scenario. Scenarios can be created from scratch, imported from some other organization (mod any IP restrictions), or 'forked' from an existing WSTF scenario. Vendors and other web services implementation providers are expected to 'vote with their feet' and implement those scenarios that align with their customer base and technical direction." [Pilz]
FAQ Question 5: What kind of commitment is expected of members? How much time and effort will be required? The WSTF is purely a volunteer organization and each member is free to choose their level of involvement. For example, at one end of the spectrum, a member may choose to simply monitor the mailing lists. A slightly more active member may choose to offer feedback or suggestions for new scenarios. And at the other end of the spectrum, a very active member may choose be very involved in the development of a scenario. And, of course, a member's interest in Web Services will most likely dictate how they participate. For example, Web Service vendors will probably be very active on the development of the endpoints, while users/customers of the products produced by those vendors may find themselves more on the scenario development side of things. But, again, this is all voluntary and members are free to take on whatever role (or roles) they wish..."
"Anyone with an interest in Web Services and is willing to sign the Participation Agreement... Any WSTF member can create a new scenario... Anyone can request a vote by simply sending a note to the Administrator..." [FAQ]
Published scenarios are available to the general public as well as to WSTF participants: "Anyone, regardless of whether they're in the Forum or not is allowed to use the endpoints listed for a scenario. Obviously, good behavior is expected. This means care should be taken to not overload any endpoint... Anyone can run regular regression tests using these endpoints...the endpoints are here to encourage testing and promote interoperability. [But] it would defeat the purpose if these endpoints needed constant monitoring due to their abuse..." [FAQ]
"The WSTF has no dues primarily to encourage as many people and organizations to participate as possible. If there is a mom-and-pop consulting shop with a good idea for a scenario, the WSTF wants to hear it. Having dues also requires you to have some sort of board to watch over the money which brings us to our next point." [Pilz]
WSFT seeks to establish interoperability by consensus: "many interoperability issues arise either because the relevant specifications simply do not cover a particular area or there are conflicting, but equally valid interpretations of a specification. The WSTF seeks to resolve such cases using a common-sense, consensus approach... Interoperability defined as compatibility with a single vendor's (or subset of vendors') products only creates islands of non-interoperability." [Wiki]
WSTF is described as a lightweight organization designed to put the community itself in control for sharing of ideas and concerns.
"[WSTF has] "no centralized control: past experience has shown that, if a subset of members is allowed to control what is and isn't tested, they tend to try and direct testing efforts towards technologies and standards that they favor and discourage testing on technologies that, for whatever reason, they do not like. In the WSTF any member is free to propose a scenario, contribute to an existing scenario, or implement a scenario as they choose. [Wiki]
"The WSTF is actually a very simple and lightweight organization. At its core, the Web site consists of nothing more than a set of scenarios..." [Davis]
WSTF has simple IPR rules designed to eliminate parent and copyright encumbrances. Insofar as (business process) patents might come into play, they are excluded from relevance by WSTF rules, so that participation in WSTF takes place on a level playing field. From the Participation Agreement:
"... For any Contributions provided to the Forum, you grant to all other participants in the Forum a non-exclusive, non-transferable, worldwide, perpetual, irrevocable, royalty-free copyright license to copy, publish, license, modify, distribute and otherwise exploit Your Contribution for use in any Output being developed by the Forum or in any specification being tested in the Forum, including any derivative works of them; provided however, that such license shall not include any license to use Your or Your company's name, likeness, logos, trademarks, trade names, service marks or service names.
Likewise, if incorporation of Your Contributions into Output or into a specification being tested in the Forum would cause an implementation of such Output or specification (as subsequently modified by the owners of the specification or a Standards Organization if it has been submitted) to infringe a patent or patent application that You own or control, You commit to grant to all implementers of such Output or specification a Royalty-Free license on reasonable and non-discriminatory terms under such patent or patent application to make, have made, use, sell, offer for sale, and import products or services that implement such Output or specification...
"Anything brought to the group during the course of the discussions — for example, an idea for a scenario —, is brought to the group under Royalty Free terms. This protects the members of the group from being sued by other members for patent infringement." [Davis]
Descriptions of the WSTF process make frequent reference to the "open souce" software development model:
- "In a certain sense, the WSTF is an attempt to apply some of the tenets of open source to the process of interoperability testing." [Pilz]
- "WSTF is a community-based forum that follows a model similar to that of open source communities." [Davis]
- Any member is welcome to modify any document at any time — similar to the open-source model... In many ways, [the WSTF scenario] approval process is similar to how open-source communities vote on whether to release a new version of their code... [Davis]
- "Once a scenario has been defined, members of the WSTF may implement it using their products or open source projects. These implementations are deployed onto publicly available systems (maintained by the individual implementers) and testing is conducted in with other implementations in a crosswise fashion. Problems and issues are discussed on the WSTF mailing lists; the scenario may need to be clarified or re-factored during this process." [Pilz]
Care has been taken in the WSTF rules to protect the privacy of any participant's testing results: they are not permitted to be published outside the WSTF forum without permission. This feature permits each participant a range of freedom to experiement, to test immature and incomplete software applications, and to control the level publicity given to early test results.
"Although confidential information is not to be exchanged within the Forum, you are expected to respect the privacy of others regarding the testing of Web Services implementations because, for example, other participants may be working with pre-release code. Results of the testing are not intended to be publicly posted outside of the Forum. By participating in the Forum, you agree not to disclose, comment on or otherwise characterize, in any manner except as permitted herein, any information pertaining to the activities in the 'private' portion of the Forum..." [PA]
"Members should be able to discuss new ideas, concerns, and even short-comings of products without worrying that their comments might later be used against them in some negative way..." [Davis]
The WSTF Charter and other documents clarify that the Web Services Test Forum is intended to supplement other interoperability efforts and provide testing scenarios for specifications created in SSOs/SDOs.
According to the Charter: "The WS Test Forum ("Forum") is meant to provide an environment in which members of the Web Services community can develop and test interoperability scenarios ("Scenario(s)") that go beyond the specification testing done elsewhere, which is typically limited to a single Web Services specification ("Specification"). For example, this Forum might test Specifications that have not yet been through the standardization process, to ensure their composability with other Specifications. It might also test the composition of Specifications that might normally not be within scope of the activities of other organizations, for example testing a Specification developed in W3C composed with others developed in OASIS..."
The WSTF Participation Agreement indicates that "If you have any comments related to the Web Services specifications being tested, you are encouraged to submit them to the standards body developing the specifications or to their authors." FAQ document (Q18) clarifies the next step to be taken if one finds a problem with one of the WS specs: Interested parties should take the issue to the appropriate forum. This will usually be the authors working on the specification..."
Several published documents reference parallel interoperability testing work (being) done elsewhere, for example:
Apache Stonehenge Project. "Stonehenge is a set of example applications for Service Oriented Architecture that spans languages and platforms and demonstrates best practise and interoperability... the aim is to develop a set of sample applications to demonstrate seamless interoperability across multiple underlying platform technologies by using currently defined W3C and OASIS standard protocols." See Paul Fremantle's blog article and Mike Champion's memo. Yahoo! Soapbuilders Group. The message archive contains several thousand messages. Compare also WS-Builders. Vendor-sponsored Interop Workshops and Plugfest events (Microsoft, WSO2, IBM, SAP, etc), often organized as F2F events, for example:
Gilbert Pilz wrote:
What About the WS-I? This may seem like polite, political spin but I view the WSTF and the WS-I as complementary. The WS-I's job is to address interoperability problems by writing specifications (called profiles) that constrain and clarify existing standards. The WS-I functions best when the interoperability problems it is addressing are already well understood by the working group participants. For example, the Basic Profile has been successful because it addresses the common interoperability problems in SOAP 1.1 and WSDL 1.1. These problems were well understood largely due to the efforts of the SoapBuilders community/mailing list (the WSTF was deliberately patterned after SoapBuilders). It is the WSTF's job to find interoperability problems but it is not its job to create new specifications or profiles to address those problems. The most the WSTF should do is (a) notify the WS-I or relevant SDO of the issue and (b) craft a work-around that avoids the issue.
In fact, the WSTF is a necessary pre-cursor to the successful functioning of the WS-I. It is true that, in recent years, the WS-I has gotten ahead of itself with regards to practical experience in the standards that is is profiling (e.g. the Reliable Secure Profile Working Group - RSP) but, hopefully, the WSTF can come up to speed quickly enough to help the WS-I in this area. As of this writing, the WSTF has already found a number of issues that have been contributed to the WS-I's Basic Profile Working Group [Charter]..."
Chris Ferris wrote:
"... we see WSTF as complimentary to WS-I. WS-I is still providing a great service to the community in helping the community understand where the bar is, so to speak, and also in helping the vendor community find that bar in a manner that ensures consensus. It may be a slow process in some people's eyes, but it is definitely a value add. Next year, we should see the approval of at least three (3) new WS-I profiles (Basic Profile 1.2, Basic Profile 2.0, and Reliable Secure Profile 1.0) as well as implementations that conform to these new profiles shipping from your favorite vendors.
WSTF provides us with a forum that allows us to test not only the technologies covered in the new (and past) WS-I profiles, but also allows for the testing of technologies that may not be finished (from a standardization perspective) as well as to test some of the more 'creative' aspects of applying those technologies that are included in the WS-I profiles, but which may not have been incuded in testing to date..."
Doug Davis wrote:
"[WSTF] can test a specification during, or even before, it is in a standards body.
For example, WS-I can only examine specifications that have exited a standards body. Often times this means that it is too late to address any serious issues found with the specification. By contrast, the WSTF could choose to go ahead and begin testing and providing feedback to the new W3C Working Group standardizing several specifications such as WS-MetadataExchange. The WSTF could begin such an effort long before the Working Group actually begins its own testing and well before the specification is complete. This can uncover obvious bugs in the specification as well as less obvious compositional issues that might only be found through bringing several Web service specifications together at once..."
Karla Norsworthy (IBM) said:
"We think WS-I has served us very well to define some of the profiles and focus on interoperability work for some of those basic profiles. [WSTF] is kind of an evolution," for Web services interoperability... WSTF allows particpants to maintain interoperable endpoints enabling, for example, interoperability with IBM's WebSphere application server... Asked why the new efforts required formation of a whole new industry organization, Norsworthy said the WSTF offers a more lightweight approach to interoperability and enables more customer input. WS-I, she said, has been good for building consensus around a small set of key profiles for everyone to implement. IBM plans to continue its WS-I participation... 'We really do want to make sure we're not [forming a new organization] every time we turn around but we found this to be important and complementary'... Sun and Microsoft noted participation in WS-I when asked why they are not climbing aboard WSTF. Sun is taking a wait-and-see approach to WSTF, the company said in a statement. 'Sun is strongly committed to Web services interoperability and testing. Sun has been a member of WS-I and has been collaborating with Microsoft to ensure .Net and Java Web Services interoperability for some time,' Sun said. Microsoft stressed its WS-I efforts...
A public announcement was made for the Web Services Test Forum on December 8, 2008. Fourteen companies were named with initial membership: Active Endpoints, AIAG, Axway, CISCO, eviware, FORD Motor Co., Fujitsu, Hitachi, IBM, Oracle, Red Hat, Software AG, Teamlog, TIBCO Software Inc. The WSTF web site also provided reference to WSO2 as an early member.
The WS Test Forum Participation Agreement and WS Test Forum Charter documents were released with a publication date of 2007/10/15.
Gilbert Pilz indicated that BEA, Fujitsu, IBM, and Oracle were influential in the early stages of WSFT formation: "The WSTF was patterned by its initial creators (BEA, Fujitsu, IBM, and Oracle) after the SoapBuilders mailing list/community." [Wiki]
Doug Davis: "The WSTF was originally formed to perform scenario-based interoperability testing that was not being done elsewhere. In particular, this was to be compositional testing where a group of specifications were tested together to ensure that they really were as composable as advertised. However, before these more advanced combinatorial tests could be run, it had to be verified that: (1) the individual specifications themselves worked, and (2) the proposed scenario wasn't just a single developer's dream, and that it was based on actual customer needs. From this simple idea, the WSTF was born..." [Davis]
In addition to official documents from the Web Services Test Forum (announcement, Charter, participation Agreement, FAQ document), several initial publications were used in the creation of this report:
[December 13, 2008] Web Services Test Forum By Gilbert Pilz and others. From Wikipedia [User Page: Gilbertpilz]. "The Web Services Test Forum (WSTF) provides a framework in which members of the Web Service community can develop interoperability scenarios and test implementations of those scenarios against other implementations. The WSTF does not charge dues and has no central governing authority (i.e. board). The WSTF was patterned by its initial creators (BEA, Fujitsu, IBM, and Oracle) after the SoapBuilders mailing list/community. While its main focus is to test the various Web Service specifications, it also serves as a forum where the entire Web Service community can share ideas and concerns in an open fashion... The WSTF is founded on the following basic principles: (1) Low barriers to participation; (2) No centralized control; (3) Interoperability by consensus; (4) Independent testing; (5) Distributed testing costs... Unlike other interoperability organizations, the work of the WSTF does not center around individual specifications. Activities are organized around the concept of a Test Scenario..." [December 12, 2008] "New Group Promotes Web Services Interoperability." By Darryl K. Taft. From eWEEK. "A new industry consortium has formed to address the issue of Web services testing and interoperability, with big names such as IBM, Oracle, Cisco and Red Hat leading the way among vendors and end-user organizations such as Ford Motor Co. taking part as well... Karla Norsworthy, vice president of software standards at IBM: "We've been active in implementing scenarios and maintaining IBM product endpoints as well as socializing the benefits within the community and encouraging participation... This [Forum] extends our ability to provide customers with interoperability for Web services-based scenarios. This forum provides us with a way to test for interoperability across product and specification life cycles — so that it is easy for us to verify that a new release or fix pack still interoperates with other vendor implementations. This saves us from needing to bring up copies of products from other vendors in our labs. It also provides a lightweight way to test new scenarios of interest to customers — and a forum to bring the community together to have those conversations. This will give our customers additional assurance on interoperability promises — as well as documented best practices to help them deploy service-oriented solutions in a timely way..." [December 10, 2008] "WSTF Launch — Why You Should Care." By Chris Ferris (IBM). Blog: "Industry Standards, XML and Web services, Distributed Computing, and Interoperability." — "This past Monday, IBM, with the help of others launched the Web Services Test Forum (WSTF). My colleague from Oracle, Gil Pilz, has written a really nice piece on why WSTF is needed and explains the value that it will hopefully bring to the community... I just realized that Doug Davis also posted an article on WSTF on IBM's developerWorks site. Please do go read that as well! [...] I've been bending a few ears here at the Gartner ADI conference and have heard WSTF mentioned in a couple of contexts, including hearing from at least one prospective member indicating an interest in participating. Gil's explanation should be required reading, but allow me to just weigh in on a couple of key points..." [December 10, 2008] "Companies Link Up to Test Their Services Against Web Standards." By David Worthington. From SD Times "On Monday [2008-12-08], the Web Services Test Forum (WSTF) went public with its work. Members tested customer-based scenarios where services must function in multi-vendor environments... Thus far, three scenarios are published on the WSTF website, freely available for review. Five more are under development, said Karla Norsworthy, vice president of software standards at IBM. The tests have found that some developers' standards implementations were not properly handling the nuances of some of the scenarios. Interoperability scenarios are proposed by clients and members, and are tested after five members agree to implement them... Anyone anywhere can participate; there is no special labor or traveling. WSTF's work will improve the responsiveness of software makers, enabling them to identify and fix issues before clients find failures..." [December 09, 2008] "The Web Services Test Forum." By Gilbert Pilz (Oracle). From Gilbert Pilz's Blog: "Articles on Web Services Standards and Specifications With a Focus on Interoperability Issues." — "On Monday (12/8/2008) the creation of the Web Services Test Forum (WSTF) was announced. At first glance the WSTF may seem, as some analysts seem to think, 'yet another web services forum'. On closer examination, though, I think you'll agree that the WSTF represents a radical and innovative departure from business as usual in the interoperability space... If interoperability is a potential issue, the last thing you want to do is leave it until the end of your project. Like any problem, interop issues can be addressed but they usually take a lot more time than 'ordinary' bugs because it is not always clear which implementation is at fault (sometimes it's both, sometimes it's neither). You need to know early on if there are any interoperability problems in the products and technologies you intend to use so you can either get them fixed or re-factor your architecture to work around them... If you are familiar with web services interoperability you are doubtless familiar with the idea of a 'plug fest'. This is where one web services vendor invites a bunch of other web services vendors to a face-to-face meeting where they cross test their implementations over a set of scenarios. On the face of it, this seems a lot like what the WSTF does, but there are a number of key differences [...] The WSTF avoids these problems by creating a forum where any member can propose and create scenarios and any issues can be discussed and diagnosed in a neutral manner. Furthermore, the endpoints for a given scenario (which are listed on the WSTF website) can remain up for as long as the implementer sees fit..." [December 09, 2008] "Web Services Test Forum Announced." By Mark Little (Red Hat/JBoss). From InfoQueue. "Interoperability has always been one of the key factors pushed by vendors for the need for Web Services standards. There's even an organisation set up to address it. Over the past few years the various Web Services standards bodies such as OASIS and W3C have encouraged (mandated) interoperability demonstrations between heterogeneous vendor implementations before something can even be declared a standard. But one of the problems has always been that implementations change once these interop fests are completed and there is often limited (and ad hoc) approaches to continue to demonstrate interoperability. In recent weeks though we've seen a couple of new initiatives designed to try to bridge the gap... First there was the Apache Stonehenge project... And now there is the Web Services Test Forum (WSTF)... On the basis that Web Services standards and implementations really live or die on their interoperability, the WSTF announcement is a good thing for customers, but only if all vendors agree to abide by it, or customers put it on their check-list of must-haves when selecting a specific vendor..." [December 09, 2008] "Web Services Test Forum." By Mark Little (Red Hat/JBoss). Blog. "We've been working with IBM and others on the formation of the WSTF. Thanks to Andrew Dinn for the Web Services Transactions scenario work and Alessio Soldano for our WS-Addressing participation. Interoperability is something that I've been pushing for many years and we've always participated in as many of the official standards-based events as we possibly can. We're also members of WS-I and Apache Stonehenge, so you can be assured that Web Service interoperability will remain high on our agenda..." [December 09, 2008] "Web Services Interoperability May Get Chaotic. New Entrant to the Field May Spark a Battle Among Major Players." By Richard Adhikari. From InternetNews.com. "Fifteen companies, including some leading high-tech vendors, have launched the Web Services Test Forum (WSTF) to speed up Web services interoperability testing among their products.. Paul Cotton, Microsoft's group manager, Web services standards and partners, told InternetNews.com by email that the company has not heard of customer interest in the creation of new, alternative interoperability organizations such as that recommended by the WSTF proposal. 'Microsoft is deeply committed to Web services interoperability, as evidenced by our long-standing involvement in the Web Services Interoperability (WS-I) organization,' Cotton said. 'WS-I is a cross-industry initiative designed to accelerate the development and deployment of interoperable Web services across platforms, applications and programming languages.' Microsoft believes that WS-I provides a proven and open organization and process that best suits its customers' needs, Cotton added. WS-I, the Web Services Interoperability Organization, is an open industry organization chartered to establish best practices for Web services interoperability for selected groups of Web services standards..." [December 08, 2008] "Web Services Test Forum (WSTF): Bridging the Gap Between Promises and Reality." By Doug Davis (IBM). From IBM developerWorks. Also in PDF format. "SOAP-based Web services have come a long way since their creation many years ago. Recently, the number of new specifications being developed has slowed quite a bit, and this is allowing the community time to settle down and take a closer look at the base infrastructure that has been developed. Have the promises of Web service interoperability been met? Do the Web service specifications really work out of the box as they should? This article addresses these questions and introduces the Web Services Test Forum (WSTF). WSTF is a new community-based forum aimed at addressing interoperability issues with Web services... This article uses the term Web services as a synonym for SOAP, but the WSTF is actually not limited to just SOAP-based Web service testing. There is nothing that would prevent the WSTF to extend its testing into other Web service testing. For example, the WSTF would allow, and even encourage, the testing of domain-specific uses of SOAP/Web service. Testing of REST/Web services would also be allowed and will likely take place in the not too distant future. The WSTF isn't just about SOAP-based interoperability testing — it's about Web service interoperability testing, and the community itself will decide what that means over time... The WSTF was formed to focus on ensuring that issues with the interoperability of the WS-* set of specifications are found and addressed before customers see them. The structure of the WSTF also provides for a much broader benefit to the Web services community at large. By providing a common testing ground and location for sharing ideas, the WSTF can fill some of the gaps in the community and open development aspects of Web services. The WSTF can provide a forum where someone can speak to all types and sizes of players within the Web services community. The organizers of the WSTF have tried hard to ensure that the WSTF is focused on technical discussions with limited room for politics — hopefully, this will continue..." [December 08, 2008] "Here Comes WSTF." By William Vambenepe (Oracle). Blog. "A new Web services-related industry body has been announced today: WSTF (Web Services Test Forum). More details about it from Infoworld. My employer (Oracle) seems to be one of the drivers (along with IBM) but I am not personally involved. A lot of hand-wringing, of course, about its relationship with WS-I. Which is understandable if you consider what WS-I was originally supposed to deliver (profiles, sample applications and testing tools). But not if you consider what it has actually delivered that is relevant (a couple of profiles, some time ago). WSTF could also be compared with the SOAPBuilders Yahoo group, but since that group has seen only two emails messages so far in 2008 (last one dated April 2nd), it seems safe to consider it dead. It would be interesting to know why that is though (it used to be pretty active in the early days) and what lesson WSTF may learn from it. Another effort you may want to compare this to is the Microsoft Web Services Protocol Workshops Process. It's too early to tell, but they may turn out to be more closely related than meets the eye... I noticed this innocuous-sounding sentence in the press release: 'As an open community, WSTF has made it easy to introduce new interoperability scenarios and approve work through simple majority governance'. You may wonder why this is important enough to figure in the press release. I interpret it as a dog whistle call (heard only by those to whom it is intended) for the WS-I board... The current test scenarios seem to focus on fixing the interoperability mess that is WS-Addressing. I assume more will soon be added to test the different WS-* specifications out there. It will be interesting to see what direction WSTF takes after that. Will the payloads of the test messages be obvious dummy payloads (so that the focus is on testing the implementation of the WS-* protocols)? Or will they start to include real payloads (e.g. real purchase orders from real enterprise applications)? [...] This could become an interesting tussle between vendors as well as between vendors and buyers. Alternatively, of course, WSTF could turn into a test of how much difference there is between a 'standard' and a publicly specified and interop-tested interaction scenario..." [December 08, 2008] "IBM, Oracle, Red Hat Form Test Bed for Web Service Interoperability." By Charles Babcock. From InformationWeek. "IBM, Oracle, Ford Motor, Red Hat, Tibco Software, and other companies have banded together to form the Web Services Test Forum. It will seek to establish interoperability among products built to existing standards. It also will seek to establish interoperability between products built to emerging standards, before those standards have been finalized. Dan Toth, manager of enterprise architecture at Ford: '[WSTF] will play a key role in accelerating interoperability for Web services standards by not waiting for Web services standards to be approved before initiating testing based on customer scenarios'... Toth said Ford has already experienced some difficulties in implementing Web services with business partners and suppliers. As upgrades occur to operating systems and other parts of the infrastructure at Ford, the changes break an existing Web service that ties Ford to a business partner. While such instances were rare, Toth said, 'our widespread use of Web services is only just beginning. As we get more Web services with partners, we will need more interoperability' that can last through the lifecycle of an application or piece of enterprise middleware, including its bug fixes and upgrades'... Asked if the formation of the forum was a challenge to slow-moving standards bodies, Oracle's Harris said standard setting was 'a bit of a push/pull process with standard-setting organizations.' Many Web standards have only recently been minted and are 'still immature.' The forum will be trying to establish best practices in implementing standards, which is sometimes a gray area in standards formation... [IBM's] Norsworthy said the forum hopes nontechnology vendor members, such as Ford, continue to join the group..." [December 08, 2008] "Is the WSTF One Web Services Forum Too Many?" By Paul Krill. From InfoWorld. "Vendors including IBM and Oracle are launching an industry organization Monday [2008-12-08] for Web services interoperability, despite the existence of another Web services interoperability group with many of the same members... [WSTF] will use customer-based scenarios to validate interoperability in a multi-vendor testing environment, according to a statement from WSTF. Customers and vendors can dynamically test applications against implementations to ensure interoperability. Testing is intended to help delivery of higher quality products and open standards specifications to simplify integration and improve interoperability... Also on the list of the 15 current members are such companies as Active Endpoint, Cisco, Ford Motor, Fujitsu, Hitachi, Red Hat, and Tibco. But some of these same companies, including IBM, Oracle, Fujitsu, Hitachi, and Tibco, are also members of the Web Services Interoperability Organization (WS-I), formed in February 2002 also to promote Web services interoperability..." [December 08, 2008] "Web Service Test Forum Launched by Vendors." By Rich Seeley. From SearchSOA.com. "... The [WSTF] forum, which is also open to corporate users and has Ford Motor Co. as an initial member, will validate interoperability in a multi-vendor testing environment... WSTF seeks to bridge the gap between Web services that technically adhere to standards but may not interoperate in real-world applications. Just because a Web service is based on standards such as SOAP and WSDL, as well as the newer messaging and security standards, that doesn't automatically mean they will play well with others... [IBM's] Norsworthy said WSTF will be more agile and lightweight than the official WS* standards bodies, including OASIS and W3C, so it can be more responsive to customers' needs..." [December 08, 2008] "Web Service Testing Forum Launched. Goal is to Ensure Interoperability." By DDJ Staff. From Dr. Dobbs Journal. "The Web Services Test Forum (WSTF) has been launched, providing an open community to improve the quality of the Web services standards. The goal is for customers and vendors — independent of their geographic location — to be able to dynamically test their applications against available implementations to ensure interoperability... The WSTF plans to also work with standards bodies to help speed the standardization process for emerging Web services standards. The WSTF initiative is open to all software vendors, service providers and customers interested in furthering Web services and their use throughout the industry. Members are able to recommend and initiate work on new scenarios, supporting both emerging specifications and approved standards..."
|
 | Receive daily news updates from Managing Editor, Robin Cover.
 |
|
