Is Conformance Testing Really Needed?

Conformance testing can be abused:

• consider an example from the NAPLPS world in the 1980's:
  • vendor A invests lots and implements complete algorithms using conformance test suite for verification
  • vendor B invests less and tunes buggy implementation such that there are no problems reported by the components of the conformance test suite, though the software does not work properly in the real world
• customer hears from both vendors that each software "conforms" so uses other criteria for purchasing decision

+//ISBN 1-894049::CSL::OASIS::Presentations//DOCUMENT XML Conformance 1998-03-27 13:40//EN
XML'98 Conference 1998-03-27	Copyright © OASIS (8)