The US National Institute of Standards and Technology (NIST) has announced the published results of vendor test of facial recognition technology. The NIST HumanID Evaluation Framework (HEF), constitutes "an effort to design, implement, and deploy standards for the robust and complete documentation of the biometric system evaluation process. It incorporates an attempt to leverage contemporary technologies, specifically XML, for the formal description of such tests. The HEF was used to facilitate the administration of the 2002 Face Recognition Vendor Test (FRVT 2002). HEF defines a suite of XML-based markup formalisms for the inputs to, and outputs of, recognition systems. Unlike the XML Common Biometric Format (XCBF), the focus of the HEF is the evaluation of biometric systems, and not the biometric information itself. The HEF is designed to facilitate off-line, black-box empirical testing. A recognition engine takes two sets of biometric signatures, the enrolled and the unknown test samples, and produces some form of identification data. Currently, the HEF assumes that this output data is a collection of scores, with each score indicating the similarity between a pair of signatures. The primary objective of FRVT 2002 was to provide performance measures for assessing the capability of automatic face recognition systems to meet real-world scenarios -- verification of identity, identification of an unknown individual, and detection of an individual on a watch list." HEF XML Schemas defining formats for describing identification, verification, and watchlist scenarios have been developed and are being made public.

About the HumanID Evaluation Framework

Excerpts from the January 2003 Report, INCITS/M1-03-0032:

"The HumanID Evaluation Framework, or HEF, is a mechanism for the quantitative testing of biometric recognition systems. It is an extension of the FERET protocol, which defined a framework for the evaluation of face recognition technologies. The HEF, however, is general enough to apply to virtually any recognition task, and can be applied to arbitrary, heterogeneous mixtures of biometric systems."

"The goal of the HEF effort is to design, implement, and deploy standards for the robust and complete documentation of the biometric system evaluation process. A mandate of the scientific method is experimental repeatability -- achievement of this goal in the evaluation of biometric systems is often thwarted by inadequate documentation of experimental procedures."

"The initial version of HEF was coupled with the 2002 Face Recognition Vendor Test, or FRVT 2002. Although FRVT 2002 has played a key role in shaping the current form of the HEF, it should be noted that they are separate, but related efforts. FRVT 2002 provided a unique opportunity for the development and implementation of an initial version of HEF. NIST has already applied the HEF for an internal evaluation of a fingerprint matcher."

"The HEF is designed to facilitate off-line, black-box empirical testing. A recognition engine takes two sets of biometric signatures, the enrolled and the 'unknown' test samples, and produces some form of identification data. Currently, the HEF assumes that this output data is a collection of scores, with each score indicating the similarity between a pair of signatures. The main goal of the initial version of the HEF is to have a well-defined means of marking up sets of biometric data... For added flexibility, HEF provides not a single schema, but a family of XML schemas that may be used to validate different kinds of signature sets. Using a set of schemas allows applications to validate signature sets using domain-specific criteria, while still maintaining a high-level, and global, consistency."

"The original description of the HEF included an XML format that may be used to describe the raw signature-to-signature similarity values. However, in the FRVT 2002 High Computation test alone, participants were required to provide over 1.36 billion similarity values. Even when the participants saved the results in binary form, with 4-bytes per floating-point result, and minimal metadata, over 60 GB of data per vendor was produced. Requiring that the participants output their similarity scores in XML was not maintained as a viable option."

"Because of the wide variety of biometric systems, and the varying nature of the constraints that a recognition system may want to apply on a set of signatures, the HEF includes a family of related schemas that can be used 'as is' for face recognition systems, or easily extended to accommodate new ones. HEF uses derived typesas its main vehicle for accomplishing this kind of flexibility. The structure of an HEF signature set document follows directly from the above grouping terminology. Each term corresponds to a tag, and the containment relationship is represented by the nesting of elements. For example, since a signature set contains multiple signatures, the <signature-set> may have one or more <signature> elements as child elements -- a <signature> element may have one or more <sigmember> elements as children -- and so on..."

In adminstrating FRVT 2002, the HEF played a vital role in unambigously describing sets of biometric signatures. During the development of the the FRVT 2002 scoring software, formats for descrbing identification, verification, and watchlist scenarios, were also developed. Schemas for these formats will be made availabe for the next public release of HEF. Further augmenting the scoring suite with various XSLTs allowed the ROC, CMC and watch-list data to be transformed into scripts for use with GNUplot, R, and Splus..."

About FRVT 2002

"The Face Recognition Vendor Test (FRVT) 2002 is an independently administered technology evaluation of mature face recognition systems. FRVT 2002 provides performance measures for assessing the capability of face recognition systems to meet requirements for large-scale, real-world applications. Ten commercial firms participated in FRVT 2002. FRVT 2002 computed performance statistics on an extremely large data set -- 121,589 operational facial images of 37,437 individuals. FRVT 2002 1) characterized identification and watch list performance as a function of database size, 2) estimated the variability in performance for different groups of people, 3) characterized performance as a function of elapsed time between enrolled and new images of a person, and 4) investigated the effect of demographics on performance. FRVT 2002 shows that recognition from indoor images has made substantial progress since FRVT 2000. A new XML-based evaluation protocol was developed for FRVT 2002. This protocol is flexible and supports evaluations of biometrics in general..."

"As face recognition technology has matured and is being considered for more applications, the demand for evaluations is increasing. At the same time, the complexity and sophistication of the evaluations is increasing. This results in rising evaluation costs and increases the chance that mistakes will occur. To overcome these challenges, the FRVT 2002 introduced an evaluation protocol that is XML-based and contains a package of standard scoring routines. The XML specifications include formats for test data sets, raw output formats from systems, and performance scores. The scoring routines are designed to work with XML file formats. The FRVT 2002 evaluation protocol, XML specification, and scoring package are general and can be used to evaluate other types of biometrics. The FRVT 2002 evaluation protocol is a solid basis for establishing a standard evaluation protocol." [from the Evaluation Report]