label(s) Label Class c) b) a) 7.3 Integration Strategies 241 Combination strategy A combination strategy (also called combination scheme) is a technique used to combine the output of the individual classifiers. The most popular combination strategies at the abstract level are based on majority vote rules, which simply assign an input pattern to the most voted class (see Section 7.2). When two classifiers are combined, either a logical AND or a logical OR operator is typically used. When more than two classifiers are integrated, the AND/OR rules can be combined. For example, a biometric system may work on “fingerprint OR (face AND hand geometry)”; that is, it requires a user to present either a fingerprint or both face and hand geometry for recognition. Class set reduction, logistic regression, and Borda counts are the most commonly used approaches in combining classifiers based on the rank labels (Ho, Hull, and Srihari, 1994). In class set reduction, a subset of the classes is selected with the aim that the subset be as small as possible and still contain the true class. Multiple subsets from multiple modalities are typically combined using either a union or an intersection of the subsets. The logistic regression and Borda count methods are collectively called the class set reordering methods. The objective here is to derive a consensus ranking of the given classes such that the true class is ranked at the top. Rank labels are very useful for integration in an indexing/retrieval system. A biometric retrieval system typically outputs an ordered list of candidates (most likely matches). The top element of this ordered list is the most likely to be a correct match and the bottom of the list is the least likely match. The most popular combination schemes for combining confidence values from multiple modalities are sum, mean, median, product, minimum, and maximum rules. Kittler et al. (1998) have developed a theoretical framework in an attempt to understand the underlying mathematical basis of these popular schemes. Their experiments demonstrated that the sum or mean scheme typically performs very well in practice. A problem in using the sum rule is that the confidences (or scores) from different modalities should be normalized. This normalization typically involves mapping the confidence measures from different modalities into a common domain. For example, a biometric system may output a distance score (the lower the score, the more similar the patterns) whereas another may output a similarity score (the higher the score, the more similar the patterns) and thus the scores cannot be directly combined using the sum rule. In its simplest form, this normalization may only include inverting the sign of distance scores such that a higher score corresponds to a higher similarity. In a more complex form, the normalization may be non-linear which can be learned from training data by estimating distributions of confidence values from each modality. The scores are then translated and scaled to have zero mean, unit variance, and then remapped to a fixed interval of (0,1) using a hyperbolic tangent function. Note that it is tempting to parameterize the estimated distributions for normalization. However, such parameterization of distributions should be used with care, because the error rates of biometric systems are typically very small and a small error in estimating the tails of the distributions may result in a significant change in the error estimates (see Figure 7.3). Another common practice is to compute different scaling factors (weights) for each modality from training data, such that the accuracy of the combined classifier is maxi7 Multimodal Biometric Systems 242 mized. This weighted sum rule is expected to work better than the simple sum rule when the component classifiers have different strengths (i.e., different error rates). Figure 7.3. a) Genuine and impostor distributions for a fingerprint verification system (Jain et al., 2000) and a Normal approximation for the impostor distribution. Visually, the Normal approximation seems to be good, but causes significant decrease in performance compared to the non-parametric estimate as shown in the ROCs in b), where FMR is referred to as FAR (False Acceptance Rate) and (1-FNMR) as Genuine Acceptance Rate. ©Elsevier. Some schemes to combine multiple modalities in biometric systems have also been studied from a theoretical point of view. Through a theoretical analysis, Daugman (1999b) showed that if a strong biometric and a weak biometric are combined with an abstract level combination using either the AND or the OR voting rules, the performance of the combination will be worse than the better of the two individual biometrics. Hong, Jain, and Pankanti’s (1999) theoretical analysis that AND/OR voting strategies can improve performance only when certain conditions are satisfied confirmed Daugman’s findings. Their analysis further showed that a confidence level fusion is expected to significantly improve overall performance even in the case of combining a weak and a strong biometric. Kittler et al. (1998) introduced a sensitivity analysis to explain why the sum (or average) rule outperforms the other rules. They showed that the sum rule is less sensitive than the other similar rules (such as the “product” rule) to the error rates of individual classifiers in estimating posterior probabilities (confidence values). They claim that the sum rule is the most appropriate for combining different estimates of the same posterior probabilities (e.g., resulting from different classifier initializations). Prabhakar and Jain (2002) compared the sum and the product rules with the Neyman−Pearson combination scheme and showed that the product rule is worse than the sum rule when combining correlated features and both the sum rule and the product rules are inferior to the Neyman− Pearson combination scheme when combining weak and strong classifiers. 0 20 40 60 80 100 0 1 2 3 4 5 6 7 Normalized Matching Score Pe rc en ta ge ( % ) Imposter Genuine Nonparametric Imposter Distribution Normal Imposter Distribution Genuine Distribution 0 1 2 3 4 5 50 55 60 65 70 75 80 85 90 95 100 False Acceptance Rate (%) G en ui ne A cc ep ta nc e R at e (% ) Using Nonparametric Imposter Distribution Using Normal Imposter Distribution