Abstract-A test sequence is used to select the best rule from a rich class of discrimination rules defined in terms of the training sequence . The Vapnik-Chervonenkis and related inequalities are used to obtain distribution-free bounds on the difference between the probability of error o€ the selected rule and the probability of error of the best rule in the given class . The bounds are used to prove the consistency and asymptotic optimality for several popular classes, including linear discriminators, nearest neighbor rules, kernel-based rules, histogram rules, binary tree classifiers, and Fourier series classifiers . In particular, the method can be used to choose the smoothing parameter in kernel-based rules, to choose k in the k-nearest neighbor rule, and to choose between parametric and nonparametric rules. Index Terms-Automatic parameter selection, empirical risk, error estimation, nonparametric rule, probability of error, statistical pattern recognition, Vapnik-Chervonenkis inequality . N pattern recognition, we normally use the data, either directly (via formulas) or indirectly (by peeking), in the selection of a discrimination rule and/or its parameters . For example, a quick inspection of the data can convince us that a linear discriminator is appropriate in a given situation . The actual position of the discriminating hyperplane is usually determined from the data . In other words, we choose our discriminator from a class D of discriminators . This class can be small (e .g ., ' all k-nearest neighbor rules'') or large (e .g ., "all linear and quadratic discriminators, and all nonparametric discriminators of the kernel type with smoothing factor h > 0") . If we knew the underlying distribution of the data, then the selection process would be simple : we would pick the Bayes rule . Unfortunately, the Bayes rule is not in D unless we are incredibly lucky . Also, the underlying distribution is not known . Thus, it is important to know how close we are to the performance of the best discriminator in D . If D is large enough, then hopefully, the performance of the best discriminator in it is close to that of the Bayes discriminator . There are two issues here which should be separated from each other . 1) The closeness of the best element of D to the Bayes rule .
For a discussion of these and other related partitioning rules in the context of pattern recognition, we refer to the survey paper of REF .
Automatic Pattern Recognition: A Study of the Probability of Error
IEEE Trans. Pattern Anal. Mach. Intell.
IEEE Trans. Pattern Anal. Mach. Intell.
Computer Science