Multiple comparison procedures applied to model selection
This paper presents a new approach to model selection based on hypothesis testing. We first describe a procedure to generate different scores for any candidate model from a single sample of training data and then discuss how to apply multiple comparison procedures (MCP) to model selection. MCP statistical tests allow us to compare three or more groups of data while controlling the probability of making at least one Type I error. The complete procedure is illustrated on several model selection tasks, including the determination of the number of hidden units for feed-forward neural networks and the number of kernels for RBF networks.
multiple comparison procedures model selection generalization network size problem complexity