A fast and robust model selection algorithm for multi-input multi-output support vector machine
Multi-Input Multi-Output (MIMO) regression estimation problems widely exist in engineering fields. As an efficient approach for MIMO modeling, multi-dimensional support vector regression, named M-SVR, is generally capable of obtaining better predictions than many traditional methods. However, M-SVR is sensitive to the perturbation of hyper-parameters when facing small-scale sample problems, and most of currently used model selection methods for conventional SVR cannot be applied to M-SVR directly due to its special structure. In this paper, a fast and robust model selection algorithm for M-SVR is proposed. Firstly, a new training algorithm for M-SVR is proposed to reduce efficiently the numerical errors in training procedure. Based on this algorithm, a new leave-one-out (LOO) error estimate for M-SVR is derived through a virtual LOO cross-validation procedure. This LOO error estimate can be straightway calculated once a training process ended with less computational complexity than traditional LOO method. Furthermore, a robust implementation of this LOO estimate via Cholesky factorization is also proposed. Finally, the gradients of the LOO estimate are calculated, and the hyper-parameters with lowest LOO error can be found by means of gradient decent method. Experiments on toy data and real-life dynamical load identification problems are both conducted, demonstrating comparable results of the proposed algorithm in terms of generalization performance, numerical stability and computational cost.
model selection support vector machine mimo gradient decent leave-one-out error