Fast adaptive LDA using quasi-Newton algorithm
A new adaptive algorithm for linear discriminant analysis (LDA) based on the quasi-Newton optimization technique is presented. The proposed algorithm uses the secant method for adaptive computation of the inverse Hessian matrix and the NewtonRaphson method for optimal estimation of the step size at each iteration. Current adaptive method, based on the NewtonRaphson optimization technique, uses a direct calculation of the inverse Hessian matrix, which can be both laborious to calculate and invert for systems with large number of dimensions. The new algorithm has the advantage of automatic optimal selection of the step size using the current data samples and also adaptive computation of the inverse Hessian matrix that overcomes its sensitivity to data condition. Based on the new adaptive algorithm, we present a self-organizing neural network for adaptive computation of the square root of the inverse covariance matrix and use it in cascaded form with a principal component analysis (PCA) network for LDA. Experimental results demonstrated fast convergence and lower computational cost of the new algorithm compared to the adaptive gradient descent and NewtonRaphson LDA algorithms, respectively and justified its advantages for on-line pattern recognition applications with stationary and non-stationary multidimensional input data.
secant method newtonraphson optimization self-organizing neural network adaptive linear discriminant analysis gradient descent optimization