  An Orthogonal Least Squares (OLS) based feature selection method is proposed
for both binomial and multinomial classification. The novel Squared Orthogonal
Correlation Coefficient (SOCC) is defined based on Error Reduction Ratio (ERR)
in OLS and used as the feature ranking criterion. The equivalence between the
canonical correlation coefficient, Fisher's criterion, and the sum of the SOCCs
is revealed, which unveils the statistical implication of ERR in OLS for the
first time. It is also shown that the OLS based feature selection method has
speed advantages when applied for greedy search. The proposed method is
comprehensively compared with the mutual information based feature selection
methods in 2 synthetic and 7 real world datasets. The results show that the
proposed method is always in the top 5 among the 10 candidate methods. Besides,
the proposed method can be directly applied to continuous features without
discretisation, which is another significant advantage over mutual information
based methods.
