We derive sharp performance bounds for least squares regression with L1 regularization from parameter estimation accuracy and feature selection quality perspectives. Moreover, the result leads to an extended view of feature selection that allows less restrictive conditions than some recent work. Based on the theoretical insights, a novel two-stage L1-regularization procedure with selective penalization is analyzed. It is shown that if the target parameter vector can be decomposed as the sum of a sparse parameter vector with large coefficients and another less sparse vector with relatively small coefficients, then the two-stage procedure can lead to improved performance.
The two-stage procedure in REF applies "selective penalization" in the second stage.
Some sharp performance bounds for least squares regression with $L_1$ regularization
Annals of Statistics 2009, Vol. 37, No. 5A, 2109-2144
Mathematics