Lasso算法
算法解释 > Making λ easier to interpret with an accuracy-simplicity tradeoff

The lasso can be rescaled so that it becomes easy to anticipate and influence the degree of shrinkage associated with a given value of  \lambda . It is assumed that  X is standardized with z-scores and that  y is centered (zero mean). Let  \beta _{0} represent the hypothesized regression coefficients and let  b_{OLS} refer to the data-optimized ordinary least squares solutions. We can then define the Lagrangian as a tradeoff between the in-sample accuracy of the data-optimized solutions and the simplicity of sticking to the hypothesized values. This results in
where  q_{i} is specified below. The first fraction represents relative accuracy, the second fraction relative simplicity, and  \lambda  balances between the two.
Given a single regressor, relative simplicity can be defined by specifying  q_{i} as  |b_{OLS}-\beta _{0}|, which is the maximum amount of deviation from  \beta _{0} when  \lambda =0. Assuming that  \beta _{0}=0, the solution path can be defined in terms of  R^{2}:
If  \lambda =0, the ordinary least squares solution (OLS) is used. The hypothesized value of  \beta _{0}=0 is selected if  \lambda  is bigger than  R^{2}. Furthermore, if  R^{2}=1, then  \lambda  represents the proportional influence of  \beta _{0}=0. In other words,  \lambda \times 100\% measures in percentage terms the minimal amount of influence of the hypothesized value relative to the data-optimized OLS solution.
If an  \ell _{2}-norm is used to penalize deviations from zero given a single regressor, the solution path is given by
b_{\ell _{2}}={\bigg (}1+{\frac {\lambda }{R^{2}(1-\lambda )}}{\bigg )}^{-1}b_{OLS}. Like  b_{\ell _{1}},  b_{\ell _{2}} moves in the direction of the point  (\lambda =R^{2},b=0) when  \lambda  is close to zero; but unlike  b_{\ell _{1}}, the influence of  R^{2} diminishes in  b_{\ell _{2}} if  \lambda  increases (see figure). 
Given multiple regressors, the moment that a parameter is activated (i.e. allowed to deviate from  \beta _{0}) is also determined by a regressor's contribution to  R^{2} accuracy. First,
An  R^{2} of 75% means that in-sample accuracy improves by 75% if the unrestricted OLS solutions are used instead of the hypothesized  \beta _{0} values. The individual contribution of deviating from each hypothesis can be computed with the  p x  p matrix
where  {\tilde {y}}_{0}=y-X\beta _{0}. If  b=b_{OLS} when  R^{2} is computed, then the diagonal elements of  R^{\otimes } sum to  R^{2}. The diagonal  R^{\otimes } values may be smaller than 0 or, less often, larger than 1. If regressors are uncorrelated, then the  i^{th} diagonal element of  R^{\otimes } simply corresponds to the  r^{2} value between  x_{i} and  y.
A rescaled version of the adaptive lasso of can be obtained by setting  q_{{\mbox{adaptive lasso}},i}=|b_{OLS,i}-\beta _{0,i}|. If regressors are uncorrelated, the moment that the  i^{th} parameter is activated is given by the  i^{th} diagonal element of  R^{\otimes }. Assuming for convenience that  \beta _{0} is a vector of zeros,
That is, if regressors are uncorrelated,  \lambda  again specifies the minimal influence of  \beta _{0}. Even when regressors are correlated, the first time that a regression parameter is activated occurs when  \lambda  is equal to the highest diagonal element of  R^{\otimes }.
These results can be compared to a rescaled version of the lasso by defining  q_{{\mbox{lasso}},i}={\frac {1}{p}}\sum _{l}|b_{OLS,l}-\beta _{0,l}|, which is the average absolute deviation of  b_{OLS} from  \beta _{0}. Assuming that regressors are uncorrelated, then the moment of activation of the  i^{th} regressor is given by
For  p=1, the moment of activation is again given by  {\tilde {\lambda }}_{{\text{lasso}},i}=R^{2}. If  \beta _{0} is a vector of zeros and a subset of  p_{B} relevant parameters are equally responsible for a perfect fit of  R^{2}=1, then this subset is activated at a  \lambda  value of  {\frac {1}{p}}. The moment of activation of a relevant regressor then equals  {\frac {1}{p}}{\frac {1}{\sqrt {p_{B}}}}p_{B}{\frac {1}{\sqrt {p_{B}}}}={\frac {1}{p}}. In other words, the inclusion of irrelevant regressors delays the moment that relevant regressors are activated by this rescaled lasso. The adaptive lasso and the lasso are special cases of a '1ASTc' estimator. The latter only groups parameters together if the absolute correlation among regressors is larger than a user-specified value.