在統計學和機器學習中，Lasso 演算法（，又譯最小絕對值收斂和選擇運算元、套索演算法）是一種同時進行特徵選擇和正則化（數學）的迴歸分析方法，旨在增強統計模型的預測準確性和可解釋性，最初由斯坦福大學統計學教授羅伯特蒂布希拉尼於 1996 年基於 Leo Breiman 的非負引數推斷（Nonnegative Garrote NNG）提出 Tibshirani Robert 1996 Regression Shrinkage and Selection via the lasso Journal of the Royal Statistical Society Series B methodological 58 1 Wiley 26788 httpwwwjstororgstable2346178 。Lasso 演算法最初用於計算最小二乘法模型，這個簡單的演算法揭示了很多估計量的重要性質，如估計量與嶺迴歸（Ridge regression，也叫吉洪諾夫正則化）和最佳子集選擇的關係，Lasso 係數估計值和軟閾值（soft thresholding）之間的聯絡。它也揭示了當協變數共線時，Lasso 係數估計值不一定唯一（類似標準線性迴歸）。 雖然最早是為應用最小二乘法而定義的演算法，lasso 正則化可以簡單直接地拓展應用於許多統計學模型上，包括廣義線性模型，廣義估計方程，成比例災難模型和 M 估計。Lasso 選擇子集的能力依賴於限制條件的形式並且有多種表現形式，包括幾何學，貝葉斯統計，和凸分析。 Lasso 演算法與基追蹤降噪聯絡緊密。 歷史來源 蒂布希拉尼最初使用 Lasso 來提高預測的準確性與迴歸模型的可解釋性，他修改了模型擬合的過程，在協變數中只選擇一個子集應用到最終模型中，而非用上全部協變數。這是基於有著相似目的，但方法有所不同的 Breiman 的非負引數推斷。 在 Lasso 之前，選擇模型中協變數最常用的方法是移步選擇，這種方法在某些情況下是準確的，例如一些協變數與模型輸出值有強相關性情況。然而在另一些情況下，這種方法會讓預測結果更差。在當時，嶺迴歸是提高模型預測準確性最常用的方法。嶺迴歸可以透過縮小大的迴歸係數來減少過擬合從而改善模型預測偏差。但是它並不選擇協變數，所以對模型的準確構建和解釋沒有幫助。 Lasso 結合了上述的兩種方法，它透過強制讓迴歸係數絕對值之和小於某固定值，即強制一些迴歸係數變為 0，有效地選擇了不包括這些迴歸係數對應的協變數的更簡單的模型。這種方法和嶺迴歸類似，在嶺迴歸中，迴歸係數平方和被強制小於某定值，不同點在於嶺迴歸只改變係數的值，而不把任何值設為 0。 基本形式 Lasso 最初為了最小二乘法而被設計出來，Lasso 的最小二乘法應用能夠簡單明瞭地展示 Lasso 的許多特性。 最小二乘 假設一個樣本包括 N 種事件，每個事件包括 p 個協變數和一個輸出值。讓 y_i 為輸出值，並且 x_ix_1x_2ldotsx_pT 為第 i 種情況的協變數向量，那麼 Lasso 要計算的目標方程就是： 對所有 sum_j1p beta_j leq t ，計算 min_beta_0 beta leftfrac 1N sum_i1N y_i beta_0 x_iT beta2 right 這裡 t 是一個決定規則化程度的預定的自由引數。 設 X 為協變數矩陣，那麼 X_ij x_i_j ，其中 x_iT 是 X 的第 i 行，那麼上式可以寫成更緊湊的形式： 對所有 beta _1 leq t ，計算 min_beta_0 beta leftfrac 1N left y beta_0 X beta right_22 right 這裡 beta _p left sum_i1N beta_i p right1p 是標準 ellp 範數， 1_N 是 N times 1 維的 1 的向量。 因為 hat beta_0 bar y bar xT beta ，所以有 y_i hat beta_0 x_iT beta y_i bar y bar xT beta x_iT beta y_i bar y x_i bar x T beta 對變數進行中心化是常用的資料處理方法。並且協方差一般規範化為 textstyle left sum_i1N x_ij2 1 right ，這樣得到的解就不會依賴測量的規模。 它的目標方程還可以寫為： min_beta in mathbb Rp leftfrac 1N left y X beta right_22 right text subject to beta _1 leq t 其拉格朗日形式為： min_beta in mathbb Rp leftfrac 1N left y X beta right_22 lambda beta _1 right 其中 t 和 lambda 的關係取決於資料特徵。 正交協變數 現在考慮一些 Lasso 迴歸估計的基本性質。 首先假定所有的協變數都是正交的，即 x_i mid x_j delta_ij ，其中 delta_ij 為 克羅內克 δ 函式。等價的矩陣寫法為 XT X I ，使用次梯度法可有如下的表達形式 begin align hat beta_j S_N lambdahat betatext OLS_j hat betatext OLS_j max left 0 1 frac N lambdahat betatext OLS_j right text 其中 hat betatext OLS XT X1 XT y end align S_alpha 用於表示軟閾值運算元，當這個值非常小的時候為 0。一個與之相近的記號 H_alpha 用來表示硬閾值運算元，將較小的數值記為 0 的同時保留原有的較大數值。 與嶺迴歸相比較，其中嶺迴歸的目標在於最小化 min_beta in mathbb Rp leftfrac 1N y X beta _22 lambda beta _22 right 即有 hat beta_j 1 N lambda1 hat betatext OLS_j 因此嶺迴歸是對 OLS 迴歸中所有的係數以一致的係數 1 N lambda1 縮放，並不會進行變數選擇。 同樣也可以對 best subset selection 演算法進行比較，其目標在於最小化 min_beta in mathbb Rp leftfrac 1N left y X beta right_22 lambda beta _0 right 其中 cdot _0 表示 ell0 norm，即 0 範數，被定義為該向量中非零元的個數。在這個例子中，可以得到 hat beta_j H_sqrt N lambda left hat betatext OLS_j right hat betatext OLS_j mathrm I left left hat betatext OLS_j right geq sqrt N lambda right 其中 H_alpha 被稱為軟閾值運算元， mathrm I 為示性函式。 總的來說，Lasso 估計量展現出了嶺迴歸和最佳子劃分演算法的係數收縮的優點，使得部分系數為 0。此外，在嶺迴歸全部使用一個常數係數縮放的時候，Lasso 迴歸會將一個接近 0 的係數變為 0。 相關協變數 對於一般的情況中，不同的協變數之間可能並不是獨立的，其中一種特例即為變數存在重複，例如變數 j 和變數 k，有 x_j x_k ， x_ji x_ij 。在這種情況下引數 beta_j 和 beta_k 的 Lasso 迴歸的估計量不是唯一確定的。 事實上，如果有一些 hat beta 中存在 hat beta_j hat beta_k geq 0 ，尋找一個 s in 01 進行變換，將 hat beta_j 變換為 s hat beta_j hat beta_k 的同時有 hat beta_k 變換為 1 s hat beta_j hat beta_k ，並保留其他引數不變，此時 Lasso 迴歸具有有效的連續性質。一些基於 Lasso 迴歸的改進，例如彈性網路正則化，旨在解決這個缺點。 一般形式 Lasso 正則化可以擴充套件為其他目標函式，例如廣義線性模型，廣義估計方程，比例風險模型和 M 估計。 有目標函式 frac 1N sum_i1N f x_i y_i alpha beta 其中 Lasso 正則化迴歸給出了下面模型的估計量 min_alpha beta frac 1N sum_i1N f x_i y_i alpha beta text subject to beta _1 leq t 在這裡只有 beta 是一個懲罰項， alpha 是一個自由變數，與最基本的模型中的 beta_0 變數一樣。 演算法解釋 幾何解釋 Lasso 迴歸可以使得某些項係數為 0，從幾何上來看，不同約束邊界形狀的嶺迴歸則不能。他們都可以解釋為最小化相同的目標函式 min_beta_0 beta leftfrac 1N left y beta_0 X beta right_22 right 但是有不同的約束條件：在 Lasso 迴歸中為 beta _1 leq t 而在嶺迴歸中為 beta _22 leq t 。 The figure shows that the constraint region defined by the ell1 norm is a square rotated so that its corners lie on the axes in general a crosspolytope while the region defined by the ell2 norm is a circle in general an nsphere which is rotationally invariant and therefore has no corners As seen in the figure a convex object that lies tangent to the boundary such as the line shown is likely to encounter a corner or a higherdimensional equivalent of a hypercube for which some components of beta are identically zero while in the case of an nsphere the points on the boundary for which some of the components of beta are zero are not distinguished from the others and the convex object is no more likely to contact a point at which some components of beta are zero than one for which none of them are Making λ easier to interpret with an accuracysimplicity tradeoff The lasso can be rescaled so that it becomes easy to anticipate and influence the degree of shrinkage associated with a given value of lambda It is assumed that X is standardized with zscores and that y is centered zero mean Let beta_0 represent the hypothesized regression coefficients and let b_OLS refer to the dataoptimized ordinary least squares solutions We can then define the Lagrangian as a tradeoff between the insample accuracy of the dataoptimized solutions and the simplicity of sticking to the hypothesized values This results in min_beta in mathbb Rp leftfrac yXbetayXbetayXbeta_0yXbeta_0 2lambda sum_i1p frac beta_ibeta_0iq_i right where q_i is specified below The first fraction represents relative accuracy the second fraction relative simplicity and lambda balances between the two Given a single regressor relative simplicity can be defined by specifying q_i as b_OLSbeta_0 which is the maximum amount of deviation from beta_0 when lambda0 Assuming that beta_00 the solution path can be defined in terms of R2 b_ell_1 begin cases 1lambdaR2 b_OLS mbox if lambda leq R2 0 mbox if lambdaR2 end cases If lambda0 the ordinary least squares solution OLS is used The hypothesized value of beta_00 is selected if lambda is bigger than R2 Furthermore if R21 then lambda represents the proportional influence of beta_00 In other words lambdatimes100 measures in percentage terms the minimal amount of influence of the hypothesized value relative to the dataoptimized OLS solution If an ell_2norm is used to penalize deviations from zero given a single regressor the solution path is given by b_ell_2bigg 1frac lambdaR21lambdabigg1 b_OLS Like b_ell_1 b_ell_2 moves in the direction of the point lambda R2 b0 when lambda is close to zero but unlike b_ell_1 the influence of R2 diminishes in b_ell_2 if lambda increases see figure Given multiple regressors the moment that a parameter is activated ie allowed to deviate from beta_0 is also determined by a regressors contribution to R2 accuracy First R21frac yXbyXbyXbeta_0yXbeta_0 An R2 of 75 means that insample accuracy improves by 75 if the unrestricted OLS solutions are used instead of the hypothesized beta_0 values The individual contribution of deviating from each hypothesis can be computed with the p x p matrix RotimesXtilde y_0Xtilde y_0 XX1tilde y_0tilde y_01 where tilde y_0yXbeta_0 If bb_OLS when R2 is computed then the diagonal elements of Rotimes sum to R2 The diagonal Rotimes values may be smaller than 0 or less often larger than 1 If regressors are uncorrelated then the ith diagonal element of Rotimes simply corresponds to the r2 value between x_i and y A rescaled version of the adaptive lasso of can be obtained by setting q_mbox adaptive lassoib_OLSibeta_0i If regressors are uncorrelated the moment that the ith parameter is activated is given by the ith diagonal element of Rotimes Assuming for convenience that beta_0 is a vector of zeros b_i begin cases 1lambdaR_iiotimes b_OLSi mbox if lambda leq R_iiotimes 0 mbox if lambdaR_iiotimes end cases That is if regressors are uncorrelated lambda again specifies the minimal influence of beta_0 Even when regressors are correlated the first time that a regression parameter is activated occurs when lambda is equal to the highest diagonal element of Rotimes These results can be compared to a rescaled version of the lasso by defining q_mbox lassoifrac 1p sum_l b_OLSlbeta_0l which is the average absolute deviation of b_OLS from beta_0 Assuming that regressors are uncorrelated then the moment of activation of the ith regressor is given by tilde lambda_text lassoi frac 1psqrt Rotimes_i sum_l1psqrt Rotimes_l For p1 the moment of activation is again given by tilde lambda_text lassoiR2 If beta_0 is a vector of zeros and a subset of p_B relevant parameters are equally responsible for a perfect fit of R21 then this subset is activated at a lambda value of frac 1p The moment of activation of a relevant regressor then equals frac 1pfrac 1sqrt p_B p_Bfrac 1sqrt p_Bfrac 1p In other words the inclusion of irrelevant regressors delays the moment that relevant regressors are activated by this rescaled lasso The adaptive lasso and the lasso are special cases of a 1ASTc estimator The latter only groups parameters together if the absolute correlation among regressors is larger than a userspecified value Bayesian interpretation Just as ridge regression can be interpreted as linear regression for which the coefficients have been assigned normal prior distributions lasso can be interpreted as linear regression for which the coefficients have Laplace prior distributions The Laplace distribution is sharply peaked at zero its first derivative is discontinuous at zero and it concentrates its probability mass closer to zero than does the normal distribution This provides an alternative explanation of why lasso tends to set some coefficients to zero while ridge regression does not Convex relaxation interpretation Lasso can also be viewed as a convex relaxation of the best subset selection regression problem which is to find the subset of leq k covariates that results in the smallest value of the objective function for some fixed k leq n where n is the total number of covariates The ell0 norm cdot _0 the number of nonzero entries of a vector is the limiting case of ellp norms of the form textstyle x _p left sum_i1n x_j p right1p where the quotation marks signify that these are not really norms for p 1 since cdot _p is not convex for p 1 so the triangle inequality does not hold Therefore since p 1 is the smallest value for which the ellp norm is convex and therefore actually a norm lasso is in some sense the best convex approximation to the best subset selection problem since the region defined by x _1 leq t is the convex hull of the region defined by x _p leq t for p 1 應用 LASSO 已被應用於經濟和金融領域，可以改善預測結果並選擇有時被忽視的變數。例如：公司破產預測和高增長公司預測。 參見 降維 特徵選擇 參考文獻 Category 迴歸分析 Category 統計方法