The Solution Path of the Group Lasso
Abstract
We prove continuity of the solution path for the group lasso, a popular method of computing groupsparse models.
Unlike the more classical lasso method, the group lasso solution path is non-linear and cannot be evaluated algorithmically.
To circumvent this, we ﬁrst characterize the group lasso solution set and then show how to construct an implicit function for the min-norm path.
We prove our implicit representation is continuous almost everywhere and extend this to continuity everywhere when the group lasso solution is unique.
These results can be viewed as extending solution path analyses from the lasso to the group lasso and imply that grid-search is a sensible approach to hyper-parameter selection.
Our work applies to linear models as well as convex reformulations of neural networks and provides new tools for understanding solution paths of shallow ReLU models.
1. Introduction
The group lasso extends the classical lasso algorithm (R. Tibshirani, 1996) for feature-sparse regression to problems with group structure.
Similar to the lasso, the group lasso is an embedded method ; it achieves group sparsity by augmenting the training objective with a sparsity-inducing penalty function, in this case the sum of the (cid:96) 2 norms of each feature group (Bakin et al., 1999;
Lin and
H. H. Zhang, 2003; Yuan and Lin, 2006).
Solving the resulting non-smooth optimization problem gives a group-sparse solution, with the degree of sparsity controlled by a regularization parameter.
Group-sparsity is desirable in learning problems for both computational and statistical reasons.
Computationally, group-sparse models require less memory and can speed-up prediction by leveraging fast algorithms for sparse linear algebra (Wen et al., 2016).
They may also generalize better at test time than dense models (Huang and T. Zhang, 2010; Mitra and C.-H. Zhang, 2016).
Finally, group-sparse models like the group lasso are consistent in certain settings, meaning they recover the support of the ground-truth model (Bach, 2008; Liu and J. Zhang, 2009; Nardi and Rinaldo, 2008).
The group lasso also has deep connections to neural network optimization.
Recent work by Pilanci and Ergen (2020) shows that two-layer neural networks with ReLU activations can be reformulated as a group lasso problem with additional constraints.
Dropping these constraints yields a simpler “gated ReLU” neural network with the same expressive power (Fiat et al., 2019; Mishkin et al., 2022).
In both cases, the degree of group sparsity corresponds directly to the number of hidden units in the ﬁnal network.
Thus, properties of the group lasso translate into properties of neural nets.
This paper studies the solution function—the mapping from regularization parameter to optimal model—of the linear group lasso problem.
Our main contribution is a proof that the solution function is continuous when the group lasso solution is unique.
Although technical, continuity of the solution function is necessary for efﬁcient tuning of the regularization parameter (see, e.g. Nesterov et al.
(2018, Sec. 1.1)) and our result is an important “sanity check” that parameter tuning is possible for the group lasso.
As part of our proof, we characterize the set of optimal models as well as the minnorm model, and provide new sufﬁcient conditions for the group lasso solution to be unique.
Finally, our work lays the foundation for analyzing the solution functions of (convex) neural networks.
1.1. Related Work
The Lasso Solution Path : Efron et al.
(2004) and Osborne et al.
(2000) develop homotopy methods for directly computing the lasso solution path, which is piecewise linear.
These works establish continuity of the lasso solution as a side affect, but also require the solution to be unique.
Hastie et al.
(2007) connect the lasso path to forward stage-wise regression, while R. J. Tibshirani (2013) extend many results, including path continuity, to the min-norm solution.
Beyond the Lasso : Limited results exist outside of the lasso setting.
R. J. Tibshirani and Taylor (2011) analyze the generalized lasso and provide path-computation algorithms in this setting.
Vaiter et al.
(2012) establish almost-everywhere continuity of the group lasso solution function with respect to the targets, rather than the regularization parameter.
2. Notation
We use lower-case a to denote vectors and upper-case A for matrices.
Calligraphic letters C denote sets.
The boundary of a set is written bd ( C ) .
For a block of indices b i ⊆ { 1 , . . .
, d } , we write A b i for the sub-matrix of columns indexed by b i .
Similarly, a b i is the sub-vector indexed by b i .
If M is a collection of blocks, then A M is the submatrix and a M the sub-vector with columns/elements indexed by blocks in the collection.
Finally, |M| is cardinality of the union of blocks in M .
3. The Group Lasso
Let X ∈ R n × d be a data matrix, y ∈ R n the associated vector of targets, and B = { b 1 , . . .
, b m } a disjoint partition of the feature indices { 1 , . . .
, d } .
Given a regularization parameter λ ≥ 0 , the linear group lasso solves the following regularized regression problem:
[Equation]
Solutions to Eq.
(1) are block sparse when λ is sufﬁciently large, meaning w b i =
for some subset of b i .
This is similar to the feature sparsity given by the lasso, to which the group lasso naturally reduces when b i = { i } for each b i ∈ B .
Our primary interest in this work is the solution function or “regularization path”,
[Equation]
For a general data matrix, f λ is not strictly convex and the linear group lasso problem may admit multiple solutions.
As such, W ∗ ( λ ) ⊂
R d is set-valued and only the min-norm solution mapping
[Equation]
[Figure]
o e ff i
c i e n t s
[Figure]
deﬁnes a function.
Even in the non-unique case, the model ﬁt ˆ y ( λ ) =
Xw is the same for all w ∈ W ∗ ( λ ) (Vaiter et al., 2012, Lemma 2).
Moreover, if λ > 0 , then it is straightforward to deduce that the sum of group norms (cid:80) i ∈B (cid:107) w b i (cid:107) 2 is also constant over W ∗ ( λ ) .
Uniqueness of ˆ y ( λ ) extends to the residual vector r ( λ ) =
y − ˆ y ( λ ) and the block correlation vectors v b i ( λ ) =
X (cid:62) b i r ( λ ) .
We write v ∈ R d as the concatenation of the v b i blocks; v plays a critical role in deﬁning W ∗ ( λ ) .
Unlike the lasso, the group lasso solution function is not piece-wise linear (see Fig. 1).
This prevents us from obtaining an explicit expression for the solution path between breakpoints, where features enter or exit the active set.
We overcome this difﬁculty by relying only on an implicit characterization of the path provided by the implicit function theorem.
We start towards this goal by developing optimality conditions for the group lasso problem.
Since Eq.
(1) is a convex optimization problem, ﬁrst-order (FO) optimality conditions are both necessary and sufﬁcient for weights w to be globally optimal.
Sub-differentiating the optimization problem gives the following FO conditions:
[Equation]
otherwise, (2) showing that w b i = 0 when (cid:107) v b i (cid:107) 2 < λ .
As a result, the equicorrelation set
[Equation] [Equation]
contains all blocks which may be active for ﬁxed λ .
The complement of E λ in B is the inactive set, denoted I λ .
We combine FO conditions with uniqueness of ˆ y ( λ ) to characterize the solution function in terms of Null ( X E λ ) .
Proposition 1 Let λ > 0 and N λ =
Null ( X E λ ) (cid:84) { z : z b i ∈ Span ( v b i ) , i ∈ E λ } .
The optimal set is
[Equation]
Proposition 1 extends a similar result for the lasso solution to the group lasso (R. J. Tibshirani, 2013,
Eq. 9).
It implies the group lasso solution is unique when the columns of X are linearly independent; we will use it later to obtain a more reﬁned condition for uniqueness.
See Appendix A for proof.
4. The Minimum-norm Solution
Now we turn our attention to the min-norm solution and its path.
Unlike W ∗ , the min-norm solution is a single-valued function of λ and can be analyzed precisely.
Our focus is on developing new characterizations of w ∗ .
All proofs are deferred to Appendix B.
First, we introduce additional notation which will be necessary throughout our discussion.
Divide the equicorrelation set into active and transitioning blocks, respectively,
[Equation]
Intuitively, T λ ( w ) is the set of blocks which may become active (i.e. enter A λ ) or cease to be equicorrelated in a neighbourhood of λ .
We write A ∗ λ and T ∗ λ for the active and transiting blocks of the min-norm solution.
By manipulating FO conditions, we now give a concise expression for w ∗A ∗ λ in terms of the correlation vector v .
Lemma
For λ > 0 , the active blocks of the min-norm solution are given by
[Equation]
As a corollary, w ∗A ∗ λ is the unique solution to the group lasso which is orthogonal to Null ( X A ∗ λ ) .
Although we state this lemma in terms of X A ∗ λ , it is easily extended to X E λ by including FO conditions for the blocks T ∗ λ in the same argument.
Thus, Lemma 2 also has computational consequences: given any solution w ∈ W ∗ ( λ ) , the unique correlation vector v ( λ ) can be computed and used to ﬁnd w ∗ as the solution to a linear system.
Since w ∗A ∗ λ has no component of Null ( X A ∗ λ ) , we can also compute it via a reduced-block optimization problem constrained to Row ( X A ∗ λ ) .
Lemma 3 Suppose the support of the min-norm solution is A ∗ λ .
Then the active blocks are the unique solution to the following optimization problem:
[Equation]
Introducing dual variables η ∈ R | A ∗ λ | , the KKT system for Eq.
(5) is
[Equation]
Satisfying Eq.
(6) is both necessary and sufﬁcient for a primal-dual pair ( w A ∗ λ , η ) to be optimal because the reduced-block problem is convex with linear constraints.
It turns out that the space of dual solutions has a simple structure which is easily incorporated into the KKT system.
Lemma 4 Fix λ ≥ 0 and suppose the support of the min-norm solution is A ∗ λ .
Then the space of dual solutions to the KKT system is exactly Row ( X A ∗ λ ) .
As a result, the min-norm dual solution is simply η ∗ = 0 and is attained by adding the constraint
X + A ∗ λ
X A ∗ λ η = 0 .
Since ( I − X + A ∗ λ X A ∗ λ ) and X + A ∗ λ X A ∗ λ project onto orthogonal spaces, we can incorporate this constraint into the KKT system as follows:
[Equation]
[Equation]
We call Eq.
(7) the unique KKT system.
The following proposition uses the unique KKT system to obtain a local, implicit solution function for the reduced-block problem.
We then extend to this to the full min-norm solution and obtain local continuity of the solution path as a consequence.
Proposition 5 Let Λ be an open interval on which A ∗ λ is constant.
For every λ ∈ Λ , there exists a neighbourhood of O of λ on which w ∗ is continuously differentiable.
Moreover, the path for w ∗A ∗ λ is the same on O as that for the reduced-block problem in Eq.
Proposition 5 is the key technical result from which we derive continuity of the solution path.
4.1. Continuity of the Solution Path
Now we consider the case where the solution to the group Lasso is unique.
In this setting, the solution map W ∗ is equal to the min-norm solution w ∗ and the analysis is considerably eased.
To facilitate the analysis, we introduce a sufﬁcient condition for uniqueness of the group lasso solution.
Assumption 6 (Group General Position) For every E ⊆ B , |E| ≤ n + 1 , there do not exist unit vectors z b i ∈ R | b i | such that for any j ∈ E ,
[Equation]
We call Assumption 6 group general position because it is a natural extension of general position to groups of column vectors.
General position itself is an extension of afﬁne independence and is sufﬁcient for the lasso solution to be unique (R. J. Tibshirani, 2013). Group general position is strictly weaker than linear independence of the columns of X , but neither implies nor is implied by general position (Proposition 11).
See Proposition 12 for a formal proof of sufﬁciency.
Following Vaiter et al.
(2012), deﬁne the transition space to be
[Equation]
By construction, H is the set of regularization parameters at which a block b i transitions from being active to inactive or vice-versa.
Note that, as a direct consequence of Proposition 5, H is the exactly set of potential discontinuities of W ∗ .
Crucially, we show H contains no intervals.
Lemma 7 The transition space H is discrete.
Since H is discrete and Proposition 5 implies W ∗ is continuous away from H , proving continuity of the path now reduces to showing that the left and right limits of the unique solution are equal at each “break point” in H .
Doing so yields our main result.
Theorem 8 Suppose group general position holds.
Then the unique group lasso solution path is continuous.
The proof of Theorem 8 relies on uniqueness of the solution; without uniqueness, we can only show the right- and left-hand limits of the min-norm solution at ¯ λ ∈ H are also solutions to the group lasso problem.
Since we do not recover their active blocks from the argument, we cannot show they are the min-norm solution.
Thus, we develop machinery for characterizing the min-norm solution path, but additional work is required to extend continuity to this more general setting.
5. Conclusion
This paper analyzes the path of the group lasso solution set as the regularization parameter is varied.
We prove that the path, called the solution function , is continuous when the group lasso admits a unique solution and provide a new sufﬁcient condition for such uniqueness to hold.
Outside of the unique-solution setting, we characterize the optimal solution set and show how to compute the minnorm solution as a linear system of the block correlation vectors.
In several cases, our research extends results for the lasso to the more general setting of the group lasso.
The main technical difﬁculty in our analysis is that the group lasso path cannot be computed algorithmically, unlike the piecewise linear lasso.
In particular, non-linearity makes computing the active set of the min-norm solution challenging when the group lasso admits multiple solutions.
As such, we leave proving continuity of the min-norm path to future work.
A consequence of our results is that convex reformulations of neural networks with gated ReLU activations also have continuous solution paths under mild assumptions.
We believe it is possible to extend continuity to convex reformulations of ReLU neural networks by incorporating the necessary constraints into our analysis.
We leave this exciting direction of research to future work.
Appendix A. Group Lasso: Proofs
Proposition 1 Let λ > 0 and N λ =
Null ( X E λ ) (cid:84) { z : z b i ∈ Span ( v b i ) , i ∈ E λ } .
The optimal set is
W
[Equation]
Proof For ease of notation, let
[Equation]
Recall that ˆ y
=
Xw is constant over W ∗ ( λ ) and each w ∈ W ∗ ( λ ) can only have support on E λ by
FO conditions.
Dropping the zero entries of w , ˆ y = X E λ w E λ
and we deduce that
[Equation]
for every w ∈ W ∗ ( λ ) .
Let us show that z ∈ N λ .
If w ∗ b i (cid:54) = 0 , then ﬁrst order optimality conditions require
[Equation]
from which we have w ∗ b i ∈ Span ( v b i ) .
Thus, w b i = 0 implies z b i = − w ∗ b i ∈ Span ( v b i ) (this also trivially holds when w ∗ b i = 0 ).
Otherwise, if w b i (cid:54) = 0 , then ﬁrst-order optimality again implies w b i
[Equation]
since w ∗ b i ∈ Span ( v b i ) .
As a result, z ∈ N λ .
It is straightforward to check that ﬁrst-order optimality implies w I λ =
so that W ∗ ⊆
X .
For the reverse inclusion, let w ∈ X .
Clearly
[Equation]
so that w has the correct model ﬁt.
If w b i =
and i ∈ I λ , then
[Equation]
shows FO conditions are satisﬁed.
Alternatively, if i ∈ E λ , then X (cid:62) b i ( y − Xw )
= v b i and FO conditions are satisﬁed.
Finally, we check blocks for which w b i (cid:54) = 0 .
Membership in X implies
[Equation]
and w b i again satisﬁes ﬁrst-order optimality conditions.
Putting these conditions together implies w ∈ W ∗ .
This completes the proof.
Appendix B. The Minimum-norm Solution: Proofs
Lemma
For λ > 0 , the active blocks of the min-norm solution are given by
[Equation]
Proof Recall the deﬁnition of the block correlation vector,
[Equation]
which shows v b i ∈ Row ( X b i ) for each b
i ∈
A λ and v A ∗ λ ∈ Row
( X A ∗ λ ) .
As a result, v A ∗ λ is unchanged by the projection onto Row ( X A ∗ λ ) , i.e.
[Equation]
Combining this with the deﬁnition of v b i and summing over the non-zero blocks of the min-norm solution, we obtain
[Equation]
which implies that w (cid:48) satisfying
[Equation]
and w (cid:48)B\A ∗ λ = 0 deﬁnes one solution to the group Lasso.
Moreover, w (cid:48)A ∗ λ is orthogonal to Null ( X A ∗ λ ) .
Now, suppose that the min-norm solution is not orthogonal to Null ( X A ∗ λ ) .
Then w ∗A ∗ λ = w (cid:48)A ∗ λ + a , where z ∈ Null ( X A ∗ λ ) , z (cid:54) = 0 and
[Equation]
which is a contradiction.
We conclude w ∗A ∗ λ = w (cid:48)A ∗ λ as this is only solution which is orthogonal to
Null ( X A ∗ λ ) .
Lemma 3 Suppose the support of the min-norm solution is A ∗ λ .
Then the active blocks are the unique solution to the following optimization problem:
[Equation]
Proof Recall that w ∗ A ∗ λ is orthogonal to Null (cid:16)
X
A ∗ λ (cid:17) .
Thus,
[Equation]
and w ∗A ∗ λ is a feasible point for the reduced-block problem.
Since A ∗ λ contains all active blocks of w ∗ , we know
[Equation]
so that w ∗A ∗ λ is optimal for the reduced-block problem.
Let w (cid:48)A ∗ λ (cid:54) =
w ∗A ∗ λ be another solution to the reduced-block problem.
Vaiter et al.
(2012, Lemma 2) implies w (cid:48)A ∗ λ must have the same model
ﬁt ˆ y =
X
A ∗ λ w (cid:48)A ∗ λ as w ∗A ∗ λ .
Thus, z = w (cid:48)A ∗ λ − w ∗A ∗ λ ∈
Null ( A ∗ λ ) , which implies
[Equation]
We deduce z = 0 and w ∗A ∗ λ is the unique solution.
Lemma 4 Fix λ ≥ 0 and suppose the support of the min-norm solution is A ∗ λ .
Then the space of dual solutions to the KKT system is exactly Row ( X A ∗ λ ) .
Proof Let η ∈ Row ( X A ∗ λ ) .
The min-norm solution w ∗ is member of W ∗ ( λ ) .
Thus, FO conditions imply
[Equation]
for all b i ∈ A ∗ λ .
We also have
[Equation]
by choice of η , which implies any η ∈ Row ( X A ∗ λ ) satisﬁes the ﬁrst |A ∗ λ | constraints of the KKT system.
Since w ∗ is the min-norm solution, it is orthogonal to Null ( X A ∗ λ ) , implying
[Equation]
We conclude that ( w ∗ , η ) is a KKT point.
Lemma 9 Fix λ ≥ 0 and ¯ w, ¯ η ∈ R |A ∗ λ | .
The Jacobian of the unique KKT system is full rank at ( ¯ w, ¯ η ) .
Proof Let M ( w ) be the block-diagonal projection matrix given by
[Equation]
The Jacobian of the unique KKT system with respect to ( w A ∗ λ , η ) is
[Equation]
where P A ∗ λ and P ⊥A ∗ λ are the projection matrices for Row ( X A ∗ λ ) and Null ( X A ∗ λ ) , respectively.
Let w, η ∈ R |A
∗ λ | such that ( w, η ) (cid:54) = 0 and assume
[Equation]
If P A ∗ λ η (cid:54) = 0 or P ⊥A ∗ λ w (cid:54) =
then the second block cannot be zero.
Thus, η ∈ Null ( X A ∗ λ ) and w ∈ Row ( X A ∗ λ ) must hold.
We deduce
[Equation]
Since w ∈ Row ( X A ∗ λ ) , it holds that X A ∗ λ w (cid:54) = 0 and thus
[Equation]
since M ( ¯ w ) is positive semi-deﬁnite and w ⊥ η .
But this is a contradiction.
We conclude Null ( J λ ) = { 0 } .
Proposition 5 Let Λ be an open interval on which A ∗ λ is constant.
For every λ ∈ Λ , there exists a neighbourhood of O of λ on which w ∗ is continuously differentiable.
Moreover, the path for w ∗A ∗ λ is the same on O as that for the reduced-block problem in Eq.
Proof Let λ ∈ Λ
Then, there exists a unique η such that Eq.
(7) is satisﬁed at ( w ∗A ∗ λ , η ) .
(In fact,
Lemma 4 implies η = 0 is exactly the optimal dual parameter.)
Furthermore, Lemma 9 shows that the Jacobian of the unique KKT system at ( w ∗A ∗ λ , η ) is invertible.
The implicit function theorem implies there exists an open interval ˜ O ⊆ Λ containing λ such that λ (cid:55)→ (˜ g ( λ ) , h ( λ )) , where ˜ g :
˜ O
→ R | A ∗ λ | and h : ˜ O
→ R | A ∗ λ | , is a unique, continuously differentiable function satisfying (˜ g ( λ ) , h ( λ ))
= ( w ∗A ∗ λ , η ) and (˜ g ( λ (cid:48) ) , h ( λ (cid:48) )
) is a zero of the unique
KKT system for all λ (cid:48) ∈
˜ O .
That is, (˜ g, h ) is locally the primal-dual solution function for the reduced problem (Eq.
; we will show that an extension of ˜ g to R d locally provides the min-norm solution function for the full group Lasso problem.
Deﬁne the extension of ˜ g to be g : ˜ O
→ R d such that g ( λ (cid:48) )
A ∗ λ = ˜
g
( λ (cid:48) ) and g ( λ (cid:48) )
B\A ∗ λ = 0 .
For i ∈ I λ , ﬁrst-order optimality says
[Equation]
Continuity of g guarantees there exists O ⊆
˜ O on which
[Equation]
This implies g I λ satisﬁes FO conditions on O .
By assumption, A ∗ λ is constant on Λ and thus on O .
Applying Lemma 3, we deduce that w ∗A ∗ λ is given by the unique solution to the KKT system on O .
Thus, g A ∗ λ also satisﬁes FO conditions on O and ˆ y ( λ (cid:48) )
=
Xg ( λ (cid:48) ) provides the unique model ﬁt on O .
It remains only to check g T ∗ λ ( λ (cid:48) ) .
Suppose there exists b i ∈ T ∗ λ such that
[Equation]
Since Xg ( λ (cid:48) )
=
Xw ∗ ( λ (cid:48) ) by uniqueness of the model
ﬁt, it must be that
[Equation]
which is a contradiction.
So FO conditions are also satisﬁed for all blocks in T ∗ λ .
Putting these results together, we have shown g satisﬁes the (sufﬁcient) FO conditions for the group lasso on O .
We conclude that w ∗ ( λ ) is continuously differentiable on O and corresponds with the min-norm solution for the reduced-block problem.
B.1. Continuity of the Min-norm Path: Proofs
Lemma 10 The minimum-norm solution w ∗ ( λ ) is bounded for all λ ≥ 0 .
Proof For λ = 0 , the min-norm solution reduces to the min-norm least-squares solution, which is bounded.
For λ > 0 , we show boundedness directly.
Optimality of w ∗ implies
[Equation]
from which we deduce the residual vector r ( λ ) is bounded for any choice of λ .
Thus, the block correlation vectors satisfy,
[Equation]
The right-hand quantity is independent of λ ; therefore, v ( λ ) is also bounded for all λ .
By Lemma 2, w ∗A ∗ λ is given by
[Equation]
where the right-hand side is bounded.
This completes the proof.
Lemma 7 The transition space H is discrete.
Proof To see that that H is discrete, deﬁne
[Equation]
so that H =
(cid:83) i ∈B H i .
The (Lebesgue) measure of H i is zero because it is the boundary of a onedimensional set and thus is discrete.
By properties of the Lebesgue measure (denoted µ ),
[Equation]
since B is ﬁnite.
We deduce that H is discrete.
Proposition 11 Group general position (Assumption 6) does not imply the columns of X are in general position.
Similarly, general position of the columns of X does not imply group general position.
Proof Consider the simple case where we have two groups: b 1 = { 1 } and b 2 = { 2 , . . .
, d } .
Group general position is violated if there exists a unit vector z b 2 such that
[Equation] where B n − 1 = (cid:8) z ∈ R d
− 1 : (cid:107) z (cid:107) ≤ 1 (cid:9) .
In contrast, general position is violated if
[Equation]
{+Taking X b 
+}
=
I , it is trivial to see that group general position can hold when general position is violated and vice-versa.
Proposition 12 Suppose Assumption 6 holds and λ > 0 .
Then the group Lasso solution is unique.
Proof Suppose the group Lasso solution is not unique.
Then, Proposition 1 implies
[Equation]
is non-empty.
That is, there exist s b i ∈ { +1 , − 1 } and α b i ≥ 0 such that
[Equation]
Taking inner-products on both sides with the residual r ,
[Equation]
Thus, we deduce that
[Equation]
where (cid:80) b i ∈E λ \ j β b i = 1 .
Now, suppose that |E λ | > n + 1 .
Then, { X b i v b i : b i ∈ E λ \ j } are linearly dependent and, by eliminating dependent vectors
X b i v b i , we can repeat the above proof with a subset E (cid:48) of at most n + 1 blocks.
Noting (cid:107) v b i (cid:107) 2 = λ for each b i ∈ E λ and rescaling both sides of Eq. (9) by λ implies the existence of unit vectors z b i which contradict Assumption 6.
This completes the proof.
Theorem 8 Suppose group general position holds.
Then the unique group lasso solution path is continuous.
Proof Let ¯ λ ∈ H .
By Lemma 7, H is discrete so that there exists some open interval O = (cid:0) ¯ λ, λ 0 (cid:1) containing no points of H .
Let g be the continuous solution function on O , which exists by Proposition 5 and recall that A ∗ λ is constant on this set.
Let λ k ↓ ¯ λ
so that λ k ∈ O for all sufﬁciently large k .
By Lemma 10, the min-norm solution g ( λ k ) is bounded.
Thus, dropping to a convergent subsequence if necessary, the limit ¯ w = lim k g ( λ k ) exists.
It is trivial to argue in the same fashion for the dual problem, where h ( λ k ) =
for all k implies ¯ η = 0 is the limit.
For each b i ∈ A ∗ λ , FO conditions give
[Equation]
which, by taking limits on both sides, yields
[Equation]
, with (cid:107) u b i (cid:107) 2 = 1 .
If ¯ w b i (cid:54) = 0 , then
[Equation]
and FO conditions are satisﬁed.
On the other hand, if ¯ w b i = 0 , then taking norms shows b i ∈
E ¯ λ and FO conditions are also satisﬁed.
Thus, ¯ w A ∗ λ satisﬁes FO conditions at ¯ λ .
Similarly, taking limits for the blocks in b i ∈ I λ
∪ T ∗ λ gives
[Equation]
showing the extension ¯ w b i =
satisﬁes FO conditions for the group Lasso.
We conclude that ¯ w is a solution to the group Lasso problem.
Since the solution to the group Lasso is unique under Assumption 6, it must be that ¯ w is the unique solution at ¯ λ . Since this argument holds for every limit point of g ( λ k ) , we have shown that the limit from the right exists and W ∗ is continuous from the right.
Arguing identically for the left-hand limit establishes continuity of the solution function at ¯ λ .