Paper: Learning No-Regret Sparse Generalized Linear Models with Varying Observation(s) (ICLR-2024)

Reviewer comment (Theory):
Lemma 3, eq 27, Why is there a minus sign in the second equation? However the proof below from eq 28 is correct.

Author response:
> Lemma 3, eq 27, Why is there a minus sign in the second equation? However the proof below from eq 28 is correct.

Thank you for raising a good question. At first **we verified the correctness of Equation (27)** in the proof process of Lemma 3. A minus sign appears multiple times in (27). Here's a step-by-step reasoning for the inclusion of the minus sign:

1. The **first** term $d_{-\boldsymbol{e}_j} \Theta(\beta)$ is the directional derivative, which is taken in the negative direction of the $j$-th coordinate axis. A minus sign before $e_j$ is to reflect this opposite direction.

2. Regarding the **second** minus sign, it arises from our calculations. According to the definition of the directional derivative, we obtain
$$
\begin{align*}
&  d\_{-\boldsymbol{e}\_j} \Theta(\beta)  \\\\
& = -\nabla\_j \left[\rule{0pt}{10pt}  \ell^{\diamond}\hspace{-2pt}\left(\rule{0pt}{9pt}\beta\right)+\mu\left(t\right)\ell^{\textbf{+}}\hspace{-2pt}\left(\rule{0pt}{9pt}\beta\right)+\nu\left( s\right) \ell^{\textbf{--}}\hspace{-2pt}\left(\rule{0pt}{9pt}\beta\right)\right]-(-1)\cdot\sum\_{k=1}^d\frac{\partial P\_\eta \left(\left\|{\beta}\_j\right\|, \lambda\_k\right)}{\partial\left\|\beta\_j\right\|} \\\\
& =-\nabla\_j \left\[\rule{0pt}{10pt}  \ell^{\diamond}\hspace{-2pt}\left(\rule{0pt}{9pt}\beta\right)+\mu\left(t\right)\ell^{\textbf{+}}\hspace{-2pt}\left(\rule{0pt}{9pt}\beta\right)+\nu\left( s\right) \ell^{\textbf{--}}\hspace{-2pt}\left(\rule{0pt}{9pt}\beta\right)\right\]+\sum\_{k=1}^d\frac{\partial P\_\eta\left(\left|{\beta}\_j\right|, \lambda\_k\right)}{\partial\left|\beta\_j\right|}
\end{align*}
$$

For a local minimum, both directional derivatives must be non-negative (the rationale could be found in **[1]**), which is equivalent to (25) with $\omega_j \in[-1,1]$. Thus, our proof should be correct.

Reviewer comment (Writing):
Algo 1, line 5, what does “solve” mean? Solve the whole trajectory of $\hat \beta$?

Author response:
> Algo 1, line 5, what does “solve” mean? Solve the whole trajectory of $\hat{\beta}$? 

Yes, your understanding aligns with our intent. The referenced step of our algorithm, as well as the step in Line 15, is to **numerically compute the trajectory of $\hat{\beta}$ in one segment** based on its dynamics. Upon reaching the terminal point of current segment, we proceed to update the active set as delineated in Lines 6 through 10. Subsequently, we solve the trajectory of $\hat{\beta}$ in the next segment, and continues until we have traversed the entire $[\underline{s}, \bar{s}]\times [\underline{t}, \bar{t}]$ (or $\Lambda$). We believe that with the further aid of Appendix C.2 and the provided code documentation, it would be clear to the readers.

Reviewer comment (Writing):
Page 4 and later, I'm a bit confused about the set definitions and the elements in sets. For example, are $S_1,...,S_d$ elements of $S$, or $S = S_1 \cup ... \cup S_d$, is it consistent with the definition of $S_{\bar Z}$ which says $j\in S$?

Author response:
> Page 4 and later, I'm a bit confused about the set definitions ..., are $ S_1, \ldots, S_d $ elements of $S $, or is $S = S_1 \cup \ldots \cup S_d $ consistent with the definition of $\mathcal{S}_{\bar{{Z}}}$ which says $ j \in S$?

Thank you very much for pointing this out! We apologize for confusion caused. **Your first understanding is exactly what we intended in our paper.** Recall that we have defined $\mathcal{S}=\left\lbrace\mathcal{S}\_1, \mathcal{S}\_2, ..., \mathcal{S}\_k, \mathcal{S}\_{k+1}, ..., \mathcal{S}\_d\right\rbrace$, where $\mathcal{S}\_k \subset[m]$ and $m$ is the data dimensionality. We regard $\mathcal{S}_k$ as **an element** of the set $\mathcal{S}$, which conforms to our definition of the regularization term for GLMs (see (1) in page 3). This setting also facilitates subsequent derivations, avoiding unnecessary complexity and ambiguity. However, our definitions of $\mathcal{S}\_{\mathcal{Z}}$ and $\mathcal{S}\_{\bar{\mathcal{Z}}}$
in Section 3 were indeed *not* rigorous. Specifically, **they should be written as**

$$
\begin{align*}
&\mathcal{S}\_{\mathcal{Z}}(\beta)=\left\\{j \in \bigcup\_{k=1}^d\mathcal{S}\_k ~ |~\beta\_j=0\right\\}  \\\\
&\mathcal{S}\_{\overline{\mathcal{Z}}}(\beta)=\left\\{j \in \bigcup\_{k=1}^d\mathcal{S}\_k ~ | ~ \beta\_j \neq 0\right\\} ~ (\text{or equivalently, ~ } \mathcal{S}\_{\overline{\mathcal{Z}}}(\beta)=\complement\_{\bigcup\_{k=1}^d\mathcal{S}\_k}^{\mathcal{S}\_{\mathcal{Z}}}).
\end{align*}
$$

For consistency, we also modified the definition of $\mathcal{A}$ in the subsequent text (**the revised version has been uploaded**). The revised part is shown as follows:

> Set ${\mathcal{S}\_\*}=\bigcup\_{k=1}^d\mathcal{S}\_k$. For a parameter vector $\beta \in \mathbb{R}^m$, we use $\mathcal{S}\_{\mathcal{Z}}(\beta)=\left\\{j \in \mathcal{S}\_\* ~ |~\beta\_j=0\right\\}$ to represent the set of penalized parameters that are zero 
> and correspondingly, $\mathcal{S}\_{\overline{\mathcal{Z}}}(\beta)=\left\\{j \in \mathcal{S}\_\* ~ | ~\beta\_j\neq 0\right\\}$ demarcates the set of penalized parameters with non-zero values.  

> We define the **active set** $\mathcal{A}={\mathcal{S}\_{\overline{\mathcal{Z}}} \cup \overline{\mathcal{S}\_\*}}$ indexes the current active predictors having either unpenalized or non-zero penalized coefficients. 

Thank you again and we will continue to check symbols. Please feel free to inform us if you have any further questions.

Reviewer comment (Presentation):
"Formal statements ... in Appendix B.1", what if place the formal theorem and proof sketch in main body and the detailed proof in appendix?

Author response:
> "Formal statements ... in Appendix B.1", what if place the formal theorem and proof sketch in main body and the detailed proof in appendix?

We are grateful for the suggestion to relocate them. We have considered including the complete Theorem 1 and its proof sketch in the main body, and concluded that it would introduce the use of **additional space**. This, in turn, would make us to condense other elements like key equations or experimental tables / figures **due to the page limit** (9 pages), potentially detracting from the overall readability of the paper. Furthermore, those seeking the comprehensive details of the statement and proof can be directed to the appendix, where the proof can be presented in full without interrupting the flow of the main body.

Reviewer comment (Theory):
Should Thm 2. be called continuity or Lipschitz?

Author response:
> Should Thm 2. be called continuity or Lipschitz?

We appreciate the reviewers' insightful query regarding the nomenclature. Our Theorem 2, titled "*Continuity Guarantee*", is indeed focused on establishing the continuous dependence of the solution $\hat{\beta}$ on the variables of the inner and outer problems. However, if the theorem specifies a "Lipschitz condition", which is **a stronger statement** that *not only* does $\hat{\beta}$ vary continuously *but also* does so within a bounded rate of change with respect to changes in $s, t$ and other parameters [5]. Hence, we are inclined to maintain the current designation, assuming it accurately reflects the theorem's statement. Nonetheless, we are open to further discussion and will revise the terminology if it enhances the accuracy of Theorem 2.

Reviewer comment (Evaluation):
The proposed algorithm seems to require high computational resources, e.g. memory, making it incompetent with existing algorithms.

Author response:
Thank you for your feedback; as pointed out in Appendix C.2, our algorithm indeed uses more memory than existing batch algorithms. This property actually offers more benefits than drawbacks. It also aligns with a principle in efficient ML: "**Space-time trade-off**" (i.e., solving a problem in less time by using more storage space). In detail,

+ Using additional storage in training GLMs can initially seem costly in terms of resources. However, this cost is offset by the gains in model performance as well as prediction accuracy, which could be verified by our theory and experiments. Nowadays, **faster training and prediction times lead to quicker improvements and iterations, which is crucial in a fast-paced research or industrial environment** [3].

+ Our SAGO requires storing the entire matrix $X$ and maintaining a matrix $\widetilde{\mathcal{L}}\left(\hat{\beta},\lambda\right)$ of size $\mathcal{O}\left( |\mathcal{A}|^2\right)$, but this is a necessary choice to enable faster computation by following the optimizer's dynamics. **It should be noted that in contemporary machine learning systems, the space-time trade-off is extremely common** [4], e.g., by using additional memory, ML algorithms can pre-compute values (e.g. Gram matrix in kernel), or parallelize tasks (e.g. DataParallel in PyTorch), which dramatically reduce the required computation times.

+ In practice, we found that for some GLMs, certain key submatrices within their PDE and ODE systems **exhibit good structural properties, such as *sparsity* and *symmetry***. These properties could potentially be leveraged to **further save on memory usage** in future research. Moreover, when maintaining or updating matrices of size $\mathcal{O}\left( |\mathcal{A}|^2\right)$, we can employ ***low-rank approximation***, ***incremental update formula*** [5] or ***low-precision training*** to further save memory.

Reviewer comment (Experiments):
Dimension of the datasets selected for empirical studies are relatively large, but are low in model dimension. It is unclear whether the proposed SAGO algorithm can handle modern datasets of interest.

Author response:
Thank you for your feedback. Regarding the practicality of our algorithm and the robustness of experiments, we would like to point out

+ It's important to emphasize that GLMs assume a linear relationship between output and input through a link function, so **they are generally not used for high-dimensional, extra large datasets** (e.g., ImageNet [6]).  These 'modern' datasets contain unstructured data, such as images, audio and text. In modern machine learning, we often use overparameterized deep neural networks to capture the complex non-linear relationships in data, which **often exceeds the representation ability of classic GLMs**. This should *not* be considered a shortcoming of our algorithm. Additionally, it should be noted that the complexity of our algorithm depends on the cardinality of the active set $\lVert\beta\rVert_0$, **not the original dimensionality** $m$ of the observations applied. By discarding features with high noise or irrelevance, it works on a relatively **small set of selected variables** (i.e., $\lVert\beta\rVert\_0\ll m$), which is a *meaningful reduction* from the original large set of variables.

+ **In experiments, we used real-world datasets from various fields suitable for GLMs, not just simple synthetic datasets**. For instance, the `numerai28.6` dataset in Table 1 consists of global stock training data that has been cleaned and standardized. New market indicators may update every few minutes or seconds. We fed these real-time data into our online model to obtain timely predictions, which can lead to better returns for investors [7]. Hence, the practicality of our algorithm can be comprehensively tested. We kindly ask you to read Section 5 (page 8) and Appendix E (page 32) for more insights and in-depth discussion.

+ **Practically, we often use existing dimensionality reduction techniques to reduce the model size before feeding the (transformed) dataset into a GLM** for further analysis [8]. Directly applying an appropriate GLM to high-dimensional space typically relies on *strong prior knowledge*, such as how to set the $\mathcal{S}_k$ in (1), how to divide group-wise sparsity in $G$, etc. Well-known dimensionality reduction methods include approximating high-dimensional Hilbert spaces using random Fourier features. For this, **we have conducted some numerical simulations in Table 7 (Appendix F.5), which you can kindly refer to.**

+ Our SAGO can handle variant observations but not variant model dimensions (i.e., the number of features). Our method can efficiently deal with a wide dynamic range of $n$, as used in our experiments from $10$ to $10^7$. However, it is not as flexible when dealing with model dimensions due to the fact that **in real-world applications, changes in samples are common, while applications with changes in dimensions are relatively rare** [9]. But this would be an important direction for our future exploration (please see Appendices C.3 and C.4 for details). Given this characteristic, **this is why we used datasets with large sample sizes but relatively small model dimensions** in our experiments.

　

`Extra empirical evidence`:  ***To further reinforce empirical verification regarding the *active set* and *high-dimensional* model, we have attached some extra results on the side for your review.***
Considering the time limit of this phase, we will incorporate more detailed results in the final version of our paper.

Reviewer comment (Evaluation):
Comparing with existing methods ColdGrid, ColdHyBi, ColdByO — it would be good to present the computational time of SAGO and benchmark with the existing methods.

Author response:
**The training time comparisons are shown in Figure 4 and Figure 13**. We compared the efficiency of SAGO with ColdGrid (WarmGrid), ColdHyBi (WarmHyBi) and ColdByO (WarmByO) based on different GLMs including:

1. `Sparse Poisson Regression`  　　　 3. `Sparse Gamma Regression`

2. `Logistic Group Lasso`  　　　　　　4. `Inverse Gaussian GLM`

All experiments are conducted $50$ trials and **demonstrate that SAGO achieves significant improvements *in terms of runtime***. We have not specified the hyperparameters in figures since they are automatically adjusted by the outer-level problem (Section 5 and Appendix F.4).

Reviewer comment (Evaluation):
Comparing with existing methods ColdGrid, ColdHyBi, ColdByO — comparing the active sets, do the methods agree on which ones are active?

Author response:
We can confidently answer **Yes**. In our research, we focused solely on the transformation of existing GLMs from offline training to data-varying training without modifying the statistical properties (see, e.g., [1]) of the model itself.
We would like to emphasize that our approach does not involve any approximation to the features or weights (***Theorem 1***). i.e., when the $\\{\lambda_k\\}$ is fixed, **the updated solution (including $\mathcal{A}$) using SAGO is exactly identical to the solution of offline batch retraining**.

This consistency has been verified in the numerical results of *Tables 2, 4, 5, 6, and 7*, since any loss of the correct predictor by SAGO would result in a significant performance degradation. **We have revised our Appendix C.2 to highlight this property**. 

### Active set comparing

We followed the experimental settings and data partitions as performed in Table 2, then used a list to **record the different indices in two active sets** compared to the batch retraining after each inner loop. The recorded results were returned and then printed. Each test was run $20$ times. The results are shown in the table below.

| Dataset           | GLM Type | Regularization Strength | Returned List |
|-------------------|----------|------------------------|---------------|
| ACSIncome         | (i)      | 0.02                   | ∅             |
| ACSIncome         | (i)      | 0.05                   | ∅             |
| Buzzinsocialmedia | (i)      | 0.02                   | ∅             |
| Buzzinsocialmedia | (i)      | 0.05                   | ∅             |
| fried             | (i)      | 0.02                   | ∅             |
| fried             | (i)      | 0.05                   | ∅             |
| house_16H         | (i)      | 0.02                   | ∅             |
| house_16H         | (i)      | 0.05                   | ∅             |
| creditcard        | (ii)     | 0.02                   | ∅             |
| creditcard        | (ii)     | 0.05                   | ∅             |
| MiniBooNE         | (ii)     | 0.02                   | ∅             |
| MiniBooNE         | (ii)     | 0.05                   | ∅             |
| higgs             | (ii)     | 0.02                   | ∅             |
| higgs             | (ii)     | 0.05                   | ∅             |
| numerai28.6       | (ii)     | 0.02                   | ∅             |
| numerai28.6       | (ii)     | 0.05                   | ∅             |

The $\emptyset$ represents that a returned list `array([])` is empty, indicating that **our SAGO and the existing batch methods agree on which predictors are active.**

Reviewer comment (Writing):
Minor: above Eq. (2), it should probably be $\Sigma_{\{i|({\bf x_i}, y_i) \in X^\diamond\}}$ in the definition of $\ell^{\diamond}$. Same applies to $\ell^{-}$ and $\ell^{+}$.

Author response:
Thank you a lot for pointing this out! We apologize for the incorrect notation. To ensure the elements match, they indeed should be the pairs of $\mathbf{x}_i$ and $y_i$. **We have revised this part of the manuscript as:**

> For simplicity, we denote $\ell^{\diamond}\left(\beta\right):={\sum\_{\left\lbrace \rule{0pt}{5.8pt}i|\left\lbrace \mathbf{x}\_i,y\_i\right\rbrace \in X^{\diamond} \right\rbrace}}\ell\_i\left(\rule{0pt}{10pt}y\_i, f\left(\beta^{\top} \mathbf{x}_i\right) \right)$, $ ~\ell^{\textbf{+}}\left(\beta\right):={\sum\_{\left\lbrace \rule{0pt}{5.8pt}i|\left\lbrace \mathbf{x}\_i,y\_i\right\rbrace\in X^+ \right\rbrace}}\ell\_i\left(\rule{0pt}{10pt}y\_i, f\left(\beta^{\top} \mathbf{x}_i\right) \right)$ and similarly, $\ell^{\textbf{--}}\hspace{-2pt}\left(\beta\right)$. 

***We will continue to check for similar problem. Thank you once again on this observation.***

Reviewer comment (Theory):
In Algorithm 1, line 16-19: the thresholding condition is based on Lemma 1, which seems to only apply on $\ell_1$ penalty. However, it is claimed that Algorithm 1 applies to penalties satisfying Assumption 1. There seems to be a gap and I wonder how the thresholding condition can be generalized to a wider class of penalty functions.

Author response:
Thank you for your constructive comment! Indeed, we only mentioned the $\ell_1$ case in the main body because it represents a classical *statistical property* of Lasso estimator. Despite its simplicity, $\ell_1$ serves as an excellent heuristic familiar to many researchers in sparse regression, making it readily acceptable. For GLMs, one only needs to compute (or just estimate) the $\omega_j$ value corresponding to each $\hat{\beta}_j$ according to our Equation (25) (part of **Lemma 3 on Page 22**). Note that the (25), despite appearing complex, is actually very simple to solve since it is just a first-order linear **scalar equation** in terms of $\omega_j$. Then the relationship is quite apparent based on our Lemma 3:

$$
\omega_j \in 
\begin{cases} 
\{-1\} & \text{if } \hat{\beta}_j < 0 \\\\
[-1, 1] & \text{if } \hat{\beta}_j = 0 \\\\
\{1\} & \text{if } \hat{\beta}_j > 0 
\end{cases}
$$

 All the $\omega_j$ here could be computed in a vectorized parallel. To detect a transition from inactive to active for $\beta_j ~(j\in\overline{A})$, we can monitor the value of $\omega_j$ and add $j$ to the active set. This operation can be supported by **boundary conditions pre-set in the numerical solver**.
*We address below the revisions we have made concerning this gap*, and thank you again for pointing this out.

Reviewer comment (Theory):
It's unclear how a coefficient can become active from inactive in SAGO in Algorithm 1, i.e. E1 on line 18-19 — the ODE in Eq. (5) describes the dynamics of coefficients in the "active" set, so solving (5) doesn't seem to provide an update for the coefficients in inactive sets.

Author response:
Thank you for your constructive comment! Indeed, we only mentioned the $\ell_1$ case in the main body because it represents a classical *statistical property* of Lasso estimator. Despite its simplicity, $\ell_1$ serves as an excellent heuristic familiar to many researchers in sparse regression, making it readily acceptable. For GLMs, one only needs to compute (or just estimate) the $\omega_j$ value corresponding to each $\hat{\beta}_j$ according to our Equation (25) (part of **Lemma 3 on Page 22**). Note that the (25), despite appearing complex, is actually very simple to solve since it is just a first-order linear **scalar equation** in terms of $\omega_j$. Then the relationship is quite apparent based on our Lemma 3:

$$
\omega_j \in 
\begin{cases} 
\{-1\} & \text{if } \hat{\beta}_j < 0 \\\\
[-1, 1] & \text{if } \hat{\beta}_j = 0 \\\\
\{1\} & \text{if } \hat{\beta}_j > 0 
\end{cases}
$$

 All the $\omega_j$ here could be computed in a vectorized parallel. To detect a transition from inactive to active for $\beta_j ~(j\in\overline{A})$, we can monitor the value of $\omega_j$ and add $j$ to the active set. This operation can be supported by **boundary conditions pre-set in the numerical solver**.
*We address below the revisions we have made concerning this gap*, and thank you again for pointing this out.

Reviewer comment (Writing):
It's unclear how a coefficient can become active from inactive in SAGO in Algorithm 1, i.e. E1 on line 18-19 — when applying the thresholding condition, should one replace the $L$ in Lemma 1 by the validation loss $\mathcal L_{val}$? To make things concrete, probably it's better to repeat explicitly the thresholding condition in line 18-19.

Author response:
Thank you again for your insightful question! The choice between using $\mathcal{L}\_{val}$ or $\ell^+$ ($\ell^-$) in the thresholding condition **depends on which loop of Algorithm 1 we are currently in**. The thresholding conditions only provide a *general guideline* and does not strictly dictate the use of $\mathcal{L}\_{val}$ or $\ell^+$. Specifically, in implementation, we should use the loss on the training set when computing the gradient flow for the inner problem, and use the loss on the validation set for the outer problem. In response to this query, **we have further expanded the description in paper** w.r.t. thresholding conditions to better guide other readers.

Reviewer comment (Writing):
Minor: line 25 of Algorithm 1 should probably be arg zero.

Author response:
We have checked Algorithm 1 based on your comments but did not find any errors. We guess that you might be suggesting to set the value of the **hypergradient to zero** on line 25, similar to that in gradient descent. This is a common technique in conventional bilevel optimization but cannot be assured here. The hyperparameter space $\Lambda$ is determined by the user's input, making it theoretically challenging to guarantee a (local) optimum point, i.e., $\mathbf{0}\in\left\lbrace\nabla_\lambda\mathcal{L}_{val}\left(\lambda \right)|\lambda \in\Lambda\right\rbrace $ **does not always hold**. If you could provide further hints or descriptions, we are always open to continuing the discussion.

Reviewer comment (Theory):
How easy is it to optimally solve the PDE-ODE procedure in both inner and outer problems?

Author response:
The solving process is pretty straightforward, provided that the researcher has a foundational understanding of machine learning. **Even those who are new to this field would be able to independently solve both the inner and outer problems by following the main body and the appendix of our paper**. We have *explicitly* derived the gradient flows for the inner and outer problems based on our theoretical analysis, and these have been formulated into a *standard* separable form that is compatible with numerical ODE and PDE solvers.

The implementation of SAGO only involves two steps. The first is to choose appropriate parameterizers $\mu(t)$, $\nu\left(s\right)$, and manually determine the endpoint values of the solving interval $\underline{t}, \bar{t}, \underline{s}, \bar{s}$, as per Definition 2. This does not have a standard selection, although our experimental choices might serve as a useful reference. The second step involves deriving the concrete PDE and ODE systems for your loss function and regularization in GLMs based on Theorems 3 and 4. This will include the computation of simple derivatives such as $\nabla_{\beta} \ell^{\textbf{+}}(\hat{\beta})$, $\nabla_{\beta}P_\eta$ and $\widetilde{\mathcal{L}}$. Subsequently, one can directly employ **existing solvers** for the computation. For more details, please refer to our answer to the next question.

Reviewer comment (Theory):
Are there existing solvers that solve them optimally with provable convergence guarantees and what are their computational/sample complexities?

Author response:
That's a good question and our answer is definitely **Yes**! For example, `MATLAB` offers the `ode45` function, which is well-known for its robustness and efficiency in handling a variety of ODEs. MATLAB’s numerical resources also include PDE solvers [4] that have built-in features to alert users when certain events, such as the stopping rules (of event) outlined in our Algorithm 1, are triggered. These features are essential for ensuring that the algorithm terminates appropriately and provides optimal solutions within the defined criteria. Besides, the `deSolve` package [5] in `R` language is another powerful tool that can be employed for this purpose. Furthermore, Python’s `SciPy` library also contains integration and solving functions, like well-known `solve_ivp`, which is a flexible and powerful solver that can handle a wide range of problems with robust convergence properties. It allows for event handling, dense output, and has multiple solver options, making it suitable for our context.

Each of these solvers comes with theoretical guarantees for convergence under certain conditions, which are usually well-documented in their literature. The choice of solver may depend on the specific characteristics of the PDE/ODEs derived from our theorems. Notably, **we have a similar discussion on page 29** (Appendix C.2) of our paper. If you are interested, we warmly invite you to explore this part further!