Paper: Optimization over Sparse Restricted Convex Sets via Two Steps Projection (ICLR-2024)

Reviewer comment (Theory):
At present there is no proof outline, which makes it difficult to understand the novelty in the proof technique compared to the literature. I understand there is a space constraint, but given that the main contributions of the paper are theoretical, this is relevant.

Author response:
**Our response:** Thank you for your suggestion, we have added a proof outline for Theorem 1 in the main paper, and additional explanations on the key parts of our proofs at the beginning of Appendix E.1 and E.3 (from the new revision).

Reviewer comment (Experiments):
In the experiments, there is currently no comparison with existing methods. I think this would have been nice to illustrate on synthetic data, and would also provide an empirical validation of the theoretical results.

Author response:
**Our response:** In our experiment from the main paper (Section 5) about the portfolio optimization problem, we compared our algorithm with a naive algorithm: a projected gradient descent with only $\Gamma$ as constraint, followed by a hard-thresholding at the last iteration. In the new revision, we also added an additional baseline which is IHT followed, after its last iteration, by Euclidean projection onto $\Gamma$.  
Indeed, with the constraint $\Gamma$ from this experiment, there is, up to our knowledge, no closed-form for the Euclidean projection (EP) onto our mixed constraint, which is why there are no other baselines to compare our algorithm with. Regarding more synthetic experiments to illustrate our theoretical results, we compared the TSP and EP in our section F.2 which also includes some discussions on our theoretical results and the role of $\rho$ (we have updated that section to match our new Assumption 3 in the new revision), and following your suggestion, we have added a new section in Appendix F.4 with more synthetic experiments to specifically discuss our novel bounds and their dependence on $\rho$.

Reviewer comment (Writing):
I think the statement of assumption 3 is currently written in the form of a definition.

Author response:
**Our response:** Thank you, we have rewritten all assumptions in an assumption form.

Reviewer comment (Writing):
In Algorithm 2, I am a bit confused about the notation $\mathcal{S}$. Shouldn't the index set $\mathcal{S}_t$ be a subset of $[n]$? Also in the summation index, it should be $i_t$ instead of $i$?

Author response:
**Our response:**  Thank you, you are right, we have replaced $\mathcal{S}$ by $[n]$ and fixed the index $i$ into $i\_t$ in the new revision.

Reviewer comment (Theory):
The convergence results pertain to the objective value, but can something be said for the convergence of the iterates to the global minimum? (provided there is a unique global minimum of course)

Author response:
**Our response:**  Yes, actually, convergence in terms of $\\|\boldsymbol{w} - \bar{\boldsymbol{w}} \\|$, with $\bar{\boldsymbol{w}} = \arg\min\_{\boldsymbol{v} \in \mathbb{R}^d ~ \text{s.t.} ~ \\| \boldsymbol{v} \\|\_0 \leq \bar{k}} R(\boldsymbol{v})$, in our extra constraints setting, can be immediately obtained from the result without extra constraint, for an even simpler algorithm which just projects onto $\Gamma$ at the last step. Indeed, suppose one runs vanilla IHT (or any variant such as the stochastic or zeroth-order one), without the extra projection step, to obtain a bound of the form $\\|\boldsymbol{\hat{w}\_T} - \bar{\boldsymbol{w}} \\|^2 \leq A\_T + B$ for some $A\_T$ and $B$, with $\boldsymbol{\hat{w}}\_T$ the output ($A\_T$ is usually a decreasing term, and $B$ a system error (in red in Table 1)). Then, one can simply consider $\boldsymbol{\hat{z}\_T} = \Pi\_{\Gamma}(\boldsymbol{\hat{w}\_T})$, i.e. the projection of the output onto $\Gamma$. Since projection onto a convex set is non-expansive, one will then have $\\|\boldsymbol{\hat{z}\_T} - \bar{\boldsymbol{w}} \\|^2 \leq \\|\boldsymbol{\hat{w}\_T} - \bar{\boldsymbol{w}} \\|^2 \leq A\_T + B$, and therefore the bound will remain unchanged. Note that this cannot be improved in general, since if it could be, one would take $\Gamma = \mathbb{R}^d$ and one would obtain better bounds even in the case without extra constraints $\Gamma$. This is why we did not consider convergence in terms of $\\|\boldsymbol{w} - \boldsymbol{\bar{w}} \\|$ in our paper: such result is somehow trivial as it does not require any analysis.  As an intuitive explanation, we could say that adding an extra constraint will restrict the space even further, therefore it is easy to see that iterates will only get closer to $\bar{\boldsymbol{w}}$: however, what is more interesting and needs to rethink the whole proof of convergence is the impact that such extra constraints have on the risk $R$: this is what we tackle in our paper.

Reviewer comment (Presentation):
Interpretation of $\rho$ in the main result: We are generally interested in the achievable result of $R(w_t)$ and the inequality presented in Theorem 1 seems to be a more intermediate result where the role of parameter $\rho$ could be unclear to the readers. For readability, the reviewer suggests that Remarks 3 and 4 should be summarized as a main theorem, then Theorem 1 is presented as a technical result for proving them.

Author response:
**Our response:**  Thank you for your suggestion. We agree that the result in terms of $R(\boldsymbol{w}\_t)$ is indeed clearer: we have rewritten Theorem 1 so as to display the result in $R(\boldsymbol{w}\_t)$ as per Remark 3 and 4 in the new revision.

Reviewer comment (Theory):
As the presented theorem requires a sparsity constraint k that grows unbounded with respect to the condition number, would it be possible to modify the analysis or the algorithm to extend the results for any large constant k?

Author response:
**Our response:** In general, it would not be possible to modify the analysis to extend the result for any large constant $k$, since it is known that $k$ must be in $O(\kappa^2 \bar{k})$, where $\kappa$ is the condition number, and such dependence is not improvable in the analysis of IHT (as described in [1]). This fundamental bottleneck is essentially due to the NP-hardness of such a non-convex optimization problem.