Paper: A Semi-smooth, Self-shifting, and Singular Newton Method for Sparse Optimal Transport (ICLR-2024)

Reviewer comment (Theory):
In the main theorem 3, no clear assumption is given on the step size $\eta_k$. The authors should write clearly under which assumptions the proposed algorithm converges.

Author response:
Our assumption on the step size is described in Section 2.2. Specifically, we require that the step sizes are bounded from both below and above: $\eta_k\in[\tilde{m},\tilde{M}]$, $0<\tilde{m}<\tilde{M}<+\infty$.

Reviewer comment (Theory):
The condition on the step size, loosely given in the main text as "In fact, the only requirement is that $\eta_k$ needs to be bounded", seems too loose. If a sequence of exponentially decreasing step sizes is used, how can the proposed algorithm converge? Usually, this is why one has to resort to costly line-search techniques. The authors should comment on this.

Author response:
We apologize for the confusion on the term "bounded". A more precise statement is that we require the step sizes to be **bounded from both below and above**: $\eta_k\in[\tilde{m},\tilde{M}]$, $0<\tilde{m}<\tilde{M}<+\infty$. We have clarified this point in the revision to avoid confusions.

Reviewer comment (Experiments):
The experiments should be reshaped. Experiments with run time below a second may not be significant/useful, as for such timings, implementation matters much more than the algorithms themselves. These experiments should possibly be removed.

Author response:
Thanks for the suggestions. In fact, as can be inferred from Figures 1-3, for most first-order methods, they take much longer than one second to converge, and we truncate the iteration number for better illustration. Under the same setting, Newton-type methods are able to finish in shorter times.

We think our timing results are comparable, as we spent lots of efforts in implementing every algorithm in efficient C++ code, and tried our best to reduce the overhead caused by programming language.

For the illustration of results, since we have considered many existing algorithms, it would be visually undesirable if we put all curves in the same plot. Instead, we categorize the algorithms into three groups based on their characteristics, and then compare their performance with the S5N algorithm.

For the BCD algorithm, it is normal to see a very long 'flat' stage, as this is also observed in other articles such as Figures 1 and 2 in [2]. It will converge after a sufficiently large number of iterations.

[1] Lin, H., Mairal, J. and Harchaoui, Z., 2015. A universal catalyst for first-order optimization. Advances in neural information processing systems, 28.

[2] Pasechnyuk, D.A., Persiianov, M., Dvurechensky, P. and Gasnikov, A., 2023. Algorithms for euclidean regularised Optimal Transport. arXiv preprint arXiv:2307.00321.

Reviewer comment (Presentation):
For clarity, around 5 algorithms should be selected, and all the benchmark algorithms should be displayed on a single graph, with multiple values of the regularization parameter $\gamma$, similar to Figure 1 of Lin et al. (2015).

Author response:
Thanks for the suggestions. In fact, as can be inferred from Figures 1-3, for most first-order methods, they take much longer than one second to converge, and we truncate the iteration number for better illustration. Under the same setting, Newton-type methods are able to finish in shorter times.

We think our timing results are comparable, as we spent lots of efforts in implementing every algorithm in efficient C++ code, and tried our best to reduce the overhead caused by programming language.

For the illustration of results, since we have considered many existing algorithms, it would be visually undesirable if we put all curves in the same plot. Instead, we categorize the algorithms into three groups based on their characteristics, and then compare their performance with the S5N algorithm.

For the BCD algorithm, it is normal to see a very long 'flat' stage, as this is also observed in other articles such as Figures 1 and 2 in [2]. It will converge after a sufficiently large number of iterations.

[1] Lin, H., Mairal, J. and Harchaoui, Z., 2015. A universal catalyst for first-order optimization. Advances in neural information processing systems, 28.

[2] Pasechnyuk, D.A., Persiianov, M., Dvurechensky, P. and Gasnikov, A., 2023. Algorithms for euclidean regularised Optimal Transport. arXiv preprint arXiv:2307.00321.

Reviewer comment (Theory):
The block coordinate descent algorithm does not seem to converge. The authors should comment on this.

Author response:
Thanks for the suggestions. In fact, as can be inferred from Figures 1-3, for most first-order methods, they take much longer than one second to converge, and we truncate the iteration number for better illustration. Under the same setting, Newton-type methods are able to finish in shorter times.

We think our timing results are comparable, as we spent lots of efforts in implementing every algorithm in efficient C++ code, and tried our best to reduce the overhead caused by programming language.

For the illustration of results, since we have considered many existing algorithms, it would be visually undesirable if we put all curves in the same plot. Instead, we categorize the algorithms into three groups based on their characteristics, and then compare their performance with the S5N algorithm.

For the BCD algorithm, it is normal to see a very long 'flat' stage, as this is also observed in other articles such as Figures 1 and 2 in [2]. It will converge after a sufficiently large number of iterations.

[1] Lin, H., Mairal, J. and Harchaoui, Z., 2015. A universal catalyst for first-order optimization. Advances in neural information processing systems, 28.

[2] Pasechnyuk, D.A., Persiianov, M., Dvurechensky, P. and Gasnikov, A., 2023. Algorithms for euclidean regularised Optimal Transport. arXiv preprint arXiv:2307.00321.

Dear reviewer,

Thanks for the suggestions. We have removed small-scale experiments ($n=256$) in the article, and added larger ones ($n=4096$) in Appendix D.5 (Figures 9-14). The medium-sized experiments ($n=512$) are kept, since they reflect some common practices in gradient-based learning (taking mini-batches of size around 512). They may be useful for users who want to estimate how fast each algorithm can be under a typical mini-batch computation.

The block coordinate descent method in [2] is labelled as "Sinkhorn", since the Sinkhorn algorithm for entropic regularized OT is by nature a block coordinate descent algorithm.

We hope the explanations above have properly addressed the previous concerns, and we are open to further discussions.

Reviewer comment (Experiments):
Authors use grid-search to find the best step size on each step, which can be time-consuming in practice.

Author response:
Thanks for the comments. We shall point out that unlike classical line search methods, the choice of the step size in S5N is **optional**, since the theory only requires that the step sizes are bounded from both below and above. Therefore, users are allowed to set the number of candidates, and can keep it small. Also, Algorithm 2 is **not** a full grid search. It immediately stops when function value decreases, and it typically only takes one or two steps.

Reviewer comment (Theory):
Proposed algorithm converges globally, but it is not clear, what is the speed of global convergence.

Author response:
Thanks for raising this question. To the best of our knowledge, existing global convergence results for semi-smooth Newton methods are almost all asymptotic. On the other hand, although first-order methods have non-asymptotic rates, the empirical results shown in Figures 1-3 suggest that they are much slower than Newton-type methods. We agree that the global convergence rate of semi-smooth Newton is indeed a topic that deserves future research.

Reviewer comment (Writing):
Lack of definitions of some important concepts. Maybe it is clear for those, who are very familiar with this topic, but I think, introduction of such definitions may improve the overall look of papers (see questions).

Author response:
Thanks for the suggestions. We have added the relevant definitions in Appendix A in the revision.

Reviewer comment (Presentation):
Please, increase the font size of text in the figures.

Author response:
Thanks for all these helpful suggestions. We have updated the definitions related to local Lipschitz continuity and upper semi-continuity in Appendix A, and have enlarged the font size of the text in the figures.

Reviewer comment (Writing):
Please, provide the definitions of local Lipschitz continuity and upper semi-continuity.

Author response:
Thanks for all these helpful suggestions. We have updated the definitions related to local Lipschitz continuity and upper semi-continuity in Appendix A, and have enlarged the font size of the text in the figures.

Reviewer comment (Novelty):
The results in Sec. 2.4 seems not new. They seem to be standard results of semi-smooth Newton. Thus, the author should cite some references.

Author response:
Thanks for the comments. While the expressions of the main conclusions in Section 2.4 appear to be standard, we have proved them for our new algorithm, S5N, under completely new settings (non-smooth gradient **and** singular Hessian). Therefore, the proof is specific to S5N, which is not a trivial consequence of existing results.

We have added references to standard results, and have made our contributions clearer in the revision.

Reviewer comment (Theory):
It seems that $f(x)$ in Eq. (4) is not a smooth function because $()_+$ operator is not smooth. Thus, Proposition 1 can not hold since $\nabla f(\alpha, \beta)$ does not exist.

Author response:
We would like to point out that $f(x)$ in Eq. (4) in fact involves $h(x)=[(x)\_+]^2$, not $(x)\_+$. It can be verified that the former **is differentiable**, with the derivative $h'(x)=2\cdot(x)_+$. Therefore, $\nabla f(\alpha,\beta)$ in Proposition 1 **indeed exists**, and Assumption 1 holds for the problem we consider.

We humbly hope the reviewer could re-evaluate our main results and raise the score when appropriate, if this misunderstanding is resolved.

Reviewer comment (Theory):
This paper claims that S5N is used to solve problems that have possibly \emph{non-differentiable} gradients and non-isolated solutions. However, the convergence analysis requires Assumption 1 which supposes that $f(x)$ is differentiable.

Author response:
We would like to point out that $f(x)$ in Eq. (4) in fact involves $h(x)=[(x)\_+]^2$, not $(x)\_+$. It can be verified that the former **is differentiable**, with the derivative $h'(x)=2\cdot(x)_+$. Therefore, $\nabla f(\alpha,\beta)$ in Proposition 1 **indeed exists**, and Assumption 1 holds for the problem we consider.

We humbly hope the reviewer could re-evaluate our main results and raise the score when appropriate, if this misunderstanding is resolved.

Reviewer comment (Novelty):
The presented results for general problems and S5N are very similar to those in papers [1] and [2], where gradient regularization is employed with some adaptive procedures. Across all three papers, the global rates are asymptotical, and the local rates are quadratic (in [2], they are even cubic). Moreover, these rates are relatively weak compared to modern optimization methods with non-asymptotical global convergence rates, such as first-order methods, which offer guaranteed non-asymptotical global rates with a much cheaper computational cost per iteration. From a theoretical perspective, the first part of the paper does not seem to contribute significantly compared to the existing literature. Clarification is requested if there is any substantial improvement or theoretical challenge over the previous papers that was overlooked.

Author response:
Thanks for the comments. Regarding the two articles mentioned above, we would like to make the following clarifications:

1. [1] introduced the ASSN algorithm, which only applies to fix-point problems of the form $T(z)=z$, where $T=(1-\alpha)I+\alpha R$, and $R$ is a nonexpansive operator. This imposes a restriction on the problem to solve, whereas in contrast, S5N has substantially broader use.
2. In addition, the local convergence result of [1] relies on the BD-regular condition, which essentially does not support singular problems. We have reflected this point in Table 1.
3. Also shown in Table 1, [2] assumes that the objective function is twice continuously differentiable and the Hessian is Lipschitz continuous. These are quite strong assumptions, and that is why it can achieve local cubic convergence.

In this sense, neither [1] nor [2] really applies to the general setting of non-smooth gradient and singular Hessian. In constract, we have obtained both global and local convergence properties of S5N under significantly weaker assumptions, and we think this is a major theoretical contribution that differs from existing literature.

As for first-order methods, although they have non-asymptotic global convergence rates, they are mostly sub-linear, thus resulting in slower convergence speed as evidenced in Figures 1-3.

More importantly, the per-iteration cost of first-order methods is not necessarily smaller than that of Newton-type methods for sparse OT problems. For example, block coordinate descent (BCD) needs to sort length-$n$ vectors $n$ times, with a total cost of $\mathcal{O}(n^2\log n)$. Newton-type methods constructs the $\sigma$ sparse matrix in Proposition 1 in $\mathcal{O}(n^2)$ time, and solves the linear equation using conjugate gradient (CG) in $\mathcal{O}(nnz\cdot\log(1/\varepsilon))$ time, where $nnz$ is the number of nonzero elements in $\sigma$, and $\varepsilon$ is the tolerance. Their total per-iteration cost can be substantially smaller than $\mathcal{O}(n^2\log n)$.

Reviewer comment (Evaluation):
In the numerical experiments, although a large number of competitors' methods are included, the presentation and comparison appear unfair. Comparing the iteration number may be biased when evaluating first-order and second-order methods. A more appropriate metric would be gradient computations. Additionally, including theoretical per-iteration computational complexities could provide better insight into the comparison. Further questions have been listed in the next section to clarify the performance of the methods.

Author response:
To compare different algorithms, we not only consider the iteration number, but also the run time as a metric for the overall performance. We have implemented every algorithm in efficient C++ to minimize the overhead and to make the run times comparable. Also, the theoretical complexity analysis from our response above also indicates that second-order methods may even have a smaller per-iteration cost than some first-order methods in the context of sparse OT.