Paper: Compressed Online Sinkhorn (ICLR-2024)

Reviewer comment (Evaluation):
The authors do not discuss the performance of the algorithm in high-dimensional situations. Real-world data often has a high dimensionality (such as datasets of images and amino acid sequences), but the authors do not discuss cases where $d > 5$. In \textbf{section A.5.2}, for data of dimension $d$, the compression error is $O(\frac{|\log m|^{d}}{m})$, which may be too large in high-dimensional situations (i.e., assumption 4 with a large coefficient for $O(m_t^{-\zeta})$). In fact, in the experimental part, Figure 2(c), when d=5, the online Sinkhorn algorithm already has lower error than the new compressed online Sinkhorn algorithm.

Author response:
As mentioned above, we have clarified the issues with dimension when applying  QMC-based Fourier method and explained the dimension dependence of the error bound. Asymptotically, the compressed online Sinkhorn can be more efficient. It is more difficult  however to get into the asymptotic regime with the dimension $d$ is high.

Reviewer comment (Experiments):
The Algorithm 2 proposed by the author improves the speed compared to the original online Sinkhorn algorithm by using a measure compression technique to compress $u_t$ and $v_t$ from $n$ atoms to $m$. However, there is a trade-off between accuracy and speed. According to assumption 4, the smaller the value of $m$, the faster the algorithm but the larger the error. The article seems to lack a detailed discussion on this matter, such as how to choose an appropriate batch size $m_t$ when solving actual OT problems.

Author response:
We made some changes to Assumption 4, please check the new version. The choice of $m_t$ depends on the compression algorithm, such that Assumption 4 holds. The purpose of this assumption is that the compression error does not exceed the online Sinkhorn error in Algorithm 1, and we focus on the overall efficiency of the compressed algorithm. This allows for an increased error if there is sufficient reduction in computational effort to allow more iterations. This is now shown more effectively in the numerical experiments, where we ensure the error in the objective for the different algorithms match in the Figure 2.

Reviewer comment (Experiments):
The experimental sections lack the application of the algorithm on real-world data and more complex distributions.

Author response:
Thanks for this valid point. This work is a first attempt at a compression method and the improvements are mainly in the asymptotic regime and lower dimensional setting. As future work, we plan on refining this by investigating adaptive choices of the parameters to enhance the fast transient behaviour in the error bound, making the algorithm more competitive for real-world data.

Reviewer comment (Experiments):
The experiments are done in settings of very low dimensionality.

Author response:
We have clarified the issues with dimension when applying  QMC method. Asymptotically, the compressed online Sinkhorn can be more efficient. It is difficult however to get into the asymptotic regime due to the small contraction rate ($\kappa\approx 1$) and available compression methods.

Reviewer comment (Evaluation):
For the one of greater dimension (d=5), the uncompressed method starts to look quite better.

Author response:
We have clarified the issues with dimension when applying  QMC method. Asymptotically, the compressed online Sinkhorn can be more efficient. It is difficult however to get into the asymptotic regime due to the small contraction rate ($\kappa\approx 1$) and available compression methods.

Reviewer comment (Experiments):
I wonder if what I mentioned above regarding the uncompressed method working better and better with increasing dimension is a general trend.

Author response:
We have clarified the issues with dimension when applying  QMC method. Asymptotically, the compressed online Sinkhorn can be more efficient. It is difficult however to get into the asymptotic regime due to the small contraction rate ($\kappa\approx 1$) and available compression methods.

Reviewer comment (Writing):
The constant $\kappa$ and the fact that it is at most $1$ are explained for the first time.

Author response:
A definition $\kappa = 1-\exp(-L\text{diam}\mathcal{X})$ has been added to the sketch proof of Theorem 1.

Reviewer comment (Theory):
I believe the Lipschitz constant in Lemma 4 (with the notation employed) should be $L$, or maybe the formula for $T_\beta$ is missing a $1/\epsilon$ factor in the exponent.

Author response:
This is true, we have missed out an $\epsilon$ in $T_{\mu}$. Please see the updated definition of $T_{\mu}$ before Lemma 4, thanks for pointing it out.

Reviewer comment (Theory):
The last equality in the first math display in page 13 should probably be an upper bound.

Author response:
Sorry about the confusion, it is defined in Lemma 7 that $M(x)=\int_y\exp((f^*(x)+g(y)-C(x,y))/\epsilon)\mathrm{d}\beta(y)$ and this equality is a simplification of the notation.