Paper: Sparsistency for inverse optimal transport (ICLR-2024)

Reviewer comment (Theory):
I find that the article doesn't put in enough effort to explain the practical implications of the various theoretical results, which remain somewhat abstract. It's quite challenging to understand how these results can be practically applied.

Author response:
We have added an ``intuitions'' section to explain the relation between the certificate and optimality conditions. There is also now an "interpretation" after Propositions 8 and 9.

Reviewer comment (Writing):
The concept of a pre-certificate condition is rather very abstract (unlike the case of Lasso that we can somehow interpret). It would be interesting to provide the reader with a bit more guidance to offer a small interpretation of this quantity.

Author response:
We have added an ``intuitions'' section to explain the relation between the certificate and optimality conditions. There is also now an "interpretation" after Propositions 8 and 9.

Reviewer comment (Theory):
Proposition 8 is difficult to extract something concrete from, aside from the qualitative interpretation that "for small regularization, under somewhat abstract assumptions of certificates, and with a sufficiently large number of samples, solving the invOT problem yields a good solution." To make these results more applicable in practice, it would be helpful to either provide experiments or guarantees that demonstrate how to understand these pre-certificate assumptions or ensure that the level of regularization is appropriate. Currently, not much intuition is given, and the practical use of these results does not appear straightforward.

Author response:
Proposition 8 is the main technical result required for proving Theorem 3 which tells us that sparsistency is possible under the abstract assumption on the certificate. Theoretically proving whether the certificate is nondegenerate is challenging and in this paper, we analyse this condition only in the Gaussian setting (Prop 8 and 9 in the updated version): Some practical insights is that in the small epsilon regime, recoverability is equivalent to recoverability in the graphical lasso setting while for large epsilon, recoverability is approximated by the lasso setting (e.g. diagonal covariance matrix in alpha and beta will always lead to this condition being satisfied). We added comments on ``interpretation'' after Prop 8 and 9. In our opinion, the result here is inline with general intuition that small epsilon leads to more dependent couplings (and hence graphical lasso in the limit) while large epsilon leads to independent couplings, so the interaction matrix recovers has a different interpretation in each setting.

Reviewer comment (Experiments):
My main criticism concerns the experiments section. I find the experiments too limited and somewhat confusing, making it difficult to obtain meaningful information.

Author response:
The purpose of Figure 1 was to numerically illustrate the behaviour of certificates in the Gaussian setting. One of the messages of our analysis is that in the large epsilon regime, the OT coupling becomes increasingly independent and iOT approximates a Lasso problem; in the small epsilon regime, the OT coupling becomes more dependent and iOT approximates graphical Lasso which is ``harder'' for sparsistency. The identity covariance is chosen because in this case, as $\epsilon$ increases, our theory tells us that the certificate becomes nondegenerate so sparsistency will be guaranteed. We added also another figure to illustrate the setting where one attempts to recover a nonsymmetric cost function: the connection to graphical lasso is only in the case of symmetric matrices A and when A is not symmetric, the certificate in fact is not guaranteed to have a limit as epsilon goes to 0; this corroborates with numerical observations made in previous works where it has been noted that the recovery of non-symmetric costs is harder (page 23 https://arxiv.org/pdf/2002.09650.pdf and section 4.4 of https://arxiv.org/pdf/1905.03950.pdf).

Reviewer comment (Presentation):
Figure 2 is equally unenlightening. It aims to determine the influence of the number of samples on the estimation in the very simple case of a circular graph and Gaussian measures. However, the x-axis on these three figures is unclear. Is it the number of iterations? The geodesic distance? If it's the geodesic distance, it's not clear because the y-axis is a global performance measure over C, while the x-axis appears to be a local measure, making it unclear how one evolves with the other.

Author response:
We have added labels to the axis. The x-axis is $\lambda$. For each regularisation parameter $\lambda$, we solve iOT and record the number of wrongly estimated positions.

Reviewer comment (Theory):
The experimental conclusion appears to be that the estimation is "good" when the entropic regularization is sufficiently large, and the number of samples is also large. However, there is a significant gap: when $\epsilon \leq 10$, the estimation is consistently poor. It is unclear if this is a result expected by the theory regarding $\epsilon$.

Author response:
When $\epsilon$ is small, the certificate is degenerate and hence support recovery is unstable. We added a remark on this: `` in particular,
one can observe from (7) that for large $\epsilon$, the certificate is trivially non-degenerate whenever
$\Sigma_\alpha$, $\Sigma_\beta$ are diagonal."

Reviewer comment (Theory):
How practical is inverse OT? I understand that the current work considers linear cost functions. Can one, albeit without strong theoretical guarantees, learn a general (perhaps, regular) cost function from the samples drawn from the coupling?

Author response:
Thank you for your comments.

>  How practical is inverse OT? I understand that the current work considers linear cost functions. Can one, albeit without strong theoretical guarantees, learn a general (perhaps, regular) cost function from the samples drawn from the coupling?

iOT can indeed be applied to recover other types of cost functions. There are works in this direction such as https://arxiv.org/abs/2002.09650 which considers cost functions and Kantorovich potentials given by neural networks.  We have cited several of these works in the introduction, but to clarify, we also added a footnote on page 4. Note however that, in this setting it is far from trivial to give theoretical guarantees, our main goal with this work.

Reviewer comment (Writing):
The authors could really afford to improve the pedagogy of the paper, which is quite heavy in terms of notation. Theory is quite involved, requiring background references to other works such as Carlier or Galichon.

Author response:
We have added an ``intuitions'' section to explain the relation between the certificate and optimality conditions. There is also now an "interpretation" after Propositions 8 and 9. Regarding the experiments, we added an additional example for the anti-symmetric case and clarified the interpretation of the results.

Reviewer comment (Theory):
Why does Proposition 8 imply that \( z_\infty \) is nondegenerate? Why, given its definition 1 line above, is it of the form written in (PrC)?

Author response:
$z_\infty$ is actually the dual solution for $\lambda$ fixed. It corresponds, in the original submission, to $z^\lambda$ of Proposition 5. We don't claim that $z_\infty$ has the form  written in (PrC) in the original solution. What we have is that $\lim_{\lambda \to 0} z_\infty\to z_A^\ast$ and hence, non-degeneracy of $z_A^\ast$ implies that the magnitude of $z_\infty$ outside the support of $A$ is less than one (for $\lambda$ sufficiently small). 

We agree with the reviewer that this was not  clear in the original submission and we have changed this in the new submission it to make it so.

Reviewer comment (Theory):
In Proposition 9, shouldn't the limit of \( \epsilon_n \) be \( \infty \)?

Author response:
There was in fact a typo and in the $\epsilon\to \infty$ proposition in the Gaussian case. We have corrected this.

Reviewer comment (Theory):
In Theorem 3, \( \hat A \) is such that \( \hat \pi = \text{Sink}(\hat A, \epsilon) \); this has been introduced earlier but given the large number of variables defined in the paper, it may not hurt to recall it here; in addition, before, Sink was applied to \( c \), so it should be \( \text{Sink}(c_{\hat A}, \epsilon) \). In Proposition 10 and Proposition 11, the order of the arguments is reversed.

Author response:
We agree with the reviewer and we have modified the new submission accordingly.

Reviewer comment (Novelty):
Can the authors explain the name "precertificate" (what's "pre" about it? Is there a difference with what Dunner and coauthors call "Dual certificate" in https://arxiv.org/abs/1602.05205)?

Author response:
The use of ``pre'' was to make a distinction with the minimal norm certificate defined afterwards. Under non-degeneracy the two are the same but not necessarily in the degenerate case. As this distinction was hurting readability, we have changed the name and reformulated the section. Moreover, we moved to the appendix the connection with duality.

Reviewer comment (Writing):
What's "the convex dual of W(A)"? Is it the convex conjugate of a function (as in Proposition 4) or the convex dual of a convex optimization problem?

Author response:
We agree with the reviewer that this was not the best terminology and we modified it.

Reviewer comment (Writing):
Is the notation \( \langle c, A \rangle \) for a non-scalar result?

Author response:
The notation $\langle c, A\rangle$ corresponds to a function: given a set of fixed cost functions $\lbrace C_1(\cdot,\cdot),\ldots, C_t(\cdot,\cdot)\rbrace$, the notation meant a linear combination of those with coefficients given by the vector $A$, \emph{i.e.,}
\begin{align*}
    \langle c, A\rangle(x,y):= \sum_j A_j C_j(x,y)
\end{align*}
In fact, $\langle c,A\rangle(\cdot,\cdot)$ was just another notation for $c_A(\cdot,\cdot)$ that was meant to highlight the linear dependence of $c_A(\cdot,\cdot)$  on $A$. Given that it was never re-used, we decided to drop it to avoid confusion with the inner product notation.

Reviewer comment (Writing):
Below equation (9), in the \( A_n \) definition, both \( \epsilon \)'s should be \( \epsilon_n \)? Same in Proposition 11.

Author response:
Yes. This was a typo that is now corrected.

Reviewer comment (Writing):
Typographical error: "in a series of paper*s*" should be pluralized correctly.

Author response:
All the typos suggested by the reviewer were corrected.

Reviewer comment (Writing):
Typographical error: "minimial norm" should be "minimal norm."

Author response:
All the typos suggested by the reviewer were corrected.

Reviewer comment (Writing):
Typographical error: "sufficinetly$" should be corrected to "sufficiently."

Author response:
All the typos suggested by the reviewer were corrected.

Reviewer comment (Writing):
The phrase "result above implies that for provided that the number" is grammatically incorrect and unclear.

Author response:
All the typos suggested by the reviewer were corrected.

Reviewer comment (Writing):
The phrase "as exposed in Section ??" is unclear and has a placeholder instead of a section number.

Author response:
All the typos suggested by the reviewer were corrected.

Reviewer comment (Writing):
Typographical error: "Cauchy Schwartz" should be "Cauchy–Schwarz."

Author response:
All the typos suggested by the reviewer were corrected.