Paper: Energy-guided Entropic Neural Optimal Transport (ICLR-2024)

Reviewer comment (Experiments):
While promising results have been demonstrated in experiments on high-dimensional unpaired image-to-image translation, it is worth noting that this approach couples with a pretrained GAN model and conducts training in latent spaces. I hold reservations about its direct applicability to image spaces.

Author response:
**(1.1) "[...]it is worth noting that this approach couples with a pretrained GAN model and conducts training in latent spaces.**

We want to leave a comment regarding our unpaired I2I setup in the latent space of StyleGAN. We want to underline that our Energy-based approach is not a usual method for learning something in a latent space. To the best of our knowledge, a typical method operating in a latent space requires not only a decoder (a model which transforms latent embeddings to data) but also an encoder (which transforms data of interest to the latent embeddings). At the same time, our method *does not need any encoder*.

Reviewer comment (Theory):
It appears that equation 18 is akin to the maximum likelihood training of the EBM defined in equation 13. Could you offer a more intuitive explanation of why maximizing the likelihood of equation 13 is valid and how it equates to entropy-regularized optimal transport?

Author response:
**(2) Clarifications regarding Eqs. 13 and 18**

We are not sure if we catch the question of the reviewer exactly, but we make a try. At first, we propose to compare objectives (18) and (11).  In fact, the potential $f_{\theta}$ in (18) serves as a test function which compares *the second marginal* of distribution $d \pi^{f_\theta}(x, y) = d \mu_x^{f_\theta}(y) d \mathbb{P}(x)$ and $\mathbb{Q}$. Similarly, the potential $E_{\theta}$ in (11) plays the same role by comparing the generative distribution $\mu_\theta$ and GT distribution $\mu$. In other words, our Energy-based methodology for solving EOT is the combination of **three** principles. We look for a plan $\pi \in \mathcal{P}(\mathcal{X} \times \mathcal{Y})$,

* with the first marginal $\pi(x)$ (projection to $\mathcal{X}$) given by the source distribution $\mathbb{P}$. This condition is satisfied by construction (as we build our plan as $d \pi(x, y) = d \mathbb{P}(x) \cdot d \pi(y \vert x)$).
 * with the second marginal $\pi(y)$ (projection to $\mathcal{Y}$) given by the target distribution $\mathbb{Q}$. In order to satisfy this condition, we need the optimization given by Eq. (18).
 * whose conditional distributions $\pi(y \vert x)$ satisfy Eq. (13), i.e., $\pi(y \vert x) = \mu_x^f(y)$ for some potential $f$. This condition is also satisfied by construction. Note, that it is the condition which ensures that plan $\pi$ is EOT plan between its first and second marginals.

Combining principle 3.) with 1.) and 2.), our methodology results in Energy-based EOT solver. Does our comments address your question?

Reviewer comment (Theory):
Can equation 17 be optimised using alternative EBM training techniques, such as noise contrastive estimate, and score matching?

Author response:
**(3) Alternative EBM training procedures (noise contrastive estimate, score matching) for EOT**

It is the really good and interesting question. We express special gratitude to the reviewer. In short, it seems that these techniques (noise contrastive estimate - NCE and score matching - SM) are not straightforwardly applicable for EOT. The key challenge here is that we aim to model a distribution (in our case, it is the EOT plan $\pi^*$) samples from which **are not known**. This makes our considered setup completely different from a typical generative modeling problem when we **have samples** from an unknown distribution, and we want to somehow approximate the underlining distribution based on these samples. In particular, both NCS and SM require knowledge of samples from the distribution of interest. In fact, in our approach, we manage to alleviate this lack of data by utilizing special properties of EOT plan. It is still an open question whether we can somehow adapt SM-based techniques to solve EOT in a similar way. Well, if you are interested in this topic, we encourage you to think of this question. We are sure that a hypothetical SM-based EOT solver will be of interest to the ML community. Some works which are a good points for inspiration and reflections: [Daniels], [Chen] (the latter utilizes the connection between EOT and the Schrodinger bridge problem).

[Zhao] Zhao et. al., Learning Energy-Based Generative Models via Coarse-to-Fine Expanding and Sampling.

[Du] Du et. al., Improved Contrastive Divergence Training of Energy-Based Model.

[Daniels] Daniels et. al., Score-based generative neural networks for large-scale optimal transport.

[Chen] Chen et. al., Stochastic control liaisons: Richard Sinkhorn meets Gaspard Monge on a Schroedinger bridge.