C ONNECTING CONVEX ENERGY - BASED INFERENCE AND OPTIMAL TRANSPORT FOR DOMAIN ADAPTATION
A BSTRACT
The connection of optimal transport and neural networks ﬁnds a rich application in machine learning problems.
In this paper, we propose a simple algorithm for the mutual improvement of optimal transport and energy-based models for the semisupervised domain adaptation.
Having target and source domain samples we use convex energy-based inference to create a new domain that is class-wise cyclical monotone to the target domain and its samples contains features from the source domain examples.
Mapping from target to such domain can be solved by optimal transport much more successfully.
We study the performance of our approach by benchmarking it on a range of optimal transport methods and showed that in our settings optimal transport can achieve much higher results
I NTRODUCTION
Optimal transport provides a theoretical framework to solve mass moving problems for continuous and discrete distributions (Villani, 2008; Peyr ´ e & Cuturi, 2019) and it seems to be a natural solution to the domain adaptation problem (Courty et al., 2015).
But in practice, computational optimal transport techniques are very sensitive to regularization terms (Courty et al., 2015) and without special scaling often cannot achieve acceptable results in a high dimensional space.
Furthermore, optimal transport is a cyclical monotone structured mapping, which makes it not applicable to many empirical distributions.
When we solve the domain adaptation problem with optimal transport, we trying to ﬁnd a map that can make the target distribution closer to the source distribution.
In our approach, we do not use the source distribution during domain adaptation.
The core of our method is the idea of creating an ”easy” distribution that is class-wise cyclical monotone to the target distribution and still contains features of the real source dataset.
Having a such distribution that is cyclical monotone to target one we can apply optimal transport for domain adaptation much more efﬁciently.
According to Rockfellar (Rockafellar, 1966) and Breirer’s theorems (Theorem 1.22 (Santambrogio, 2015)), we know that the gradient of a convex function is cyclical monotone and solves a Monge problem (Villani, 2008).
Owing to this theoretical insight we connect optimal transport and input convex energy-based model to build an efﬁcient domain adaptation framework.
Our method is based on Input Convex Neural Networks (ICNNs)
(Amos et al., 2017), which are energy-based models that were used for multi-label classiﬁcation, structured predictions, and reinforcement learning.
Recently, there has been a strong push to further incorporate ICNNs in optimal transport problems (Villani, 2008; Peyr´e & Cuturi, 2019; Taghvaei & Jalali, 2019; Makkuva et al., 2020).
Further development of this approach enabled the construction of the non-minimax
Wasserstein-2 generative framework (Korotin et al., 2019).
ICNNs were also used for WassersteinBarycenters estimation (Fan et al., 2020; Korotin et al., 2021).
These approaches explored ICNNs as parameterized convex potentials for optimal transport but did not explore their abilities as energybased models.
Our contribution is two-fold because we improve both optimal transport and energy-based learning method for semi-supervised domain adaptation problems.
Usually, the energy-based classiﬁers are not used in domain adaptation because have signiﬁcant challenges in inference.
On the contrary, we show that convex energy-based inference with a connection to optimal transport can achieve greater results and improve optimal transport performance in comparison to standard settings.
B ACKGROUND AND R ELATED W ORK 2. O PTIMAL T RANSPORT
Optimal Transport aims at ﬁnding a solution to transfer mass from one distribution to another with the least effort.
Monge’s problem was the ﬁrst example of the optimal transport problem (Villani, 2008) and can be formally expressed as follows:
[Equation]
The Monge’s formulation of optimal transport aims at ﬁnding a mapping T : Ω µ → Ω ν of the two probability measures µ
and ν and a cost function c : Ω × Ω → [0 , + ∞ ] , where T # µ s = ν t represents the mass preserving push forward operator.
In Monge’s formulation, T cannot split the mass from a single point.
The problem is that such constraints on the mapping T may not even exist and the solution for that mapping, respectively.
To solve this problem, Kantorovitch proposed a relaxation (Villani, 2008).
Instead of obtaining a mapping, the goal is to seek a joint distribution over the source and the target that determines how the mass is allocated.
For a given symmetric cost function c : Ω × Ω → [0 , + ∞ ] , the primal
Kantorovitch formulation can be expressed as the following problem:
[Equation]
The primal Kantorovitch formulation has linear objective and linear constraints (Villani, 2008).
In this notation, we look for the joint distribution γ with µ s and µ t as marginals that minimize the expected transportation cost.
If the independent distribution γ ( x , y ) = µ ( x ) ν ( y ) respects the constraints, linear program is convex and always has a solution for a semi lower continuous c : Π ( µ s , [Equation]
[Equation]
The primal Kantorovitch formulation can be also presented in dual form as stated by the RockafellarFenchel theorem (Villani, 2008):
[Equation]
After ﬁnding a solution to the transport problem, OT provides a measure of similarity between the two distributions in the form of the optimal transport rate.
This similarity is also called Wasserstein distance (Villani, 2008):
[Equation]
where c ( x , y ) = (cid:107) x − y (cid:107) p and p > 1 .
The Wasserstein distance encodes the geometry of the space through the optimization problem and can be used on any distribution of mass.
2. E NERGY -B ASED L EARNING
Energy-based learning provides a uniﬁed framework for many probabilistic and non-probabilistic approaches, particularly for non-probabilistic training of graphical models including discriminative and generative approaches, as well as conditional random ﬁelds, graph-transformer networks, maximum margin Markov networks, and several manifold learning methods (LeCun et al., 2006).
In energy-based settings for some given ﬁxed x and possibly some ﬁxed elements of y we can perform inference by:
[Equation]
Energy-based learning approaches can be considered as an alternative to probabilistic estimation for prediction, classiﬁcation, or decision-making tasks.
The energy-based representation must capture both, the discriminative interactions between x and y , and also allow for efﬁcient combinatorial optimization over y .
2. D OMAIN A DAPTATION
Domain adaptation generalizes a learner across different domains of different distributions.
The importance of the divergence between the data probability distribution functions of the different domains was theoretically researched in (Ben-David et al., 2010a;b; Germain et al., 2013).
This works proposed a primary way of solving the domain adaptation problem by transforming data to make different domain distributions “closer”.
Globally, we can divide domain adaptation problems into three categories: (i) unsupervised domain adaptation, when the set of labels in the target domain is unavailable; (ii) semi-supervised domain adaptation, when we have a “small” set of labeled target examples; (iii) supervised domain adaptation, when labels are available for all considered target domain examples.
In our paper, we research an semi-supervised settings when we have source domain samples X s = { x si } N s i =1 with labels
Y s = { y si } N s i =1 and target domain samples X t
= { x ti } N t i =1 with small number of labels Y t =
{ y ti } N t
i =1 .
A PPROACH 3. E ASY DOMAIN
Current domain adaptation methods estimate both target and source distributions and ﬁnd a function that can make these distributions closer and more similar.
In our approach instead of making target distribution closer to the source, we try to make it closer to the distribution which we call “easy”.
It was shown that even after the ﬁrst epoch, there exist samples that can be easily classiﬁed (Arpit et al., 2017).
Our idea is that by using gradient descent in the classiﬁer input space, we can ﬁnd a transformation over the target distribution, that makes it “easy” for the classiﬁer.
This procedure is similar to adversarial attacks (Szegedy et al., 2014).
We can say that in our case we apply targeted adversarial attack but using the true class as an aim for the targeted attack.
As the result, we will have a target samples that is similar to the source domain samples with respect to the inductive biases of the classiﬁer.
A problem can arise when we try to ﬁnd an optimal transport map from target to the “easy” domain.
The resulting “easy” samples can be less similar to the target than the source.
In settings with standard feed-forward neural architectures, we have no control over the transformations from target to the “easy” samples and can’t make them monotone.
To have control over these transformations, we turn to ICNNs energy-based classiﬁer, while for this model, “easy” samples generation can be a convex optimization problem.
3. I NVERSED CONVEX ENERGY - BASED INFERENCE
Here is our approach in more detail.
First of all, we build a model f
θ that is Partly Input Convex
Neural Network (PICNNs) (Amos et al., 2017) over x instead of y (Amos et al., 2017), it’s make possible to apply convex inference over x .
Secondary we train our model f
θ
on the source domain ( X s , Y s ) in setup equal to Structured Prediction Energy Networks (SPEN)
(Belanger & McCallum, 2016), but instead of multi-label classiﬁcation, we simply perform multi-class classiﬁcation.
In the next step we apply the core idea of our approach, we use a model trained on ( X s , Y s ) , to create an “easy” domain which is cyclical monotone to the target domain ( X t , Y t ) and contain features from the source domain examples.
To make it, we simply input x ti samples to the model f θ and apply inference over it with the ﬁxed y ti , in general it can be presented as follows:
[Equation]
As stated before, Rockafellar’s Theorem(Rockafellar, 1966) says that every cyclical monotone mapping g is contained in a sub-gradient of some convex function f : X → R .
Furthermore, according to Brenier’s Theorem, these gradients uniquely solve the Monge problem:
Brenier’s Theorem, Theorem 1.22 of (Santambrogio, 2015)).
Let µ , ν be probability measures with a ﬁnite second moment, and assume µ has a Lebesgue density p X .
Then there exists a convex potential
G such that the gradient map g := ∇ G (deﬁned up to a null set) uniquely solves the Monge problem eq.
(1) with the quadratic cost function c ( x, y )
= (cid:107) x − y (cid:107) 2 .
We used the gradient descent method to inference in (7) and while our f θ is input convex over the x , applying this procedure over the target samples we can collect a new dataset ( X e , Y e ) where X e samples are class-wise cyclical monotone to X t .
And according to the theorems presented before, there exists optimal transport that can solve transportation from X t to X e inside each class.
The procedure of solving the semi-supervised domain adaptation problem by our method is presented in
Algorithm 1.
: Input network
Train EBM f θ on ( X s , Y s ) . for x t , y t in ( X t , Y t ) do [Equation]
y
[Equation]
y
[Equation]
Append x e and y e to “easy” dataset ( X e , Y e ) end for
Use OT to ﬁnd a map between t and
E XPERIMENTAL EVALUATION
We tested our model on MNIST (LeCun & Cortes, 2010) and USPS datasets (Hull, 1994).
In both experiments, we train PICNNs with a one hidden layer size of 100, in the SPEN settings using
SGD (Ruder, 2016) optimizer with a learning rate equal to 1e-3 and momentum equal to 0.9.
[Table]
After training, we obtain an MNIST classiﬁer with 95 .
70% test set accuracy and USPS trained classiﬁer with an accuracy equal to 90 .
The accuracies are lower compared to the standard machine learning classiﬁers because energy-based learning has signiﬁcant challenges in inference.
Then when we have a trained model, we can collect “easy” domain examples.
For example: for the network trained on the MNIST dataset our aim is to improve its accuracy on the USPS dataset, so the “easy” dataset for the MNIST model will be the USPS dataset transformed by the input gradients of the network trained on MNIST.
Collected “easy” examples can contain very strong source domain features.
The accuracy of the model can be higher on “easy” examples than on the source.
For example, the accuracy of the model trained on MNIST is 95 .
70% and the accuracy of the same model on USPS samples transformed to the “easy” USPS is equal to 96% .
For the model trained on USPS, the accuracy on the MNIST samples transformed to “easy” MNIST is even equal to 99% .
When we have such datasets, it remains only to ﬁt the optimal transport on this data and then transport samples without a label to this distribution to solve domain adaptation.
Based on the POT library (Flamary & Courty, 2017) we tested the different variations of optimal transport in our task.
First of all, we tested and basic Earth Moving Distance (EMD) and
Sinkhorn (Sinkh) (Cuturi, 2013) algorithms.
Then we tested regularized versions of the Sinkhorn algorithm with a group lasso regularization (L1L2) and Laplacian regularization (L1LP) (Courty et al., 2015).
Finally, we benchmarked the semi-supervised (SS) Sinkhorn algorithm and Linear OT mapping estimators (MapT) (Perrot et al., 2016) in our problem.
We provide experiments in semisupervised settings with 10 and 100 known labels per class for each dataset, the results presented in Table 1.
In the tables, we can see that mapping to ”easy” representation greatly improves the accuracy of the optimal transport algorithms.
All these scores were computed on the test sets of the
MNIST and USPS.
C ONCLUSION AND FUTURE WORK
We propose a simple algorithm for the mutual improvement of optimal transport and energy-based learning for the semi-supervised domain adaptation problem.
Using input gradients of the energybased model, we transform our target examples and get a new source distribution that can be accurately classiﬁed by the energy-based model.
These transformations are class-vise cyclical monotone and can be approximated by optimal transport approaches much more successful than a standard target to the source domain transformation.
We tested our algorithm over the set of optimal transport methods on the MNIST and USPS datasets.
This paper is a work in progress.
Using Langevin dynamics for sampling from the underlying distribution deﬁned by EBM (instead of ﬁnding min) can improve the performance of the proposed methods.
In the future, we plan to test our method over the more complicated datasets and optimal transport methods.
Also, it is important to test convolution ICNNs and develop new input convex architectures to get not class-wise but full cyclical monotonicity between target and “easy” domains.