R EALISM I NDEX : I NTERPOLATION IN G ENERATIVE M ODELS WITH A RBITRARY P RIOR
Anonymous authors
Paper under double-blind review
A BSTRACT
In order to perform plausible interpolations in the latent space of a generative model, we need a measure that credibly reﬂects if a point in an interpolation is close to the data manifold being modelled, i.e. if it is convincing.
In this paper, we introduce a realism index of a point, which can be constructed from an arbitrary prior density, or based on FID score approach in case a prior is not available.
We propose a numerically efﬁcient algorithm that directly maximises the realism index of an interpolation which, as we theoretically prove, leads to a search of a geodesic with respect to the corresponding Riemann structure.
We show that we obtain better interpolations than the classical linear ones, in particular when either the prior density is not convex shaped, or when the soap bubble effect appears.
I NTRODUCTION
Since the advent of the Variational Auto-Encoder (VAE) (Kingma & Welling, 2013) and the Generative Adversarial Network (GAN) (Goodfellow et al., 2014), generative models became an area of intensive research, with new models being developed (e.g. Kingma & Dhariwal (2018); Larsen et al.
; Tolstikhin et al.
In these models, the data distribution is mapped into the latent space.
An important proﬁt from introducing the latent space is the ability to construct interpolations, i.e. traversals between latent representations of two different objects.
Meaningfulness of decoded interpolations is often used as a supporting argument for networks generalisation capability (Bowman et al., 2015; Dumoulin et al., 2016).
Interpolation is used commonly to show that models do not overﬁt, but generalise well (Dumoulin et al., 2016;
Goodfellow et al., 2014; Higgins et al., 2016; Kingma & Welling, 2013).
Intuitively, a good interpolation should decode to meaningful objects, give a gradual transformation, and reﬂect the internal structure of the dataset.
More precisely, we require that the interpolation curve (after transporting to the input space) is smooth and relatively short, while at the same time it goes through regions of high probability in the latent space (see leftmost projection in Fig. 1).
However, in some cases, even for a Gaussian prior, a linear interpolation could be of poor quality, e.g. due to the so called ”soap bubble effect” (Husar, 2017; Lesniak et al., 2019).
This may result in low quality samples in the middle of the path (White, 2016).
The above argument puts the usability of simple linear interpolation in question and motivates further research in this area (Agustsson et al., 2017;
Brock et al., 2016; Kilcher et al., 2017; Larsen et al., 2015; Lesniak et al., 2019; White, 2016).
In this paper, we construct a general interpolation scheme, which works well for arbitrary priors.
We introduce a notion of a realism index of an element of the latent space, which naturally generalises to arbitrary curves.
We show that in general the proposed method can be regarded as a search for geodesics in a respectively modiﬁed local Riemann structure.
The realism index can be either deﬁned internally, with the use of the prior latent density, or by some external feature space, e.g. similarly to Fr´echet Inception Distance (FID) score (Heusel et al., 2017).
In addition, we propose a simple to implement iterative algorithm, that optimises an interpolation with respect to the introduced index.
As a consequence of our approach, we obtain an interpolation which simultaneously tries to optimise the two following features:
• the interpolating curve goes through regions where the generated points are realistic, • the length of the curve transported to the input data space is small.
[Figure]
The above approach is especially valuable if either the prior density is not convex-shaped, or when a kind of a soap-bubble effect appears (i.e. when in the generative model, the points generated near the origin are of much worse quality than the ones chosen randomly).
R EALISM INDEX OF A POINT
Let X ⊂ R N be a dataset.
Consider the case when we are given a manifold model for X , which consists of a decoder G from the latent Z = R D to the input data space:
[Equation]
This case covers both GAN-like and AE-like models.
We deﬁne the realism index on Z as a function
[Equation]
such that high values of ri( z ) indicate that G ( z ) is indistinguishable from the elements of X .
In general, the optimal choice of ri can be nontrivial, and may depend on the generative model in question.
In this paper we study two possible natural constructions.
If the density f in Z is given, and its high values at point z imply that G ( z ) is more realistic, we can base the construction of the realism index on the density.
This assumption was empirically observed to be true for some generative models, such as the GLOW model (Kingma & Dhariwal, 2018).
A different external approach can be constructed by using a separate network, which checks if a given point has similar features to that of the samples in the dataset (idea based on the Fr ´ echet Inception Distance FID score).
Normalisation procedure.
In all of our constructions we assume that we are given a function
[Equation]
with higher values indicating more realistic points.
To obtain the realism index based on f
we ﬁrst need to apply the following normalisation procedure 1 .
Let us ﬁrst show the basic idea.
Suppose that we have ordered the elements of the set X (in the representation given from the latent Z ) according to the order introduced by f .
Then as the realism index of a given point x we understand its normalised order in the sequence f
( i ) .
This motivates us to state the following deﬁnition.
Deﬁnition 1.
Let X be a random vector in Z , such that our data comes from the distribution G ( X ) .
We deﬁne the realism index ri based on the function f by the formula
[Equation]
where p denotes the probability.
Observe that ri ( z, f ) =
(cid:82) { w : f ( w ) ≤ f ( z ) }
f ( s ) ds .
Clearly it is the PDF of random variable f ( X ) at point f
( z ) .
The proposed index lies in the [0 , 1] interval, and attains the value of 1 only for points where f attains maximal value.
If f is clear from the context, for notational convenience we shall write ri( z ) instead of ri( z ; f ) .
To practically apply the above concept, we need to be able to tune it to different generative models.
To achieve this, we choose a value ε < 1 and, similarly to the approach used in GLOW (Kingma &
Dhariwal, 2018), we rescale ri
[Equation]
so that ri ε ( z ) ∈
[ ε, 1 − ε ] .
See Sec.
4 for further discussion on this rescaling.
Realism index based on the normal density.
We shall now discuss the realism index based on the standard normal density f = N (0 , I ) in the D -dimensional latent Z .
Let us choose a point z from the latent and let X denote the random vector with f density.
Note that almost all points generated from the Gaussian density in R D lie inside the sphere S (0 , D 1 / 2 ) .
Points outside of this sphere can be considered unrealistic, while those inside might be obtained with random sampling.
We want to compute the probability
[Equation]
By some easy calculations we get (see Appendix A for more details)
[Equation]
We can see that the density based realism index correctly identiﬁes the latent points that lay inside the sphere S (0 , D 1 / 2 ) as having an index approximately 1 , and points outside sphere as having index approximately 0 .
Observe, that this behaviour is not recognised by the density itself, which has no clear change at the border of the sphere S (0 , D 1 / 2 ) .
Realism index based on the Fr´echet Inception Distance (FID).
If the density is either not available or not completely reliable, we can base the realism index on an external measure of sample credibility.
To that end, we use an approach inspired by the Fr ´ echet Inception Distance (Heusel et al., 2017) (FID).
To compute FID score we start with a pretrained Inception network I .
The entire training set T is passed through I to obtain a set of feature vectors, and we estimate its density
N ( µ T , Σ T ) by computing its mean µ T and covariance matrix Σ T .
We apply the similar procedure to the data W generated by our model, and obtain the density N ( µ W , σ W ) .
Then we compute the
Fr´echet distance between two normal densities by the formula
[Equation]
However, in contrast to the original deﬁnition, we aim to compute the credibility of a single generated point, not the distance between two distributions.
To achieve that goal, we use simply the likelihood of the point x transported through I with respect to density N ( µ T , Σ T ) :
[Equation]
and apply ri f as the constructed realism index.
Observe that this realism index is based also on the density, but not in the latent space itself, but in some feature space constructed with the use of external network I .
Numerical estimation of the realism index.
Clearly, for an arbitrary function f the realism index does not have a closed form.
In order to obtain differentiable estimation of ri , we draw sample
W = ( w i ) i from the random variable X (or simply choose it from the dataset), and compute values of ( f ( w i ))
i .
Since the considered values are non-negative, to estimate the density we ﬁrst proceed by logarithm to whole of R by taking l i = log f ( w i ) , and compute either kernel or GMM density estimation g of the random variable log( f ( X )) .
Finally, we obtain that the estimator of the realism index ri( z ; f ) is given with the use of the cumulative density function of g : ri( z ; f ) ≈ cdf g
(log f ( z ))) .
R EALISM INDEX OF AN INTERPOLATION
Our concept for the deﬁnition of the realism index for a path is inspired by transition between movie frames.
The interpolation may be viewed as set of frames,where the ﬁrst frame denotes the beginning of the path and last its end.
Interpreting realism index as a probability that a given frame is realistic, we can deﬁne the respective index of the curve as the product of all its points.
Realism index for naturally parameterised curves.
Let γ : [0 , T ] → Z be an interpolating curve, such that
[Equation]
for some given x, y ∈ Z .
Additionally, we assume that the G γ is naturally parameterised
[Equation]
We discretize the curve by ﬁxing a time-step T/k (where k denotes the number of frames) and consider the sequence of intervals [ γ (0) , γ ( T/k )] , . . .
, [ γ ( T − T/k ) , γ ( T )] .
To obtain the reality measure of this sequence we compute the product of all realisticity values of its points to the power equal to their ”duration” T/k
[Equation]
where t i are the arbitrarily chosen intermediate points in the intervals [ iT/k, ( i + 1) T/k ] .
By taking the logarithm of the above expression and proceeding with k → ∞ we get
[Equation]
Consequently, we introduce the realism index of a naturally parameterised curve γ : [0 , T ] → R N by the formula
[Equation]
Since every curve can be uniquely naturally parameterised, we interpret its index as the index of its natural reparameterisation.
Therefore we arrive at the following general deﬁnition.
Deﬁnition 2.
Let ri be given realism index in Z .
For an arbitrary curve γ : [0 , T ] → Z we deﬁne the realism index ri of γ with
[Equation]
We further prove that the realism index of a curve is equal to its length with respect to a certain
Riemann structure on the latent space.
We will utilise this result in the next section in order to connect the search of optimal interpolation to the search of geodesics.
Directly from the deﬁnition we get the formula for the realism index in terms of the latent
[Equation]
where d G ( x ) denotes the derivative of G at point x .
Consequently, we obtain the following theorem:
Theorem 1.
Let the Riemann structure in the latent space Z be deﬁned with the local scalar product (cid:104) , (cid:105) z at a point
z using the following formula
[Equation]
Then
[Equation]
) where length is the number of points in a path.
O PTIMAL INTERPOLATION
Considering the results from the previous section, we are able to formulate the deﬁnition of an ri -optimal curve.
Deﬁnition 3.
Let ri be a realism index in Z for a generative model G : Z → R N .
Let x,
y ∈
Z be ﬁxed.
We call a curve γ : [0 , T ] → Z such that γ (0)
=
x , γ ( T ) =
y ri -optimal
(or, shortly, optimal) interpolation if it has the maximal realism index from all curves joining x with y .
Study ﬁrst the issue of searching for, at least locally, optimal interpolations.
Theorem 1 allows us to reformulate the problem as a task of ﬁnding geodesics.
Consequently, the standard results from
Riemann geometry apply (see Spivak (1999, Chapter 9)).
Without loss of generality we can reduce the problem to the case T = 1 and optimise the length functional.
However, due to the uniqueness of the local minima and local convexity of the functional, we can minimise the energy functional instead
[Equation]
The additional advantage is that curves which minimise the energy functional are parameterised proportionally to the natural parameterisation.
Concluding, by applying Theorem 1, the optimal curve γ : [0 , 1] → Z , γ (0) = x, γ (1) =
y , with respect to the realism index minimises:
[Equation]
In general, the search for geodesics can lead to nontrivial computations involving second derivatives.
However, for some special cases, we can signiﬁcantly simplify the minimisation process.
To justify this claim, we ﬁrst introduce the formula for the discretization of the integral in the energy functional.
Let γ : [0 , 1] →
R N be a curve such that
[Equation]
and divide the interval [0 , 1] into k equal sub-intervals, and denote the values
[Equation]
For k − 1 vectors x 1 , . . .
, x k − 1 in R N (where x 0 = x , x k = y ), approximate the value of (16) with 2 E yx [Equation] ) ≈
[Equation]
Considering all the above computations
[Equation]
to compute an optimal interpolation we need to minimise
[Equation]
19) over x 1 , . . .
, x k − 1 ∈ R N .
Optimisation procedure.
In all the experiments in the paper we achieve this goal using the standard gradient descent method initialising x
i with a linear interpolation x i = (cid:0) 1 − ik (cid:1) x + ik y for i = 1 , . . .
, k − 1 .
However, to accelerate the process of minimisation 2 , we alternate the gradient step with the following one: ﬁrst choose two random numbers i < j from the set { 0 , . . .
, k } and then consider the linear interpolation between x i and x j given with
[Equation]
Finally, if the linear interpolation has smaller energy then the original part, i.e. when
[Equation]
we replace x l by ¯ x l for l = { i + 1 , . . .
, j − 1 } .
Effects of ε -regularisation on optimisation.
Given a reality index ri( · ) , see eq.
(4), we have introduced its regularisation ri ε ( z )
=
ε +(1 − 2 ε )
· ri( z ) , ε ∈ [0 , 1 / 2] .
In practice, in all experiments we typically choose ε = 0 .
The reason behind such regularisation is twofold.
First, if ε = 0 and ri is zero (or numerically close to zero) on some subset of the domain (for example if the density is zero), the optimisation for points with initial interpolating interval there is unmanageable (observe that in the realism of the curve we take the logarithm of the index).
The case when ε = 0 and ri =at some subset of the domain can also cause problems, as then logarithm of the reality index in this set is zero, and consequently the interpolating curve has no cost of staying or going through this region.
Consequently, in our experience it is best to regularise ri by restricting its image to a subset of interval [ ε, 1 − ε ] .
Let us now discuss the special limiting case when ε = 1 / 2 .
Then clearly ri ε ≡ 1 / 2 .
Let us ﬁrst recall that we are given a decoder (generator) G : Z → R N , such that the data set lies in the manifold given by M = G ( Z ) .
Let us consider the interpolating curve γ : [0 , 1] → Z .
Observe that in that case the energy function does not depend on ri and equals
[Equation]
Thus the minimal E yx ( γ ) value would be obtained for a curve joining x with y whose length measured in the input space is minimal.
Thus if ¯ x = G ( x ) and ¯ y = G ( y ) are points in the input space, then the interpolating curve with minimal energy is equal to one that connects points ¯ x and ¯ y with a curve in the manifold M which has minimal length (measured in input space).
Consequently, although the limiting case does not take into account the realism index of the interpolation, it still will usually produce interpolations comparable to the common linear.
See Appendix D for additional analysis.
E XPERIMENTS
In all the considered experiments we apply the regularised version or realism index ri ε with ε = 0 .
First we are going to consider the case when the index is based on the prior density in the latent, and next we brieﬂy discuss the realism index based on the FID score.
Density based index.
In this part we demonstrate our method’s ability to produce more meaningful interpolations.
In order to achieve this goal we use a DCGAN model (Radford et al., 2015) trained on MNIST and Celeb-A datasets (LeCun et al., 1998; Liu et al., 2015).
We consider a non-trivial latent created from a conjunction of three multidimensional Gaussian distributions
[Equation]
where µ i :=
(˜ µ i , 0 , . . . , 0) T ∈ R 20 for ˜ µ 1 = (2 , 6) , ˜ µ 2 = (0 , 0) , ˜ µ 3 = (2 , − 6) and Σ
i := (cid:2) M ( i )
M 12 M 21 M 22 (cid:3) for M 12 =
M 21 = 0 , M 22 is 2-D array with 0 .
5 on the diagonal and zeros elsewhere, and M (1)11 = (cid:2) 5 2 2 2 (cid:3) , M (2)11 = (cid:2) 1 0 0 3 (cid:3) , M (3)11 = (cid:2) 5 − 2 − 2 2 (cid:3) .
From here on we shall call it a semicircle, see
Fig.
1 (leftmost subﬁgure).
Results of sampling from a GAN trained with the semicircle prior are shown in Fig.
2 with respective projection shown in Fig.
We randomly choose two points from different ends of the distribution and start the algorithm’s minimisation procedure given with eq.
(19) with a linear interpolation (green line in Fig. 1).
It is obvious that the midpoints of initial path are sampled from a very lowdensity areas, completely outside the prior.
In Fig.
1 you can also the ri index values for the linear (center) and proposed (rightmost image) interpolations.
Our method allows for more dynamic behaviour in this matter, especially at the end points.
Images in the path obtained with proposed algorithm differ more, and at the same time seem more real.
FID-score based realism index.
In order to test our FID-score based realism index, we perform experiments on a GAN, trained on the Celeb-A dataset.
Since samples from a high dimensional
Gaussian approximately cluster on a sphere, points in the latent space that are very close to the origin are virtually never seen during training.
Therefore, samples generated in that area may or may not be realistic.
We selected a GAN model, for which we noticed that decoding latent codes near the origin gives low quality samples (the earlier mentioned soap-bubble effect), which makes for a perfect test environment for the FID-score based realism index.
We aim to choose interpolation endpoints x and y in such a way, that the linear interpolation between them would pass close to the origin in the latent space.
To that end, we ﬁrst sample x uniformly, and then produce y from x by multiplying by − 1 a random subset of dimensions.
The results of these experiments can be seen in Fig.
We can see that the linear interpolations yield very low quality samples (especially in the middle of the interpolation).
After optimising the linear paths using the FID-score realism index, the paths avoid unrealistic regions in the latent space, producing samples of much higher quality.
Note that this experiment does not require access to the density.
Instead, it is based only on the training samples.
Even though FID score has some limitations (Shmelkov et al., 2018), observe that its application in our model results in more realistic interpolations, as compared to linear ones.
Use of a measure more adapted to evaluating quality of data could result in further improvement in the realism of interpolations.
Further discussion.
In this part we discuss when the linear approximations will be close to those given by linear.
Let us ﬁrst consider the case when the generator is linear with prior uniform distribution in the latent.
Observation 1.
Let U be a convex bounded set in Z and let f denote the uniform distribution on
U .
We consider the case of linear generative model, i.e. where G is a linear map x → Ax (with A injective).
Then the ri -optimal interpolations are given by the linear interpolations.
Proof.
Clearly, we can equivalently compute the realism index of a curve γ connecting two points x, y by computing the standard euclidean distance of Aγ in the convex set AU .
More precisely: ri( γ ; uni U ) =
exp( − length( Aγ )) .
Since linear maps move intervals onto intervals, we obtain the assertion of the observation.
In practice, similar behaviour happens when the derivative of the generator has small variation and the realism index is close to being a constant one.
Observe, that by equation (6), for a generative model with high dimension of the latent space and Gaussian prior, the realism index is almost constant on a linear interpolation of arbitrary randomly chosen points (except for a possibly small neighbourhood of the endpoints).
Consequently, when the derivative of generator G does not vary too much in the vicinity of the linear interpolation, the linear interpolation will be close to optimal ﬁnd by our approach.
In practice, we observe this behaviour in auto-encoder based generative in models, such as Wasserstein Auto-Encoders, see Fig. 4.
This follows from the fact that too high variation (of derivative) of the decoder is penalised (regularised) by its approximate inverse given by the encoder.
C ONCLUSIONS
In this paper we studied the problem of generating a meaningful interpolation from a previously trained generative model, either a GAN or a generative auto-encoder.
We claim that a good interpolation should both reveal the hidden structure of the dataset, as well as be smooth and follow the true data distribution, i.e. produce realistic elements.
In order to produce curves satisfying these conditions we deﬁne a realism index of a path, which takes into account both density values and and differences between consecutive decoded images to ensure smoothness.
We show how to deﬁne realism index using either a known density or, in case it is either not available or not reliable, how to base it on some external measure, e.g. the FID score.
We have proved that this interpolation procedure is equal to ﬁnding a geodesics with reality index equal to its length in respect to some latent space Riemann structure.
For the practical use, we have deﬁned the notion of an optimal interpolation and proposed a simple and efﬁcient numerical procedure for its search.
The experiments show that the constructed interpolations are in superior to the linear ones, making it possible to escape regions of low data density or low data quality, both for the density- and FID-based approaches.
This is especially visible if the prior density is not Gaussian, when the linear interpolations often proceed through regions in space of extremely low density.
Another example when the linear interpolations are suboptimal to our method is given by the standard GAN model when the soap bubble effect appears
A C ASE OF THE NORMAL DENSITY .
We shall now compute the probability index of the standard normal density f = N (0 , I ) in the
D -dimensional latent Z .
Let us choose a point z from the latent and let X denote the random vector with density f .
We want to compute the probability ri
[Equation]
Observe that from the deﬁnition of normal density we have
[Equation]
Since (cid:107) X (cid:107) 2 has the chi-square distribution with D degrees of freedom
[Equation]
we obtain that
[Equation]
where F ( r ; D ) denotes the cumulative distribution function of the chi-square density χ 2 ( D ) (with
D degrees of freedom).
Let us now proceed with an asymptotic analysis.
Observe that
[Equation]
If Y ∼ χ 2 ( D ) then for large D >
[Equation]
√ 2 Y − √ 2 D − 1 is approximately normally distributed (see Johnson et al.
(1994, formula (18.23) on p. 426)).
Consequently
[Equation]
where Φ denotes cdf of standard Gaussian.
Since Φ( r ) = 12 (cid:2) 1 + erf( r/ √ 2) (cid:3) we get ri
[Equation] [Equation]
Remark 1.
The above formula implies that for the normal density f = N (0 , I ) in R D the realism index is approximately 1 in the ball B (0 , r D − 3) and approximately 0 outside of the ball B (0 , r D + 3) , where r D = (cid:113) D − 12 .
This behaviour is natural and expected, as most of the points generated from normal density concentrate around the sphere S (0 , r D ) .
Let us now discuss why we use the normalisation given by eq.
(4) , and not the seemingly natural given for bounded f by the formula
[Equation]
Consider the case when f = N (0 , I ) in R D .
Then ri( z ) =
exp( −(cid:107) x (cid:107) 2 / 2) , which means that ri(0) = 1 , but ri( z ) =
exp( − D/ 2 + 1 / 4) for z ∈ S (0 , r D ) .
Since the randomly generated point from N (0 , I ) has norm close to r D , this implies that almost every point which comes from the normal density would have realism index close to 0 , which would be of undesired and pathological behaviour.
In practice, we observe this behaviour in auto-encoder based generative models, such as Wasserstein
Auto-Encoders, see Fig. 4.
This follows from the fact that too high variation (of derivative) of the decoder is penalised by its approximate inverse given by the encoder.
B D ENSITY BASED REALISM INDEX IN GAN MODELS
In this section we present additional results for the optimisation of the density based realism index of an interpolation curve in a GAN model.
The experiments are conducted on the Celeb-A and
MNIST datasets, using the DCGAN architecture.
The optimisation procedure is initialised with a linear interpolation consisting of 50 points and implemented using Adam optimiser.
To show the relation of the starting linear interpolation to the one obtained from the proposed procedure, see e.g.
Fig.
1, we perform a projection of k interpolation z i points onto point ( x
, y ) ∈ R 2 such that
[Equation]
where ˜ z i :=
X ( X T X ) − 1 X T z
i are the latent space points z
i ∈ Z and X
=
[ z 0 , z k ] .
[Figure]
The resulting images for Celeb-A for the Gaussian prior are shown in Fig.
5, and Fig.
6 shows their path projections.
It can be easily noticed that the proposed interpolation gives more dynamical objects.
Also the interpolation is pulled by the density.
Similar observations can be observed for semicircle prior.
The resulting images for Celeb-A together with their path projections for semicircle prior are shown in Fig.
We made also the same experiments for MNIST dataset.
The resulting images for this data for the Gaussian and semicircle prior are shown in Fig.
In the case of auto-encoder based architectures we advice the use of the WAE model, since it obtains slightly better interpolations and reconstructions than VAE – see Fig. 9.
[Figure] C O PTIMISATION PROCEDURE In the performed experiments we use the optimisation described in Section 4 to reduce the number of necessary iterations in our method. In the initial phase of the optimisation of the linear interpolation, the value of the gradient for the middle points (located in the middle of the interpolation curve) can be extremely small due to the diminishing density.
This is particularly evident for a DCGAN model with a semicircle prior, see Fig.
In consequence, this part of the curve converges slower than the points lying closer to the ends of the curve.
Our experiments for a semicircle distribution show that we obtain the same ﬁnal result without using the described acceleration, but with up to 3 times more iterations.
This optimisation procedure is crucial only for non typical densities; for the classical normal prior no signiﬁcant impact on the number of iterations is observed.
D T HE ε VALUE IMPACT ON THE INTERPOLATION CURVE
In Fig.
11 we present squared L 2 distances between consecutive points in the optimised path from the last example from Fig.
In this experiment we use different values of epsilon ε = 10 − 5 , 10 − 4 , 10 − 3 , 0 . 01 , 0 . 1 , 0 .
25 for ri ε (see eq.
As one can see, distances between consecutive points do not vary much for ε values much larger than zero.
Higher ε values result in more equally spaced interpolations, and the interpolation optimisation process is faster.
However, higher ε result in puts a different emphasis on terms of the formula optimised (see eq.
(19) and discussion on ε in Sec. 4), i.e. lengths start to be more important (hence the inter-point distances) than individual point realities.
On the other hand, lower ε values result in non-equally spaced points on the interpolation, which is easily visible in Fig.
11, while the optimisation process is slower.
Empirically, we found ε = 0 .
1 to be optimal, and therefore it is used as default value.
Experiments conducted on a Celeb-A dataset gave similar results.
[Figure]
[Figure]