Original: NeurInt-Learning Interpolation by Neural ODEs
Revised: NeurInt-Learning Interpolation by Neural ODEs

Original: Anonymous Author(s)
Revised: avibose@ ac in Aniket Das Indian of aniketd@iitk.ac.in in Piyush Rai

Original: AfﬁliationAddress email
Revised: Indian piyush@cse.iitk.ac.in

Original: Abstract
Revised: Abstract

Original: A range of applications require learning image generation models whose latentspace effectively captures the high-level factors of variation in the data distribution, which can be judged by its ability to interpolate between images smoothly.
Revised: A range of applications require learning image generation models whose latent space effectively captures the high-level factors of variation in the data distribution, which can be judged by its ability to interpolate between images smoothly.

Original: However, most generative models mapping a ﬁxed prior to the generated imageslead to interpolation trajectories lacking smoothness and images of reduced quality.
Revised: However, most generative models mapping a ﬁxed prior to the generated images lead to interpolation trajectories lacking smoothness and images of reduced quality.

Original: We propose a novel generative model that learns a ﬂexible non-parametric priorover interpolation trajectories, conditioned on a pair of source and target images.
Revised: We propose a novel generative model that learns a ﬂexible non-parametric prior over interpolation trajectories, conditioned on a pair of source and target images.

Original: Instead of relying on deterministic interpolation methods like linear or sphericalinterpolation in latent space, we devise a framework that learns a distribution oftrajectories between two given images using Latent Second-Order Neural Ordinary Differential Equations.
Revised: Instead of relying on deterministic interpolation methods like linear or spherical interpolation in latent space, we devise a framework that learns a distribution of trajectories between two given images using Latent Second-Order Neural Ordinary Differential Equations.

Original: Through a hybrid combination of reconstruction andadversarial losses, the generator is trained to map the sampled points from thesetrajectories to sequences of realistic images of improved quality that smoothlytransition from the source to the target image.
Revised: Through a hybrid combination of reconstruction and adversarial losses, the generator is trained to map the sampled points from these trajectories to sequences of realistic images of improved quality that smoothly transition from the source to the target image.

Original:  Introdcution
Revised:  Introduction

Original: In the past few years, deep generative models’ incredible success has demonstrated their abilityto represent the underlying factors of variations in high dimensional data, such as images via lowdimensional latent variables.
Revised: In the past few years, deep generative models’ incredible success has demonstrated their ability to represent the underlying factors of variations in high dimensional data, such as images via low dimensional latent variables.

Original: These factors of variation are commonly visualized by interpolatingbetween images by traversing particular paths in the latent space.
Revised: These factors of variation are commonly visualized by interpolating between images by traversing particular paths in the latent space.

Original: Given any two images, it is oftendesirable to obtain a distribution over various possible trajectories of smooth and realistic image spaceinterpolations.
Revised: Given any two images, it is often desirable to obtain a distribution over various possible trajectories of smooth and realistic image space interpolations.

Original: Learning such distribution would allow a more extensive analysis of the factors ofvariation in data.
Revised: Learning such distribution would allow a more extensive analysis of the factors of variation in data.

Original: In this work, we propose an approach that jointly trains an encoder and a generatorto successively transform latent vectors on a trajectory to interpolations from a given source to atarget image.
Revised: In this work, we propose an approach that jointly trains an encoder and a generator to successively transform latent vectors on a trajectory to interpolations from a given source to a target image.

Original: To ﬂexibly model distributions over trajectories of latent vectors, we parameterizetheir dynamics in continuous-time using Neural Ordinary Differential Equations [5].
Revised: To ﬂexibly model distributions over trajectories of latent vectors, we parameterize their dynamics in continuous-time using Neural Ordinary Differential Equations [5].

Original: We refer to ourapproach as Neural Interpolation (NeurInt) (Figure 1).
Revised: We refer to our approach as Neural Interpolation (NeurInt) (Figure 1).

Original: Ideally, we wish every point in the latent space to map to a unique, real image.
Revised: Ideally, we wish every point in the latent space to map to a unique, real image.

Original: However, it isunrealistic to expect a model to learn the entire data distribution over inﬁnitely many real imagesgiven only a ﬁnite dataset [2].
Revised: However, it is unrealistic to expect a model to learn the entire data distribution over inﬁnitely many real images given only a ﬁnite dataset [2].

Original: Thus, given any image, we can only expect the model to learn totransform it to neighboring realistic images through suitable incremental changes.
Revised: Thus, given any image, we can only expect the model to learn to transform it to neighboring realistic images through suitable incremental changes.

Original: In view of this,instead of generating an image from random noise, our approach encourages the model to traversethrough realistic images while maintaining smoothness and a net movement towards the target image.
Revised: In view of this, instead of generating an image from random noise, our approach encourages the model to traverse through realistic images while maintaining smoothness and a net movement towards the target image.

Original: By learning to interpolate instead of using a deterministic interpolation technique, we allow themodel to generate different categories of interpolations for different source and target images.
Revised: By learning to interpolate instead of using a deterministic interpolation technique, we allow the model to generate different categories of interpolations for different source and target images.

Original: Thisis achieved by learning a distribution over trajectories conditioned on the source and target images,parameterized by second-order Neural ODEs.
Revised: This is achieved by learning a distribution over trajectories conditioned on the source and target images,

Original: Leveraging a data-dependent latent space distribution,parameterized through Neural ODEs, lends our approach the following major advantages: 1.

Original: Directutilization of real images while sampling latent vectors allows our approach to incorporate the beneﬁts Input: Dataset P data ( x ) , Hyperparameter λ  Integration time T  2 Set  trajectory z 0 :
Revised: Input: Dataset P data ( x ) , Hyperparameter λ and Integration time T 1 Sample x S and x T from the dataset P data ( x ) 2 Set initial latent z 0 of trajectory z 0 :=

Original: = E ( x S ) 3 Sample Initial Velocity : (cid:15) ∼ N (0 , I ) [Equation] ( z 0 , E ( x T ))
Revised: E ( x S ) 3 Sample Initial Velocity : (cid:15) ∼ N (0 , I ) v 0 = µ v ( z 0 , E ( x T ))

Original: + (cid:15) (cid:12) σ v ( z 0 , E ( x T ))
Revised: + (cid:15) (cid:12) σ v ( z 0 , E ( x T ))

Original: 4 Solve ODE System for for z t : [0 , T ] −→ X : d 2 z t 
Revised: 4 Solve ODE System for for z t : [0 , T ] −→ X : d 2 z t dt 2 = f

Original: [Equation]
Revised: ( z t , ˙ z t ) ˙

Original: 5 Compute Reconstruction Loss
Revised: z 0 = v5 Compute Reconstruction Loss

Original: [Equation]
Revised: L AE =

Original: 6{+(cid:107) x S − G ( z 0 ) (cid:107) 
 + (cid:107) x T − G ( z T ) (cid:107)6+} Sample t 1 , ..., t N from Uniform (0 , T ) without replacement 7 Sample x 1 , ..., x N from dataset
Revised: {+(cid:107) x S − G ( z 0 ) (cid:107) 
 + (cid:107) x T − G ( z T ) (cid:107)6+} Sample t 1 , ..., t N from Uniform (0 , T ) without replacement 7 Sample x 1 , ..., x N from dataset

Original: [Equation]
Revised: L GAN = (cid:80) Ni =1 log( D ( x i )) + log(1 − D ( G ( z t i )))

Original: 8 Optimize the minimax game with (Stochastic) Gradient Descent Ascent
Revised: 8 Optimize the minimax game with (Stochastic) Gradient Descent Ascent

Original: [Equation]
Revised: [Equation]

Revised: Input: Source Image x S , Target Image x T and Integration time T

Revised: Output: Continuous-time Interpolation Curve ˜ x t : [0 , T ] −→ X 1 Set initial latent z 0 of latent space trajectory z 0 :

Revised: = E ( x S ) 2 Sample Initial Velocity : (cid:15) ∼ N (0 , I ) v 0 = µ v ( z 0 , E ( x T ))

Revised: + (cid:15) (cid:12) σ v ( z 0 , E ( x T ))

Revised: 3 Solve ODE System for for z t : [0 , T ] −→ X : d 2 z t dt 2 = f

Revised: ( z t , ˙ z t ) ˙

Original: {+z 0 = v
+}
Revised: {+z 0 = v
+}

Revised: Generate Interpolation Curve ˜ x t : [0 , T ] −→ X

Original:  of non-parametric approaches like the ability to incorporate additional data into the generative modelwithout retraining the parameters.
Revised: [Equation] parameterized by second-order Neural ODEs. Leveraging a data-dependent latent space distribution, parameterized through Neural ODEs, lends our approach the following major advantages: 1. Direct utilization of real images while sampling latent vectors allows our approach to incorporate the beneﬁts of non-parametric approaches like the ability to incorporate additional data into the generative model without retraining the parameters.

Original: Flexibility to learn different latent space distributions dependingon the training data’s complexity and size.
Revised: Flexibility to learn different latent space distributions depending on the training data’s complexity and size.

Original: The second-order formulation allows our model tomap randomly sampled initial velocities from a simple Gaussian prior to a highly expressive classof smooth trajectories corresponding to different vector ﬁelds on the data distribution manifold. 4.
Revised: The second-order formulation allows our model to map randomly sampled initial velocities from a simple Gaussian prior to a highly expressive class of smooth trajectories corresponding to different vector ﬁelds on the data distribution manifold. 4.

Original: Continuous nature of the ODE allows us to sample an arbitrary number of points in each trajectoryto obtain the desired level of smoothness in the interpolation trajectories where the smoothnessis naturally enforced by the ODE formulation.
Revised: Continuous nature of the ODE allows us to sample an arbitrary number of points in each trajectory to obtain the desired level of smoothness in the interpolation trajectories where the smoothness is naturally enforced by the ODE formulation.

Original: Jointly enforcing smoothness and realism ofinterpolated images prevents the model from simply memorizing training data.
Revised: Jointly enforcing smoothness and realism of interpolated images prevents the model from simply memorizing training data.

Original: Since our model’slatent space distribution directly depends on the encoder, we do not require explicit matching of priorand posterior distributions, unlike other encoder-generator based approaches such as VAE [21] and
Revised: Since our model’s latent space distribution directly depends on the encoder, we do not require explicit matching of prior and posterior distributions, unlike other encoder-generator based approaches such as VAE [21] and

Original: ALI [10] 7.
Revised: ALI [10] 7.

Original: The computational cost of sampling trajectories can be varied during training and testtimes by using different discretization schemes depending on the available computational resources.
Revised: The computational cost of sampling trajectories can be varied during training and test times by using different discretization schemes depending on the available computational resources.

Original:  Neural Interpolation (NeurInt)
Revised:  Neural Interpolation (NeurInt)

Original: Our approach, NeurInt, models a distribution over smooth continuous-time interpolation curves˜ x t : [0 , T ] −→ X (where T ∈ R + and X represents the image manifold) which start from a givensource image x S and end at a target image x T .
Revised: Our approach, NeurInt, models a distribution over smooth continuous-time interpolation curves ˜ x t : [0 , T ] −→ X (where T ∈ R + and X represents the image manifold) which start from a given source image x S and end at a target image x T .

Original: Similar to other latent variable-based generativemodels, we deﬁne the generated data distribution p ( x ) as the distribution obtained by transforming alatent space distribution p ( z ) through a generator G .
Revised: Similar to other latent variable-based generative models, we deﬁne the generated data distribution p ( x ) as the distribution obtained by transforming a latent space distribution p ( z ) through a generator G .

Original: However, unlike generative models with ﬁxedparametric priors, the latent space distribution p ( z ) in our model is deﬁned through a distributionover latent trajectories z t : [0 , T ] −→ Z conditioned over source and target images.
Revised: However, unlike generative models with ﬁxed parametric priors, the latent space distribution p ( z ) in our model is deﬁned through a distribution over latent trajectories z t : [0 , T ] −→ Z conditioned over source and target images.

Original: Time evolutionof these trajectories is governed by a Second-Order Neural ODE of the form d 2 z t dt 2 = f
Revised: Time evolution of these trajectories is governed by a Second-Order Neural ODE of the form d 2 z t dt 2 = f

Original: ( z t , ˙ z t ) .
Revised: ( z t , ˙ z t ) .

Original: Inorder to ensure that all image interpolation curves that are conditioned on x S and x T begin at thesource, a Position Encoder E is used to project x S to Z , and the initial position z 0 of the trajectory isset to E ( x S ) .
Revised: In order to ensure that all image interpolation curves that are conditioned on x S and x T begin at the source, a Position Encoder E is used to project x S to Z , and the initial position z 0 of the trajectory is set to E ( x S ) .

Original: The distribution over latent trajectories is deﬁned by placing a data dependent prior onthe initial velocity v 0 or ˙ z 0 .
Revised: The distribution over latent trajectories is deﬁned by placing a data dependent prior on the initial velocity v 0 or ˙ z 0 .

Original: For our modeling purposes, we choose the prior p ( v 0 | x S , x T ) to be a
Revised: For our modeling purposes, we choose the prior p ( v 0 | x S , x T ) to be a

Original: Diagonal Gaussian whose parameters are given by the Velocity Encoder V = ( µ v , σ v ) .
Revised: Diagonal Gaussian whose parameters are given by the Velocity Encoder V = ( µ v , σ v ) .

Original: Sampling a latent trajectory z t hence consists of evaluating z 0 and sampling an initial velocity v 0from the data dependent prior.
Revised: Sampling a latent trajectory z t hence consists of evaluating z 0 and sampling an initial velocity vfrom the data dependent prior.

Original: This ﬁxes the Initial Value Problem (IVP) for the trajectory, whichcan now be obtained by (numerical) integration of the ODE system.
Revised: This ﬁxes the Initial Value Problem (IVP) for the trajectory, which can now be obtained by (numerical) integration of the ODE system.

Original: The true image space curve ˜ x tis then obtained by transforming z t using the generator, and can be sampled at arbitrarily chosentime-points in the range [0 , T ] to produce image samples.
Revised: The true image space curve ˜ x t is then obtained by transforming z t using the generator, and can be sampled at arbitrarily chosen time-points in the range [0 , T ] to produce image samples.

Original: The learning objective of the model ensures that a given image space curve ˜ x t which is conditionedon a source target pair ( x S , x T ) begins at the source and ends at the target.
Revised: The learning objective of the model ensures that a given image space curve ˜ x t which is conditioned on a source target pair ( x S , x T ) begins at the source and ends at the target.

Original: This is ensured by apixel-MSE based reconstruction objective L AE that matches ˜ x 0 to x S and ˜ x T to x T .
Revised: This is ensured by a pixel-MSE based reconstruction objective L AE that matches ˜ x 0 to x S and ˜ x T to x T .

Original: The realismand diversity of interpolation curves is ensured via Adversarial Learning.
Revised: The realism and diversity of interpolation curves is ensured via Adversarial Learning.

Original: We use the Generative
Revised: We use the Generative

Original: Adversarial Network to jointly train a critic D : X −→ [0 , 1] which discriminates real samples drawnfrom the data distribution against evaluations of the interpolation trajectory at randomly sampledtime-points .
Revised: Adversarial Network to jointly train a critic D : X −→ [0 , 1] which discriminates real samples drawn from the data distribution against evaluations of the interpolation trajectory at randomly sampled time-points .

Original: Learning is then formulated as a minimax game where the critic D plays against theencoders E and V , generator G and the Neural ODE f .
Revised: Learning is then formulated as a minimax game where the critic D plays against the encoders E and V , generator G and the Neural ODE f .

Original: The value function of the game is taken to bea weighted combination of the reconstruction and adversarial objectives.
Revised: The value function of the game is taken to be a weighted combination of the reconstruction and adversarial objectives.

Original: The entire training processis described in Algorithm 1.
Revised: The entire training process is described in Algorithm 1.

Original:  Experiments
Revised:  Experiments

Original: We benchmark NeurInt, which leverages learnable interpolation trajectories, against the interpolationsgenerated by the Spherical (SLERP) and Linear Interpolations (LERP) on two base generativemodels, Progressive GAN (PGAN) [19] and Adversarially Learned Autoencoder (ALI) [10], on the
Revised: We benchmark NeurInt, which leverages learnable interpolation trajectories, against the interpolations generated by the Spherical (SLERP) and Linear Interpolations (LERP) on two base generative models, Progressive GAN (PGAN) [19] and Adversarially Learned Autoencoder (ALI) [10], on the

Original: Street View House Numbers (SVHN) [28] and CelebA [24] datasets.
Revised: Street View House Numbers (SVHN) [28] and CelebA [24] datasets.

Original: To maintain uniformity, thearchitecture of the Generator and Discriminator for all three models, and that of the Encoder for
Revised: To maintain uniformity, the architecture of the Generator and Discriminator for all three models, and that of the Encoder for

Original: ALI and NeurInt resemble a standard Progressive GAN.
Revised: ALI and NeurInt resemble a standard Progressive GAN.

Original: The Neural ODE component of NeurIntuses a Runge Kutta (RK4) integrator with 32 integration timesteps and a total integration time of 1second ( T = 1 ).
Revised: The Neural ODE component of NeurInt uses a Runge Kutta (RK4) integrator with 32 integration timesteps and a total integration time ofsecond ( T = 1 ).

Original: Further details regarding our architectural choices and hyperparameter selectionare described in the Appendix.
Revised: Further details regarding our architectural choices and hyperparameter selection are described in the Appendix.

Original: To evaluate our approach’s interpolation capabilities compared to thebaselines, we project the source and target images in the latent space of the respective models, andcompare the continuous-time learned interpolations of NeurInt against the interpolations generatedby SLERP and LERP for each of the baseline generative models.
Revised: To evaluate our approach’s interpolation capabilities compared to the baselines, we project the source and target images in the latent space of the respective models, and compare the continuous-time learned interpolations of NeurInt against the interpolations generated by SLERP and LERP for each of the baseline generative models.

Original: For NeurInt and ALI, projectingthe image to an encoding space is trivial since both the models jointly learn an encoder from theimage space to the latent space.
Revised: For NeurInt and ALI, projecting the image to an encoding space is trivial since both the models jointly learn an encoder from the image space to the latent space.

Original: However, since PGAN lacks any such means of projection, wetrain an encoder E P that learns to project images onto the trained Progressive GAN latent space.
Revised: However, since PGAN lacks any such means of projection, we train an encoder E P that learns to project images onto the trained Progressive GAN latent space.

Original: The architecture of E P is the same as that of the encoder E used by NeurInt and it is trained byminimising a pixel-wise MSE loss and the encoder is progressively grown to maintain consistency.
Revised: The architecture of E P is the same as that of the encoder E used by NeurInt and it is trained by minimising a pixel-wise MSE loss and the encoder is progressively grown to maintain consistency.

Original: Sample Quality, Diversity & Incorporating Unseen Data We quantitatively assess the generationand interpolation quality for NeurInt and our baselines using Frechet Inception Distance (FID) [15], astandard evaluation metric for GANs, and present the results in Table 5.
Revised: Sample Quality, Diversity & Incorporating Unseen Data We quantitatively assess the generation and interpolation quality for NeurInt and our baselines using Frechet Inception Distance (FID) [15], a standard evaluation metric for GANs, and present the results in Table 5.

Original: We randomly select 5000pairs of source-target images from a support distribution.
Revised: We randomly selectpairs of source-target images from a support distribution.

Original: For NeurInt we generate interpolationtrajectories for each pair and randomly sample two intermediate interpolants from each trajectory.
Revised: For NeurInt we generate interpolation trajectories for each pair and randomly sample two intermediate interpolants from each trajectory.

Original: For the PGAN and ALI baselines, we project each source-target pair into the latent space, and thengenerate trajectories using Spherical and Linear Interpolation.
Revised: For the PGAN and ALI baselines, we project each source-target pair into the latent space, and then generate trajectories using Spherical and Linear Interpolation.

Original: The FIDs so obtained are listed under
Revised: The FIDs so obtained are listed under

Original: LERP and SLERP in Table 5.
Revised: LERP and SLERP in Table 5.

Original: To decouple the evaluation of interpolation quality from that of samplegeneration, we also evaluate the FID of the baselines using samples drawn from their true generativemodel (listed as PGAN-PRIOR and ALI-PRIOR in Table 5), by sampling a latent code from theirrespective priors.
Revised: To decouple the evaluation of interpolation quality from that of sample generation, we also evaluate the FID of the baselines using samples drawn from their true generative model (listed as PGAN-PRIOR and ALI-PRIOR in Table 5), by sampling a latent code from their respective priors.

Revised: Adapting to Unseen Supports Since our approach models distribution over trajectories conditioned on source and target images, the generated data distribution can be ﬂexibly varied by modifying the distribution of source and target images.

Revised: This allows the trained model to improve the generated data’s diversity without retraining the parameters by incorporating additional data into the set of source and target images.

Original: [Table]
Revised: This is unlike the models based on ﬁxed parametric priors, which require retraining on new data to modify the generated data distribution. We repeat the evaluation by varying the support across 3 different distributions, namely the Train Set, the Test Set, as well as the Train and Test set combined. As demonstrated through the results in Table 1, utilizing additional data from the test set leads to improvement in FID scores. Our model’s ability to interpolate on test data also demonstrates its ability to model the entire image data manifold rather than overﬁtting on training images.

Original: Varying ODE Solver Conﬁguration
Revised: Varying ODE Solver Conﬁguration

Original: The Second-Order Neural ODE formulation of NeurInt allowsit to interpolate in continuous time.
Revised: The Second-Order Neural ODE formulation of NeurInt allows it to interpolate in continuous time.

Original: This imparts our approach the ﬂexibility of varying the the levelof discretization (number of time-steps) as well as the ODE solver at test time.
Revised: This imparts our approach the ﬂexibility of varying the the level of discretization (number of time-steps) as well as the ODE solver at test time.

Original: We validate this bygenerating trajectories while reducing the number of integrator steps from 32 to 20 in steps of 4 forboth RK4 and Euler algorithms.
Revised: We validate this by generating trajectories while reducing the number of integrator steps from 32 to 20 in steps of 4 for both RK4 and Euler algorithms.

Original: For a fair comparison, we correspondingly vary the time-resolutionof the LERP and SLERP interpolators of our baselines and benchmark the models using the FIDmetric.
Revised: For a fair comparison, we correspondingly vary the time-resolution of the LERP and SLERP interpolators of our baselines and benchmark the models using the FID metric.

Original: The results are presented in Table 2.
Revised: The results are presented in Table 3.

Original: We observe that even at very low integration time-steps,the RK4 variant of NeurInt consistently outperforms the baselines.
Revised: We observe that even at very low integration time-steps, [Table] [Table] the RK4 variant of NeurInt consistently outperforms the baselines.

Original: The same does not hold true forthe Euler integrator for very low timesteps, which could be attributed to the inherent coarseness andpiecewise linear nature of the Euler Integrator.
Revised: The same does not hold true for the Euler integrator for very low timesteps, which could be attributed to the inherent coarseness and piecewise linear nature of the Euler Integrator.

Original: The discretization invariance of NeurInt has immensepractical utility, as it allows us to train the model at a very ﬁne time resolution on a powerful hardwareconﬁguration, while deploying it at test time on less powerful hardware by reducing the accuracy andintegrator timesteps of the solver.
Revised: The discretization invariance of NeurInt has immense practical utility, as it allows us to train the model at a very ﬁne time resolution on a powerful hardware conﬁguration, while deploying it at test time on less powerful hardware by reducing the accuracy and integrator timesteps of the solver.

Original: We also report the time taken for generating the latent interpolantsin Table 2 for NeurInt as well as LERP and SLERP in PGAN.
Revised: We also report the time taken for generating the latent interpolants in Table 3 for NeurInt as well as LERP and SLERP in PGAN.

Original: We note that RK4, despite using fourintermediate increments per solver step, is consistently faster than SLERP.
Revised: We note that RK4, despite using four intermediate increments per solver step, is consistently faster than SLERP.

Original: [Table]
Revised: [Table]

Original: Ablation Study To evaluate the beneﬁts of jointly incorporating smooth interpolation on the latentspace through the use of second-order Neural ODEs and non-parametric data-dependent prior on thelatent space obtained by conditioning the generated images on the randomly sampled source andtarget images, we perform an ablation experiment ( NeurInt-PT) where we train the generative modelto map images from a ﬁxed latent space prior using the original Generative Adversarial Networksframework and subsequently utilize a Neural ODE network to learn realistic interpolation trajectorieson the ﬁxed latent space.
Revised: Ablation Study To evaluate the beneﬁts of jointly incorporating smooth interpolation on the latent space through the use of second-order Neural ODEs and non-parametric data-dependent prior on the latent space obtained by conditioning the generated images on the randomly sampled source and target images, we perform an ablation experiment ( NeurInt-PT) where we train the generative model to map images from a ﬁxed latent space prior using the original Generative Adversarial Networks framework and subsequently utilize a Neural ODE network to learn realistic interpolation trajectories on the ﬁxed latent space.

Original: In Table 5 we report the FID of NeurInt-PT.
Revised: In Table 5 we report the FID of NeurInt-PT.

Original: Choice of using a 2 nd Order ODE Free initial velocity parameter in 2 nd Order ODE allows us toparameterize a trajectory distribution for every source target pair.
Revised: Choice of using a 2 nd Order ODE Free initial velocity parameter in 2 nd Order ODE allows us to parameterize a trajectory distribution for every source target pair.

Original: Such a parameter is absent inﬁxed interpolation schemes and 1 st Order ODEs of the form d z t dt = f ( z t ) or d z t dt = f
Revised: Such a parameter is absent in ﬁxed interpolation schemes and 1 st Order ODEs of the form d z t dt = f ( z t ) or d z t dt = f

Original: ( z t , z C ) wherez C ∼ N ( µ v ( z 0 , E ( x T )) , σ v ( z 0 , E ( x T )) 2 I ) .
Revised: ( z t , z C ) where z C ∼ N ( µ v ( z 0 , E ( x T )) , σ v ( z 0 , E ( x T )) 2 I ) .

Original: Hence, using LERP, SLERP or a 1 st Order ODE in
Revised: Hence, using LERP, SLERP or a 1 st Order ODE in

Original: Algo.1 Step 4 would prevent us from learning a trajectory distribution for a source-target pair, sincesuch approaches uniquely ﬁx a trajectory given two endpoints and we quantitatively justify our choiceby training the corresponding models called LINT, SLINT, FOINT I and FOINT II.
Revised: Algo.1 Step 4 would prevent us from learning a trajectory distribution for a source-target pair, since such approaches uniquely ﬁx a trajectory given two endpoints and we quantitatively justify our choice by training the corresponding models called LINT, SLINT, FOINT I and FOINT II.

Original: As shown by
Revised: As shown by

Original: FID scores in Table 5, NeurInt outperforms these models.
Revised: FID scores in Table 5, NeurInt outperforms these models.

Original: A Related Work
Revised: A Related Work

Original: The progress in the design of latent variable-based deep generative models such as GANs [12],
Revised: The progress in the design of latent variable-based deep generative models such as GANs [12],

Original: VAEs [21] and normalizing ﬂows [31] in recent years has led to numerous applications.
Revised: VAEs [21] and normalizing ﬂows [31] in recent years has led to numerous applications.

Original: Generative
Revised: Generative

Original: Adversarial Networks (GANs), in particular, have been extensively utilized for image generation[19, 4, 9, 18], video generation [36, 6, 8], image translation [17, 41], as well as various other tasksrequiring the generation of high-dimensional data.
Revised: Adversarial Networks (GANs), in particular, have been extensively utilized for image generation [19, 4, 9, 18], video generation [36, 6, 8], image translation [17, 41], as well as various other tasks requiring the generation of high-dimensional data.

Original: This has also led to vast literature on improvinginference [10, 7, 25, 37, 26], and training stability [27, 1] of GANs.
Revised: This has also led to vast literature on improving inference [10, 7, 25, 37, 26], and training stability [27, 1] of GANs.

Original: However, work on improvingthe diversity, smoothness, and realism of interpolations has been limited.
Revised: However, work on improving the diversity, smoothness, and realism of interpolations has been limited.

Original: One of the major reasonsfor poor interpolation quality is the mismatch between latent vectors’ distributions corresponding tointerpolations and the prior distribution used during training.
Revised: One of the major reasons for poor interpolation quality is the mismatch between latent vectors’ distributions corresponding to interpolations and the prior distribution used during training.

Original: Some recent works have attempted totackle this mismatch by modifying the prior [38, 22] or using non-parametric priors [33].
Revised: Some recent works have attempted to tackle this mismatch by modifying the prior [38, 22] or using non-parametric priors [33].

Original: However,unlike our approach which ensures matching image distributions corresponding to entire trajectorieswith real data distribution, these works only focus on matching the latent space distribution formidpoints of sampled noise vectors.
Revised: However, unlike our approach which ensures matching image distributions corresponding to entire trajectories with real data distribution, these works only focus on matching the latent space distribution for midpoints of sampled noise vectors.

Original: A recent work [5] exploited Neural Ordinary Differential
Revised: A recent work [5] exploited Neural Ordinary Differential

Original: Equations’ inherently sequential nature to model continuous time dynamics of time-series data.
Revised: Equations’ inherently sequential nature to model continuous time dynamics of time-series data.

Original: However, such an approach cannot be directly applied to generating interpolations in the trainingdata.
Revised: However, such an approach cannot be directly applied to generating interpolations in the training data.

Original: Therefore, we leverage Neural ODEs [5] for traversing the latent space while ensuring realnessthrough the use of a discriminator.
Revised: Therefore, we leverage Neural ODEs [5] for traversing the latent space while ensuring realness through the use of a discriminator.

Original: Our work is also related to but different from Exemplar models[29] and Kernel Density Estimation [30].
Revised: Our work is also related to but different from Exemplar models [29] and Kernel Density Estimation [30].

Original: While Exemplar based generative models directly utilize thedataset for modeling the latent space distribution, each generated image in such models is obtained bymodifying only one randomly sampled image.
Revised: While Exemplar based generative models directly utilize the dataset for modeling the latent space distribution, each generated image in such models is obtained by modifying only one randomly sampled image.

Original: Our approach instead can ﬁnd points on image spacebetween any arbitrarily pair of images, enabling it to capture the full diversity of image data manifoldsfor both generation as well as interpolation.
Revised: Our approach instead can ﬁnd points on image space between any arbitrarily pair of images, enabling it to capture the full diversity of image data manifolds for both generation as well as interpolation.

Original: Moreover, our GAN based formulation allows directlymatching the image distribution of trajectories of interpolants with the real data distribution.
Revised: Moreover, our GAN based formulation allows directly matching the image distribution of trajectories of interpolants with the real data distribution.

Original: Thisobviates the need of utilizing a nearest-neigbour based approximation of a closed form non-parametricdistribution in the latent space.
Revised: This obviates the need of utilizing a nearest-neigbour based approximation of a closed form non-parametric distribution in the latent space.

Original: By directly enforcing smoothness and realism of interpolated images,our approach also prevents memorization of training data without utilizing regularization [29] orpseudo-inputs [35].
Revised: By directly enforcing smoothness and realism of interpolated images, our approach also prevents memorization of training data without utilizing regularization [29] or pseudo-inputs [35].

Original: The set of source and target images in our model can also be interpreted as a form of external memory,which has been shown to improve generation quality in several works [3, 14, 16, 34, 20, 23].
Revised: The set of source and target images in our model can also be interpreted as a form of external memory, which has been shown to improve generation quality in several works [3, 14, 16, 34, 20, 23].

Original: The use of Neural ODEs in our approach is inspired by their recent application to a variety of domains.
Revised: The use of Neural ODEs in our approach is inspired by their recent application to a variety of domains.

Original: On account of their continuous-time formalism, Neural ODEs, unlike traditional discrete-timesequence models, are capable of handling non-uniformly sampled temporal data with no additionaloverhead.
Revised: On account of their continuous-time formalism, Neural ODEs, unlike traditional discrete-time sequence models, are capable of handling non-uniformly sampled temporal data with no additional overhead.

Original: As a result, they are inherently suited to applications, such as time-series forecasting [5, 32]and deep generative models of continuous-time data [5, 39], moreover, due to their smoothness andinvertibility properties [40], Neural ODEs also ﬁnd application in density estimation and variationalinference as Continuous Normalizing Flows.
Revised: As a result, they are inherently suited to applications, such as time-series forecasting [5, 32] and deep generative models of continuous-time data [5, 39], moreover, due to their smoothness and invertibility properties [40], Neural ODEs also ﬁnd application in density estimation and variational inference as Continuous Normalizing Flows.

Original: B Background
Revised: B Background

Original: Since our approach is based on neural ODEs, in this section, we brieﬂy review neural ODEs andsecond-order neural ODEs, and approaches to solve these.
Revised: Since our approach is based on neural ODEs, in this section, we brieﬂy review neural ODEs and second-order neural ODEs, and approaches to solve these.

Original: Neural ODEs Various deep learning architectures, such as Residual Networks, RNNs and Normalizing Flows can be be formulated as a discrete sequence of additive transformations on a state variable z t
Revised: Neural ODEs Various deep learning architectures, such as Residual Networks, RNNs and Normalizing Flows can be be formulated as a discrete sequence of additive transformations on a state variable z t

Original: [Equation]
Revised: [Equation]

Original: where the transition function f θ is modelled by a neural network.
Revised: where the transition function f θ is modelled by a neural network.

Original: The above operation can be identiﬁed as a unit time-step Euler discretization of a continuous-time system.
Revised: The above operation can be identiﬁed as a unit time-step Euler discretization of a continuous-time system.

Original: Hence, taking the continuous limit of the additive transition, we obtain a First-Order ODE system for z t
Revised: Hence, taking the continuous limit of the additive transition, we obtain a First-Order ODE system for z t

Original: [Equation]
Revised: [Equation]

Original: The neural network f θ , which was previously the discrete-time transition function, now becomes the vector ﬁeld for the First-Order ODE governing the time evolution of the state z t .
Revised: The neural network f θ , which was previously the discrete-time transition function, now becomes the vector ﬁeld for the First-Order ODE governing the time evolution of the state z t .

Original: This framework is known as Neural Ordinary Differential Equations [5].
Revised: This framework is known as Neural Ordinary Differential Equations [5].

Original: The value of the state z t at any given time, as a function of the input or initial state z 0 , is obtained by solving the Initial Value Problem (IVP)
Revised: The value of the state z t at any given time, as a function of the input or initial state z 0 , is obtained by solving the Initial Value Problem (IVP)

Original: [Equation]
Revised: [Equation]

Original: Input: Source Image x S , Target Image x T and Integration time T

Original: Output: Continuous-time Interpolation Curve ˜ x t : [0 , T ] −→ X 1 Set initial latent z 0 of latent space trajectory z 0 :

Original: = E ( x S ) 2 Sample Initial Velocity : (cid:15) ∼ N (0 , I ) v 0 = µ v ( z 0 , E ( x T ))

Original: + (cid:15) (cid:12) σ v ( z 0 , E ( x T )) 3 Solve ODE System for for z t : [0 , T ] −→ X : d 2 z t [Equation] 4 Generate Interpolation Curve ˜ x t : [0 , T ] −→ X ˜ x t = G ( z t )
Revised: [Table]

Original: While exact solution is infeasible in most cases, the IVP can be approximately solved with highaccuracy using Numerical ODE solvers such as Runge Kutta (RK4) and Dormand Price (DOPRI5).
Revised: While exact solution is infeasible in most cases, the IVP can be approximately solved with high accuracy using Numerical ODE solvers such as Runge Kutta (RK4) and Dormand Price (DOPRI5).

Original: Gradients can either be obtained by ordinary backpropagation or by using the Adjoint State Method,[5] which allows gradient computation without backpropagating through ODE solver operations.
Revised: Gradients can either be obtained by ordinary backpropagation or by using the Adjoint State Method, [5] which allows gradient computation without backpropagating through ODE solver operations.

Original: Second-Order Neural ODEs Despite the impressive continuous-time modeling capabilities of (FirstOrder) Neural ODEs, there exist various classes of phenomena (e.g. Harmonic and Van der Pol oscillators) whose latent dynamics cannot be modeled by First-Order ODE systems.
Revised: Second-Order Neural ODEs Despite the impressive continuous-time modeling capabilities of (FirstOrder) Neural ODEs, there exist various classes of phenomena (e.g. Harmonic and Van der Pol oscillators) whose latent dynamics cannot be modeled by First-Order ODE systems.

Original: This motivates the use of Second-Order Neural ODEs [39], a framework where the time evolution of the state variable z t is governed by
Revised: This motivates the use of Second-Order Neural ODEs [39], a framework where the time evolution of the state variable z t is governed by

Original: [Equation]
Revised: [Equation]

Original: Analogous to First-Order Neural ODEs, the vector ﬁeld f θ ( z t , ˙ z t ) is modelled by a Neural Network.
Revised: Analogous to First-Order Neural ODEs, the vector ﬁeld f θ ( z t , ˙ z t ) is modelled by a Neural Network.

Original: However, a key difference lies in the fact that the vector ﬁeld is a function of the state z t as wellas the state differential ˙
Revised: However, a key difference lies in the fact that the vector ﬁeld is a function of the state z t as well as the state differential ˙

Original: z t =
Revised: z t =

Original: d z t dt .
Revised: d z t dt .

Original: This feature allows Second-Order Neural ODEs to model muchmore complex dynamical systems that cannot be modeled by First-Order Neural ODEs.
Revised: This feature allows Second-Order Neural ODEs to model much more complex dynamical systems that cannot be modeled by First-Order Neural ODEs.

Original: Moreover,
Revised: Moreover,

Original: Second-Order Neural ODEs have much better smoothness properties as they ensure the continuity ofthe second derivative of state d 2 z t dt[-
-]
Revised: Second-Order Neural ODEs have much better smoothness properties as they ensure the continuity of the second derivative of state d 2 z t dt[-
-]

Original: To facilitate numerical integration, the Second-Order Neural ODE is reduced to an equivalent Coupled
Revised: To facilitate numerical integration, the Second-Order Neural ODE is reduced to an equivalent Coupled

Original: First-Order ODE system by introducing an auxiliary state variable v t (often named velocity) asfollows.
Revised: First-Order ODE system by introducing an auxiliary state variable v t (often named velocity) as follows.

Original: [Equation]
Revised: [Equation]

Original: This ODE system can be interpreted as a First-Order Neural ODE for the augmented state [ z t , v t ] T .
Revised: This ODE system can be interpreted as a First-Order Neural ODE for the augmented state [ z t , v t ] T .

Original: Consequently, it can be transformed into an Initial Value Problem (IVP)
Revised: Consequently, it can be transformed into an Initial Value Problem (IVP)

Original: [Equation]
Revised: [Equation]

Original: which, as described in Section 3.1, can be solved by Numerical ODE solvers.
Revised: which, as described in Section 3.1, can be solved by Numerical ODE solvers.

Original: As discussed earlier, gradient computation can be performed either by backpropagating through the ODE solver’soperations or by using the Adjoint State Method [5].
Revised: As discussed earlier, gradient computation can be performed either by backpropagating through the ODE solver’s operations or by using the Adjoint State Method [5].

Original: C Additional Experiments
Revised: C Additional Experiments

Original: Adapting to Unseen Supports Since our approach models distribution over trajectories conditionedon source and target images, the generated data distribution can be ﬂexibly varied by modifying thedistribution of source and target images.

Original: This allows the trained model to improve the generateddata’s diversity without retraining the parameters by incorporating additional data into the set ofsource and target images.

Original: This is unlike the models based on ﬁxed parametric priors, which requireretraining on new data to modify the generated data distribution.

Original: We repeat the evaluation by varyingthe support across 3 different distributions, namely the Train Set, the Test Set, as well as the Train and Test set combined.

Original: As demonstrated through the results in Table 3, utilizing additional data fromthe test set leads to improvement in FID scores.

Original: Our model’s ability to interpolate on test data alsodemonstrates its ability to model the entire image data manifold rather than overﬁtting on trainingimages.

Original: [Table] Improved Representation Learning We evaluate the representation learning capabilities of NeurIntand our baselines by training a linear SVM model on the feature vectors obtained by concatenatingthe output layer and the last hidden layer of the encoder for 60,000 class balanced labeled imagesfrom the training set of the SVHN dataset.
Revised: Improved Representation Learning We evaluate the representation learning capabilities of NeurInt and our baselines by training a linear SVM model on the feature vectors obtained by concatenating the output layer and the last hidden layer of the encoder for 60,000 class balanced labeled images from the training set of the SVHN dataset.

Original: We hold out 10,000 labeled images from the trainingset as a validation set to tune hyper-parameters of the SVM model.
Revised: We hold out 10,000 labeled images from the training set as a validation set to tune hyper-parameters of the SVM model.

Original: We report the average testmisclassiﬁcation error for 10 different SVM models trained on different random 60,000 exampletraining sets.
Revised: We report the average test misclassiﬁcation error for 10 different SVM models trained on different random 60,000 example training sets.

Original: Our results are reported in Table 4.
Revised: Our results are reported in Table 4.

Original: Learning a Distribution of Trajectories To assess the diversity of the interpolation trajectoriesresulting from the distribution modeled by NeurInt, we generate different trajectories by ﬁxing asource-target pair, drawing multiple samples of v 0 conditioned on this ﬁxed source-target pair andgenerating the corresponding interpolation trajectories.
Revised: Learning a Distribution of Trajectories To assess the diversity of the interpolation trajectories resulting from the distribution modeled by NeurInt, we generate different trajectories by ﬁxing a source-target pair, drawing multiple samples of v 0 conditioned on this ﬁxed source-target pair and generating the corresponding interpolation trajectories.

Original: As observed in Rows A 1 and A 2 of Figure 3,the sampled trajectories show noticeable variation in the intermediate interpolants but successfullyconverge to the same target, as desired.
Revised: As observed in Rows A 1 and A 2 of Figure 3, the sampled trajectories show noticeable variation in the intermediate interpolants but successfully converge to the same target, as desired.

Original: To further emphasise on this variation, we repeat the processby using the same source as Rows A 1 and A 2 but using a different target image as shown in Rows A 3and A 4 .
Revised: To further emphasise on this variation, we repeat the process by using the same source as Rows A 1 and A 2 but using a different target image as shown in Rows A and A 4 .

Original: To visualize these variations in the encoding space, we plot the ﬁrst Principal Componentof each row of Figure 3 over time.
Revised: To visualize these variations in the encoding space, we plot the ﬁrst Principal Component of each row of Figure 3 over time.

Original: It is observed in the plot (Figure 3) that the PCA [11] curvesof trajectories A 1 and A 2 , starting from the same source, deviate from one another in the middle,thereby reﬂecting the variety of intermediate interpolants, and towards the end, converge very closeto each other, in the neighborhood of the target.
Revised: It is observed in the plot (Figure 3) that the PCA [11] curves of trajectories A 1 and A 2 , starting from the same source, deviate from one another in the middle, thereby reﬂecting the variety of intermediate interpolants, and towards the end, converge very close to each other, in the neighborhood of the target.

Original: The same phenomenon is observed for A 3 and
Revised: The same phenomenon is observed for A 3 and

Original: A 4 , whose PCA curves deviate signiﬁcantly from A 1 and A 2 , on account of having a differenttarget.
Revised: A 4 , whose PCA curves deviate signiﬁcantly from A 1 and A 2 , on account of having a different target.

Original: Furthermore, the PCA curves’ curvature and smoothness conﬁrm that NeurInt truly captures adistribution of smooth and non-linear interpolation trajectories.
Revised: Furthermore, the PCA curves’ curvature and smoothness conﬁrm that NeurInt truly captures a distribution of smooth and non-linear interpolation trajectories.

Original: D Architecture and Setup
Revised: D Architecture and Setup

Original: For our baselines and the proposed model, we borrow the architecture for generator and discriminatorfrom PGAN [19].
Revised: For our baselines and the proposed model, we borrow the architecture for generator and discriminator from PGAN [19].

Original: For ALI, following [10], we use two networks D X and D Z to extract featuresfrom a given image X and latent vector Z respectively which are subsequently concatenated andpassed through a joint network D X , Z to obtain D ( X, Z ) .
Revised: For ALI, following [10], we use two networks D X and D Z to extract features from a given image X and latent vector Z respectively which are subsequently concatenated and passed through a joint network D X , Z to obtain D ( X, Z ) .

Original: To ensure fairness, all models weretrained progressively.
Revised: To ensure fairness, all models were trained progressively.

Original: The encoder E ’s architecture uses the same layers as the ﬁrst 12 layers of thediscriminator architecture’s D [Table]
Revised: The encoder E ’s architecture uses the same layers as the ﬁrst 12 layers of the discriminator architecture’s D X component.

Original: For NeurInt, the position Encoder E and velocity encoder V = ( µ v , σ v ) are both one hidden layer
Revised: For NeurInt, the position Encoder E and velocity encoder V = ( µ v , σ v ) are both one hidden layer

Original: MLPs with a LeakyReLU nonlinearity, and the the vector ﬁeld f of the Second Neural ODE is a 2layer MLP with a tanh nonlinearity.
Revised: MLPs with a LeakyReLU nonlinearity, and the the vector ﬁeld f of the Second Neural ODE is alayer MLP with a tanh nonlinearity.

Original: The relative weighting hyperparameter λ in the loss for NeurIntwas decayed linearly from 1000 to 100 per half cycle of each progressive step and then kept stable at100 for the next half-cycle of the step.
Revised: The relative weighting hyperparameter λ in the loss for NeurInt was decayed linearly from 1000 to 100 per half cycle of each progressive step and then kept stable at 100 for the next half-cycle of the step.

Original: To maintain consistency, we also trained the encoder E P for PGAN by growing it progressively.
Revised: To maintain consistency, we also trained the encoder E P for PGAN by growing it progressively.

Original: We obtained an RMSE error of 0.0522 on the held-out test set upon training.
Revised: We obtained an RMSE error of 0.0522 on the held-out test set upon training.

Original: Figure 4 shows somereconstructions on held-out test data which qualitatively indicate the convergence of the encoder.
Revised: Figure 4 shows some reconstructions on held-out test data which qualitatively indicate the convergence of the encoder.

Original: We use the same optimizer for PGAN and Neurint as proposed in [19] and for ALI as in [10].
Revised: We use the same optimizer for PGAN and Neurint as proposed in [19] and for ALI as in [10].

Original: For all our experiments, we use 2 Nvidia GeForce GTX 1080 Ti GPUs. 
Revised: For all our experiments, we use 2 Nvidia GeForce GTX 1080 Ti GPUs. [Table]

Original: E FID Statistics
Revised: E FID Statistics

Original: Using the training set as the support, we compute the FID scores of NeurInt and our baselines (usingthe same procedure as described in the main paper), over 100 randomized runs, and report the meanand standard deviation of the FID scores obtained in Table 5.
Revised: Using the training set as the support [Table] we compute the FID scores of NeurInt and our baselines (using the same procedure as described in the main paper) [Table]

Revised: over 100 randomized runs

Revised: [Table]

Revised: and report the mean and standard deviation of the FID scores obtained in Table

Revised: [Table]

Original: F Additional Samples
Revised: F Additional Samples

Original: Interpolation Samples : Figures 5 and 6 qualitatively demonstrate NeurInt’s ability to generatesigniﬁcantly more realistic and smoother interpolation trajectories over the baselines.
Revised: Interpolation Samples : Figures 5 and 6 qualitatively demonstrate NeurInt’s ability to generate signiﬁcantly more realistic and smoother interpolation trajectories over the baselines.

Original: [Table]

Original: Uncurated Samples : Figure 7 shows uncurated samples from NeurInt and baselines.
Revised: Uncurated Samples : Figure 7 shows uncurated samples from NeurInt and baselines.

Original: The qualityof the generated samples of NeurInt against the baselines backs up the superior FID achieved by
Revised: The quality of the generated samples of NeurInt against the baselines backs up the superior FID achieved by

Original: NeurInt.
Revised: NeurInt.

Original: This demonstrates NeurInt to be not only a good interpolation methodology but also a goodgenerative model.
Revised: This demonstrates NeurInt to be not only a good interpolation methodology but also a good generative model.

Original: Distribution of trajectories :
Revised: Distribution of trajectories :

Original: Figure 8 demonstrates qualitatively NeurInt’s ability to drawinterpolations from a distribution of interpolation trajectories.
Revised: Figure 8 demonstrates qualitatively NeurInt’s ability to draw interpolations from a distribution of interpolation trajectories.

Original: Particularly note the transition betweena female source and male target with different facial attributes.
Revised: Particularly note the transition between a female source and male target with different facial attributes.

Original: Ablation Study : Figure 9 demonstrates the beneﬁts of our joint training mechanism, with NeurIntinterpolation trajectory being of signiﬁcantly better quality than NeurInt-PT.
Revised: Ablation Study : Figure 9 demonstrates the beneﬁts of our joint training mechanism, with NeurInt interpolation trajectory being of signiﬁcantly better quality than NeurInt-PT.