Original: U NDERSTANDING AND I MPROVING I NTERPOLATION IN A UTOENCODERS VIA AN A DVERSARIAL R EGULARIZER
Revised: U NDERSTANDING AND I MPROVING I NTERPOLATION IN A UTOENCODERS VIA AN A DVERSARIAL R EGULARIZER

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Autoencoders provide a powerful framework for learning compressed representations by encoding all of the information needed to reconstruct a data point in a latent code.
Revised: Autoencoders provide a powerful framework for learning compressed representations by encoding all of the information needed to reconstruct a data point in a latent code.

Original: In some cases, autoencoders can “interpolate”:
Revised: In some cases, autoencoders can “interpolate”:

Original: By decoding the convex combination of the latent codes for two datapoints, the autoencoder can produce an output which semantically mixes characteristics from the datapoints.
Revised: By decoding the convex combination of the latent codes for two datapoints, the autoencoder can produce an output which semantically mixes characteristics from the datapoints.

Original: In this paper, we propose a regularization procedure which encourages interpolated outputs to appear more realistic by fooling a critic network which has been trained to recover the mixing coefﬁcient from interpolated data.
Revised: In this paper, we propose a regularization procedure which encourages interpolated outputs to appear more realistic by fooling a critic network which has been trained to recover the mixing coefﬁcient from interpolated data.

Original: We then develop a simple benchmark task where we can quantitatively measure the extent to which various autoencoders can interpolate and show that our regularizer dramatically improves interpolation in this setting.
Revised: We then develop a simple benchmark task where we can quantitatively measure the extent to which various autoencoders can interpolate and show that our regularizer dramatically improves interpolation in this setting.

Original: We also demonstrate empirically that our regularizer produces latent codes which are more effective on downstream tasks, suggesting a possible link between interpolation abilities and learning useful representations
Revised: We also demonstrate empirically that our regularizer produces latent codes which are more effective on downstream tasks, suggesting a possible link between interpolation abilities and learning useful representations.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: One goal of unsupervised learning is to uncover the underlying structure of a dataset without using explicit labels.
Revised: One goal of unsupervised learning is to uncover the underlying structure of a dataset without using explicit labels.

Original: A common architecture used for this purpose is the autoencoder , which learns to map datapoints to a latent code from which the data can be recovered with minimal information loss.
Revised: A common architecture used for this purpose is the autoencoder , which learns to map datapoints to a latent code from which the data can be recovered with minimal information loss.

Original: Typically, the latent code is lower dimensional than the data, which indicates that autoencoders can perform some form of dimensionality reduction.
Revised: Typically, the latent code is lower dimensional than the data, which indicates that autoencoders can perform some form of dimensionality reduction.

Original: For certain architectures, the latent codes have been shown to disentangle important factors of variation in the dataset which makes such models useful for representation learning (Chen et al., 2016a;
Revised: For certain architectures, the latent codes have been shown to disentangle important factors of variation in the dataset which makes such models useful for representation learning (Chen et al., 2016a;

Original: Higgins et al., 2016).
Revised: Higgins et al., 2017).

Original: In the past, they were also used for pre-training other networks by being trained on unlabeled data and then being stacked to initialize a deep network (Bengio et al., 2007; Vincent et al., 2010).
Revised: In the past, they were also used for pre-training other networks by being trained on unlabeled data and then being stacked to initialize a deep network (Bengio et al., 2007; Vincent et al., 2010).

Original: More recently, it was shown that imposing a prior on the latent space allows autoencoders to be used for probabilistic or generative modeling (Kingma & Welling, 2014; Rezende et al., 2014; Makhzani et al., 2015).
Revised: More recently, it was shown that imposing a prior on the latent space allows autoencoders to be used for probabilistic or generative modeling (Kingma & Welling, 2014; Rezende et al., 2014; Makhzani et al., 2015).

Original: In some cases, autoencoders have shown the ability to interpolate .
Revised: In some cases, autoencoders have shown the ability to interpolate .

Original: Speciﬁcally, by mixing codes in latent space and decoding the result, the autoencoder can produce a semantically meaningful combination of the corresponding datapoints.
Revised: Speciﬁcally, by mixing codes in latent space and decoding the result, the autoencoder can produce a semantically meaningful combination of the corresponding datapoints.

Original: Interpolation has been frequently reported as a qualitative experimental result in studies about autoencoders (Dumoulin et al., 2016;
Revised: Interpolation has been frequently reported as a qualitative experimental result in studies about autoencoders (Dumoulin et al., 2016;

Original: Bowman et al., 2015;
Revised: Bowman et al., 2015;

Original: Roberts et al., 2018; Mescheder et al., 2017; Mathieu et al., 2016; Ha & Eck, 2018) and latent-variable generative models in general (Dinh et al., 2016;
Revised: Roberts et al., 2018; Mescheder et al., 2017; Mathieu et al., 2016; Ha & Eck, 2018) and latent-variable generative models in general (Dinh et al., 2016;

Original: Radford et al., 2015; van den Oord et al., 2016).
Revised: Radford et al., 2015; van den Oord et al., 2016).

Original: The ability to interpolate can be useful in its own right e.g. for creative applications (Carter & Nielsen, 2017).
Revised: The ability to interpolate can be useful in its own right e.g. for creative applications (Carter & Nielsen, 2017).

Original: However, it also indicates that the autoencoder can “extrapolate” beyond the training data and has learned a latent space with a particular structure.
Revised: However, it also indicates that the autoencoder can “extrapolate” beyond the training data and has learned a latent space with a particular structure.

Original: Speciﬁcally, if interpolating between two points in latent space produces a smooth semantic warping in data space, this suggests that nearby points in latent space are semantically similar.
Revised: Speciﬁcally, if interpolating between two points in latent space produces a smooth semantic warping in data space, this suggests that nearby points in latent space are semantically similar.

Revised: A visualization of this idea is shown in ﬁg. 1, where a smooth interpolation between a “2” and a “9” suggests that the 2 is surrounded by semantically similar points, i.e. other 2s.

Original: This  may suggest that an autoencoder which interpolates well could also provide a good learned representation for downstream tasks because similar points are clustered.
Revised: This property may suggest that an autoencoder which interpolates well could also provide a good learned representation for downstream tasks because similar points are clustered.

Original: A visualization of this idea is shown inﬁg. 1, where a smooth interpolation between a “2” and a “9” suggests that the 2 is surrounded by semantically similar points, i.e. other 2s.
Revised: If the interpolation is not smooth, there may be “discontinuities” in latent space which could result in the representation being less useful as a learned feature.

Original: This connection between interpolation and a “ﬂat” data manifold has been explored in the context of unsupervised representation learning (Bengio et al., 2013b) and regularization (Verma et al., 2018).
Revised: This connection between interpolation and a “ﬂat” data manifold has been explored in the context of unsupervised representation learning (Bengio et al., 2013b) and regularization (Verma et al., 2018).

Original: Given the widespread use of interpolation as a qualitative measure of autoencoder performance, we believe additional investigation into the connection between interpolation and representation learning suggests that semantically similar points may be clustered together in latent space.
Revised: suggests that semantically similar points may be clustered together in latent space.

Original: [Figure]
Revised: [Figure]

Original: A critic network tries to predict the interpolation coefﬁcient α corresponding to an interpolated datapoint.
Revised: A critic network tries to predict the interpolation coefﬁcient α corre sponding to an interpolated datapoint.

Original: The autoencoder is trained to fool the critic into outputting α = 0 .
Revised: The autoencoder is trained to fool the critic into outputting α = 0 .

Original:  is warranted.
Revised: Given the widespread use of interpolation as a qualitative measure of autoencoder performance, we believe additional investigation into the connection between interpolation and representation learning is warranted.

Original: Our goal in this paper is threefold: First, we introduce a regularization strategy with the speciﬁc goal of encouraging improved interpolations in autoencoders (section 2); second, we develop a synthetic benchmark where the slippery concept of a “semantically meaningful interpolation” is quantitatively measurable (section 3.1) and evaluate common autoencoders on this task (section 3.2); and third, we conﬁrm the intuition that good interpolation can result in a useful representation by showing that the improved interpolation ability produced by our regularizer elicits improved representation learning performance on downstream tasks (section 4).
Revised: Our goal in this paper is threefold: First, we introduce a regularization strategy with the speciﬁc goal of encouraging improved interpolations in autoencoders (section 2); second, we develop a synthetic benchmark where the slippery concept of a “semantically meaningful interpolation” is quantitatively measurable (section 3.1) and evaluate common autoencoders on this task (section 3.2); and third, we conﬁrm the intuition that good interpolation can result in a useful representation by showing that the improved interpolation ability produced by our regularizer elicits improved representation learning performance on downstream tasks (section 4).

Original: We also make our codebase available 1 which provides a uniﬁed implementation of many common autoencoders including our proposed regularizer.
Revised: We also make our codebase available 1 which provides a uniﬁed implementation of many common autoencoders including our proposed regularizer.

Original:  A N A DVERSARIAL R EGULARIZER FOR I MPROVING I NTERPOLATIONS
Revised:  A N A DVERSARIAL R EGULARIZER FOR I MPROVING I NTERPOLATIONS

Original: Autoencoders, also called auto-associators (Bourlard & Kamp, 1988), consist of the following structure: First, an input x ∈ R d x is passed through an “encoder” z = f θ ( x ) parametrized by θ to obtain a latent code z ∈ R d z .
Revised: Autoencoders, also called auto-associators (Bourlard & Kamp, 1988), consist of the following structure: First, an input x ∈ R d x is passed through an “encoder” z = f θ ( x ) parametrized by θ to obtain a latent code z ∈ R d z .

Original: The latent code is then passed through a “decoder” ˆ x = g φ ( z ) parametrized by φ to produce an approximate reconstruction ˆ x ∈ R
Revised: The latent code is then passed through a “decoder” ˆ x = g φ ( z ) parametrized by φ to produce an approximate reconstruction ˆ x ∈ R

Original: d
Revised: d

Original: x of the input x .
Revised: x of the input x .

Original: We consider the case where f θ and g φ are implemented as multi-layer neural networks.
Revised: We consider the case where f θ and g φ are implemented as multi-layer neural networks.

Original: The encoder and decoder are trained simultaneously (i.e. with respect to θ and φ ) to minimize some notion of distance between the input x and the output ˆ x , for example the squared L 2 distance (cid:107) x − ˆ x (cid:107) 2 .
Revised: The encoder and decoder are trained simultaneously (i.e. with respect to θ and φ ) to minimize some notion of distance between the input x and the output ˆ x , for example the squared L 2 distance (cid:107) x − ˆ x (cid:107) 2 .

Original: Interpolating using an autoencoder describes the process of using the decoder g φ to decode a mixture of two latent codes.
Revised: Interpolating using an autoencoder describes the process of using the decoder g φ to decode a mixture of two latent codes.

Original: Typically, the latent codes are combined via a convex combination, so that interpolation amounts to computing ˆ x α = g φ ( αz 1 +(1 − α )
Revised: Typically, the latent codes are combined via a convex combination, so that interpolation amounts to computing ˆ x α = g φ ( αz 1 +(1 − α )

Original: z 2 ) for some α ∈ [0 , 1] where z 1 = f θ ( x 1 ) and z 2 = f θ ( x 2 ) are the latent codes corresponding to data points x 1 and x 2 .
Revised: z 2 ) for some α ∈ [0 , 1] where z 1 = f θ ( x 1 ) and z 2 = f θ ( x 2 ) are the latent codes corresponding to data points x 1 and x 2 .

Original: We also experimented with spherical interpolation which has been used in settings where the latent codes are expected to have spherical structure (Husz´ar, 2017; White, 2016; Roberts et al., 2018), but found it made no discernible difference in practice for any autoencoder we studied.
Revised: We also experimented with spherical interpolation which has been used in settings where the latent codes are expected to have spherical structure (Husz´ar, 2017; White, 2016; Roberts et al., 2018), but found it made no discernible difference in practice for any autoencoder we studied.

Original: Ideally, adjusting α from 0 to 1 will produce a sequence of realistic datapoints where each subsequent ˆ x α is progressively less semantically similar to x 1 and more semantically similar to x 2 .
Revised: Ideally, adjusting α from 0 to 1 will produce a sequence of realistic datapoints where each subsequent ˆ x α is progressively less semantically similar to x 1 and more semantically similar to x 2 .

Original: The notion of “semantic similarity” is problem-dependent and ill-deﬁned; we discuss this further in section 3.
Revised: The notion of “semantic similarity” is problem-dependent and ill-deﬁned; we discuss this further in section 3.

Original: 2. A DVERSARIALLY C ONSTRAINED A UTOENCODER I NTERPOLATION (ACAI)
Revised: 2. A DVERSARIALLY C ONSTRAINED A UTOENCODER I NTERPOLATION (ACAI)

Original: As mentioned above, a high-quality interpolation should have two characteristics:
Revised: As mentioned above, a high-quality interpolation should have two characteristics:

Original: First, that intermediate points along the interpolation are indistinguishable from real data; and second, that the intermediate points provide a semantically smooth morphing between the endpoints.
Revised: First, that intermediate points along the interpolation are indistinguishable from real data; and second, that the intermediate points provide a semantically smooth morphing between the endpoints.

Original: The latter characteristic is hard to enforce because it requires deﬁning a notion of semantic similarity for a given dataset, which is often hard to explicitly codify.
Revised: The latter characteristic is hard to enforce because it requires deﬁning a notion of semantic similarity for a given dataset, which is often hard to explicitly codify.

Original: So instead, we propose a regularizer which encourages interpolated datapoints to appear realistic, or more speciﬁcally, to appear indistinguishable from reconstructions of real datapoints.
Revised: So instead, we propose a regularizer which encourages interpolated datapoints to appear realistic, or more speciﬁcally, to appear indistinguishable from reconstructions of real datapoints.

Original: We ﬁnd empirically that this constraint results in realistic and smooth interpolations in practice (section 3.1) in addition to providing improved performance on downstream tasks (section 4).
Revised: We ﬁnd empirically that this constraint results in realistic and smooth interpolations in practice (section 3.1) in addition to providing improved performance on downstream tasks (section 4).

Original: /anonymous-iclr-2019/acai-iclr-

Original: To enforce this constraint we introduce a critic network (Goodfellow et al., 2014) which is fed interpolations of existing datapoints (i.e. ˆ x α as deﬁned above).
Revised: To enforce this constraint we introduce a critic network (Goodfellow et al., 2014) which is fed interpolations of existing datapoints (i.e. ˆ x α as deﬁned above).

Original: Its goal is to predict α from ˆ x α , i.e. to predict the mixing coefﬁcient used to generate its input.
Revised: Its goal is to predict α from ˆ x α , i.e. to predict the mixing coefﬁcient used to generate its input.

Original: When training the model, for each pair of training data points we randomly sample a value of α to produce ˆ x α .
Revised: When training the model, for each pair of training data points we randomly sample a value of α to produce ˆ x α .

Original: In order to resolve the ambiguity between predicting α and 1 − α , we constrain α to the range [0 , 0 .
Revised: In order to resolve the ambiguity between predicting α and 1 − α , we constrain α to the range [0 , 0 .

Original: 5] when feeding ˆ x α to the critic.
Revised: 5] when feeding ˆ x α to the critic.

Original: In contrast, the autoencoder is trained to fool the critic to think that α is always zero.
Revised: In contrast, the autoencoder is trained to fool the critic to think that α is always zero.

Original: This is achieved by adding an additional term to the autoencoder’s loss to optimize its parameters to fool the critic.
Revised: This is achieved by adding an additional term to the autoencoder’s loss to optimize its parameters to fool the critic.

Revised: In a loose sense, the critic can be seen as approximating an “adversarial divergence” (Liu et al., 2017;

Revised: Arora et al., 2017) between reconstructions and interpolants which the autoencoder tries to minimize.

Original: Formally, let d ω ( x ) be the critic network, which for a given input produces a scalar value.
Revised: Formally, let d ω ( x ) be the critic network, which for a given input produces a scalar value.

Original: The critic is trained to minimize
Revised: The critic is trained to minimize

Original: [Equation]
Revised: [Equation]

Original: where, as above, ˆ x α = g φ ( αf θ ( x 1 ) + (1 − α )
Revised: where, as above, ˆ x α = g φ ( αf θ ( x 1 ) + (1 − α )

Original: f θ ( x 2 )) , ˆ x = g φ ( f θ ( x )) , and γ is a scalar hyperparameter.
Revised: f θ ( x 2 )) , ˆ x = g φ ( f θ ( x )) for some x (not necessarily x 1 or x 2 ), and γ is a scalar hyperparameter.

Original: The ﬁrst term trains the critic to recover α from ˆ x α .
Revised: The ﬁrst term trains the critic to recover α from ˆ x α .

Original: The second term serves as a regularizer with two functions: First, it enforces that the critic consistently outputs 0 for non-interpolated inputs; and second, by interpolating between x and ˆ x  in data space it ensures the critic is exposed to realistic data even when the autoencoder’s
Revised: The second term serves as a regularizer with two functions: First, it enforces that the critic consistently outputs 0 for non-interpolated inputs; and second, by interpolating between x and ˆ x (the autoencoder’s reconstruction of x ) in data space it ensures the critic is exposed to realistic data even when the autoencoder’s

Original: reconstructions are poor.
Revised: reconstructions are poor.

Original: We found the second term was not crucial for our approach, but helped stabilize the adversarial learning process.
Revised: We found the second term was not crucial for our approach, but helped stabilize the convergence of the autoencoder and allowed us to use consistent hyperparameters and architectures across all datasets and experiments.

Original: The autoencoder’s loss function is modiﬁed by adding a regularization term:
Revised: The autoencoder’s loss function is modiﬁed by adding a regularization term:

Original: [Equation]
Revised: [Equation]

Original: where λ is a scalar hyperparameter which controls the weight of the regularization term.
Revised: where λ is a scalar hyperparameter which controls the weight of the regularization term.

Original: Note that the regularization term is effectively trying to make the critic output 0 regardless of the value of α , thereby “fooling” the critic into thinking that an interpolated input is non-interpolated (i.e., having α = 0 ).
Revised: Note that the regularization term is effectively trying to make the critic output 0 regardless of the value of α , thereby “fooling” the critic into thinking that an interpolated input is non-interpolated (i.e., having α = 0 ).

Original: The parameters θ and φ are optimized with respect to L f,g (which gives the autoencoder access to the critic’s gradients) and ω is optimized with respect to L d .
Revised: The parameters θ and φ are optimized with respect to L f,g (which gives the autoencoder access to the critic’s gradients) and ω is optimized with respect to L d .

Original: We refer to the use of this regularizer as
Revised: We refer to the use of this regularizer as

Original: Adversarially Constrained Autoencoder Interpolation (ACAI).
Revised: Adversarially Constrained Autoencoder Interpolation (ACAI).

Original: A diagram of the ACAI is shown in ﬁg. 2.
Revised: A diagram of the ACAI is shown in ﬁg. 2.

Original: Assuming an effective critic, the autoencoder successfully “wins” this adversarial game by producing interpolated points which are indistinguishable from reconstructed data.
Revised: Assuming an effective critic, the autoencoder successfully “wins” this adversarial game by producing interpolated points which are indistinguishable from reconstructed data.

Original: We ﬁnd in practice that encouraging this behavior also produces semantically smooth interpolations and improved representation learning performance, which we demonstrate in the following sections.
Revised: We ﬁnd in practice that encouraging this behavior also produces semantically smooth interpolations and improved representation learning performance, which we demonstrate in the following sections.

Revised: Our loss function is similar to the one used in the Least Squares Generative Adversarial Network (Mao et al., 2017) in the sense that they both measure the distance between a critic’s output and a scalar using a squared L2 loss.

Revised: However, they are substantially different in that ours is used as a regularizer for autoencoders rather than for generative modeling and our critic attempts to regress the interpolation coefﬁcient α instead of a ﬁxed scalar hyperparameter.

Original: Note that the only thing ACAI encourages is that interpolated points appear realistic.
Revised: that the only thing ACAI encourages is that interpolated points appear realistic.

Original: The critic only ever sees a single reconstruction or interpolant at a time; it is never fed real datapoints or latent vectors.
Revised: The critic only ever sees a single reconstruction or interpolant at a time; it is never fed real datapoints or latent vectors.

Original: It therefore will only be able to successfully recover α if the quality of the autoencoder’s output degrades consistently across an interpolation  (as seen, for example, in ﬁg. 3a), and  ACAI’s primary purpose is to discourage this behavior.
Revised: It therefore will only be able to successfully recover α if the quality of the autoencoder’s output degrades consistently across an interpolation as a function of α (as seen, for example, in ﬁg. 3a where interpolated points become successively blurrier and darker). ACAI’s primary purpose is to discourage this behavior.

Original: In doing so, it may implicitly modify the structure of the latent space learned by the autoencoder, but ACAI itself does not directly impose a speciﬁc structure.
Revised: In doing so, it may implicitly modify the structure of the latent space learned by the autoencoder, but ACAI itself does not directly impose a speciﬁc structure.

Original: Our goal in introducing ACAI is to test whether simply encouraging better interpolation behavior produces a better representation for downstream tasks.
Revised: Our goal in introducing ACAI is to test whether simply encouraging better interpolation behavior produces a better representation for downstream tasks.

Original: Further, in contrast with the standard 
Revised: Further, in contrast with the standard Generative

Original: Generative Adversarial Network setup (Goodfellow et al., 2014), ACAI does not distinguish between “real” and “fake” data; rather, it simply attempts to regress the interpolation coefﬁcient α .
Revised: Adversarial Network setup (Goodfellow et al., 2014), ACAI does not distinguish between “real” and “fake” data; rather, it simply attempts to regress the interpolation coefﬁcient α .

Original:  A UTOENCODERS , AND H OW T HEY I NTERPOLATE
Revised:  A UTOENCODERS , AND H OW T HEY I NTERPOLATE

Original: How can we measure whether an autoencoder interpolates effectively and whether our proposed regularization strategy achieves its stated goal?
Revised: How can we measure whether an autoencoder interpolates effectively and whether our proposed regularization strategy achieves its stated goal?

Original: As mentioned in section 2, deﬁning interpolation relies on the notion of “semantic similarity” which is a vague and problem-dependent concept.
Revised: As mentioned in section 2, deﬁning interpolation relies on the notion of “semantic similarity” which is a vague and problem-dependent concept.

Original: For example, a deﬁnition of interpolation along the lines of “ αz 1 + (1 − α )
Revised: For example, a deﬁnition of interpolation along the lines of “ αz 1 + (1 − α )

Original: z 2 should map to αx 1 + (1 − α ) x 2 ” is overly simplistic because interpolating in “data space” often does not result in realistic datapoints – in images, this corresponds to simply fading between the pixel values of the two images.
Revised: z 2 should map to αx 1 + (1 − α ) x 2 ” is overly simplistic because interpolating in “data space” often does not result in realistic datapoints – in images, this corresponds to simply fading between the pixel values of the two images.

Original: Instead, we might hope that our autoencoder smoothly morphs between salient characteristics of x 1 and x 2 .
Revised: Instead, we might hope that our autoencoder smoothly morphs between salient characteristics of x 1 and x 2 , even when they are dissimilar.

Original: Put another way, we might hope that decoded points along the interpolation smoothly traverse the underlying manifold of the data instead of simply interpolating in data space.
Revised: Put another way, we might hope that decoded points along the interpolation smoothly traverse the underlying manifold of the data instead of simply interpolating in data space.

Original: However, we rarely have access to the underlying data manifold.
Revised: However, we rarely have access to the underlying data manifold.

Original: To make this problem more concrete, we introduce a simple benchmark task where the data manifold is simple and known a priori which makes it possible to quantify interpolation quality.
Revised: To make this problem more concrete, we introduce a simple benchmark task where the data manifold is simple and known a priori which makes it possible to quantify interpolation quality.

Original: We then evaluate the ability of various common autoencoders to interpolate on our benchmark.
Revised: We then evaluate the ability of various common autoencoders to interpolate on our benchmark.

Original: Finally, we test ACAI on our benchmark and show that it exhibits dramatically improved performance and qualitatively superior interpolations.
Revised: Finally, we test ACAI on our benchmark and show that it exhibits dramatically improved performance and qualitatively superior interpolations.

Original: 3. A UTOENCODING L INES
Revised: 3. A UTOENCODING L INES

Original: Given that the concept of interpolation is difﬁcult to pin down, our goal is to deﬁne a task where a “correct” interpolation between two datapoints is unambiguous and well-deﬁned.
Revised: Given that the concept of interpolation is difﬁcult to pin down, our goal is to deﬁne a task where a “correct” interpolation between two datapoints is unambiguous and well-deﬁned.

Original: This will allow us to quantitatively evaluate the extent to which different autoencoders can successfully interpolate.
Revised: This will allow us to quantitatively evaluate the extent to which different autoencoders can successfully interpolate.

Original: Towards this goal, we propose the task of autoencoding 32 × 32 black-and-white images of lines.
Revised: Towards this goal, we propose the task of autoencoding 32 × 32 grayscale images of lines.

Original: We consider 16-pixel-long lines beginning from the center of the image and extending outward at an angle Λ ∈ [0 , 2 π ] (or put another way, lines are radii of the circle circumscribed within the image borders).
Revised: We consider 16-pixel-long lines beginning from the center of the image and extending outward at an angle Λ ∈ [0 , 2 π ] (or put another way, lines are radii of the circle circumscribed within the image borders).

Original: An example of 16 such images is shown in ﬁg. 4a (appendix A.1).
Revised: An example of 16 such images is shown in ﬁg. 4a (appendix A.1).

Original: In this task, the data manifold can be deﬁned entirely by a single variable: Λ .
Revised: In this task, the data manifold can be deﬁned entirely by a single variable: Λ .

Original: We can therefore deﬁne a valid interpolation from x 1 to x 2 as one which smoothly and linearly adjusts Λ from the angle of the line in x 1 to the angle in x 2 .
Revised: We can therefore deﬁne a valid interpolation from x 1 to x 2 as one which smoothly and linearly adjusts Λ from the angle of the line in x 1 to the angle in x 2 .

Original: We further require that the interpolation traverses the shortest path possible along the data manifold.
Revised: We further require that the interpolation traverses the shortest path possible along the data manifold.

Original: We provide some concrete examples of good and bad interpolations, shown and described in appendix A.1.
Revised: We provide some concrete examples of good and bad interpolations, shown and described in appendix A.1.

Original: On any dataset, our desiderata for a successful interpolation are that intermediate points look realistic and provide a semantically meaningful morphing between its endpoints.
Revised: On any dataset, our desiderata for a successful interpolation are that intermediate points look realistic and provide a semantically meaningful morphing between its endpoints.

Original: On this synthetic lines dataset, we can formalize these notions as speciﬁc evaluation metrics, which we describe in detail in appendix A.2.
Revised: On this synthetic lines dataset, we can formalize these notions as speciﬁc evaluation metrics, which we describe in detail in appendix A.2.

Original: To summarize, we propose two metrics: Mean Distance and Smoothness.
Revised: To summarize, we propose two metrics: Mean Distance and Smoothness.

Original: Mean Distance measures the average distance between interpolated points and “real” datapoints.
Revised: Mean Distance measures the average distance between interpolated points and “real” datapoints.

Original: Smoothness measures whether the angles of the interpolated lines follow a linear trajectory between the angle of the start and endpoint.
Revised: Smoothness measures whether the angles of the interpolated lines follow a linear trajectory between the angle of the start and endpoint.

Original: Both of these metrics are simple to deﬁne due to our construction of a dataset where we exactly know the data distribution and manifold; we provide a full deﬁnition and justiﬁcation in appendix A.2.
Revised: Both of these metrics are simple to deﬁne due to our construction of a dataset where we exactly know the data distribution and manifold; we provide a full deﬁnition and justiﬁcation in appendix A.2.

Original: A perfect alignment would achieve 0 for both scores; larger values indicate a failure to generate realistic interpolated points or produce a smooth interpolation respectively.
Revised: A perfect alignment would achieve 0 for both scores; larger values indicate a failure to generate realistic interpolated points or produce a smooth interpolation respectively.

Original: By choosing a synthetic benchmark where we can explicitly measure the quality of an interpolation, we can conﬁdently evaluate different autoencoders on their interpolation abilities.
Revised: By choosing a synthetic benchmark where we can explicitly measure the quality of an interpolation, we can conﬁdently evaluate different autoencoders on their interpolation abilities.

Original: To evaluate an autoencoder on the synthetic lines task, we randomly sample line images during training and compute our evaluation metrics on a separate randomly-sampled test set of images.
Revised: To evaluate an autoencoder on the synthetic lines task, we randomly sample line images during training and compute our evaluation metrics on a separate randomly-sampled test set of images.

Original: Note that we never train any autoencoder explicitly to produce an optimal interpolation; “good” interpolation is an emergent property which occurs only when the architecture, loss function, training procedure, etc. produce a suitable latent space.
Revised: Note that we never train any autoencoder explicitly to produce an optimal interpolation; “good” interpolation is an emergent property which occurs only when the architecture, loss function, training procedure, etc. produce a suitable latent space.

Original: 3. A UTOENCODERS
Revised: A UTOENCODERS

Original: In this section, we describe various common autoencoder structures and objectives and try them on the lines task.
Revised: In this section, we describe various common autoencoder structures and objectives and try them on the lines task.

Original: Our goal is to quantitatively evaluate the extent to which standard autoencoders exhibit useful interpolation behavior.
Revised: Our goal is to quantitatively evaluate the extent to which standard autoencoders exhibit useful interpolation behavior.

Original: Our results, which we describe below, are summarized in table 1. 
Revised: Our results, which we describe below, are summarized in table 1. [Table]

Original: Base Model Perhaps the most basic autoencoder structure is one which simply maps input datapoints through a “bottleneck” layer whose dimensionality is smaller than the input.
Revised: Model Perhaps the most basic autoencoder structure is one which simply maps input datapoints through a “bottleneck” layer whose dimensionality is smaller than the input.

Original: In this setup, f θ and g φ are both neural networks which respectively map the input to a deterministic latent code z and then back to a reconstructed input.
Revised: In this setup, f θ and g φ are both neural networks which respectively map the input to a deterministic latent code z and then back to a reconstructed input.

Original: Typically, f θ and g φ are trained simultaneously with respect to (cid:107) x − ˆ x (cid:107) 2 .
Revised: Typically, f θ and g φ are trained simultaneously with respect to (cid:107) x − ˆ x (cid:107) 2 .

Original: We will use this framework as a baseline for experimentation for all of the autoencoder variants discussed below.
Revised: We will use this framework as a baseline for experimentation for all of the autoencoder variants discussed below.

Original: In particular, for our base model and all of the other autoencoders we will use the model architecture and training procedure described in appendix B. As a short summary, our encoder consists of a stack of convolutional and average pooling layers, whereas the decoder consists of convolutional and nearest-neighbor upsampling layers.
Revised: In particular, for our base model and all of the other autoencoders we will [Table] [Figure] [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: While we only show one example here, the behavior of each autoencoder was generally similar for all interpolations.

Revised: A more comprehensive measure of interpolation behavior is given in table 1.

Revised: use the model architecture and training procedure described in appendix B.

Revised: As a short summary, our encoder consists of a stack of convolutional and average pooling layers, whereas the decoder consists of convolutional and nearest-neighbor upsampling layers.

Original: For experiments on the synthetic “lines” task, we use a latent dimensionality of 64 .
Revised: For experiments on the synthetic “lines” task, we use a latent dimensionality of 64 .

Original: Note that, because the data manifold is effectively onedimensional, we might expect autoencoders to be able to model this dataset using a one-dimensional latent code; however, using a larger latent code reﬂects the realistic scenario where the latent space is larger than necessary.
Revised: Note that, because the data manifold is effectively onedimensional, we might expect autoencoders to be able to model this dataset using a one-dimensional latent code; however, using a larger latent code reﬂects the realistic scenario where the latent space is larger than necessary.

Original: After training our baseline autoencoder, we achieved a Mean Distance score which was the worst (highest) of all of the autoencoders we studied, though the Smoothness was on par with various other approaches.
Revised: After training our baseline autoencoder, we achieved a Mean Distance score which was the worst (highest) of all of the autoencoders we studied, though the Smoothness was on par with various other approaches.

Original: In general, we observed some reasonable interpolations when using the baseline model, but found that the intermediate points on the interpolation were typically not realistic as seen in the example interpolation in ﬁg. 3a.[Table]
Revised: In general, we observed some reasonable interpolations when using the baseline model, but found that the intermediate points on the interpolation were typically not realistic as seen in the example interpolation in ﬁg. 3a.

Original: only show one example behavior of autoencoder

Original: Denoising Autoencoder
Revised: Denoising Autoencoder

Original: An early modiﬁcation to the standard autoencoder setup was proposed by
Revised: An early modiﬁcation to the standard autoencoder setup was proposed by

Original: Vincent et al.
Revised: Vincent et al.

Original: (2010), where instead of feeding x into the autoencoder, a corrupted version ˜ x ∼ q (˜ x | x ) is sampled from the conditional probability distribution q (˜ x | x ) and is fed into the autoencoder instead.
Revised: (2010), where instead of feeding x into the autoencoder, a corrupted version ˜ x ∼ q (˜ x | x ) is sampled from the conditional probability distribution q (˜ x | x ) and is fed into the autoencoder instead.

Original: The autoencoder’s goal remains to produce ˆ x which minimizes (cid:107) x − ˆ x (cid:107) 2 .
Revised: The autoencoder’s goal remains to produce ˆ x which minimizes (cid:107) x − ˆ x (cid:107) 2 .

Original: One justiﬁcation of this approach is that the corrupted inputs should fall outside of the true data manifold, so the autoencoder must learn to map points from outside of the data manifold back onto it.
Revised: One justiﬁcation of this approach is that the corrupted inputs should fall outside of the true data manifold, so the autoencoder must learn to map points from outside of the data manifold back onto it.

Original: This provides an implicit way of deﬁning and learning the data manifold via the coordinate system induced by the latent space.
Revised: This provides an implicit way of deﬁning and learning the data manifold via the coordinate system induced by the latent space.

Original: While various corruption procedures q (˜ x | x ) have been used such as masking and salt-and-pepper noise, in this paper we consider the simple case of additive isotropic Gaussian noise where ˜ x ∼
Revised: While various corruption procedures q (˜ x | x ) have been used such as masking and salt-and-pepper noise, in this paper we consider the simple case of additive isotropic Gaussian noise where ˜ x ∼

Original: N ( x, σ 2 I ) and σ is a hyperparameter.
Revised: N ( x, σ 2 I ) and σ is a hyperparameter.

Original: After tuning σ , we found simply setting σ = 1 . 0 to work best.
Revised: After tuning σ , we found simply setting σ = 1 . 0 to work best.

Original: Interestingly, we found the denoising autoencoder often produced “data-space” interpolation (as seen in ﬁg. 3b) when interpolating in latent space.
Revised: Interestingly, we found the denoising autoencoder often produced “data-space” interpolation (as seen in ﬁg. 3b) when interpolating in latent space.

Original: This resulted in comparatively poor Mean Distance and
Revised: This resulted in comparatively poor Mean Distance and

Original: Smoothness scores.
Revised: Smoothness scores.

Original: Variational Autoencoder The Variational Autoencoder (VAE) (Kingma & Welling, 2014; Rezende et al., 2014) introduces the constraint that the latent code z is a random variable distributed according to a prior distribution p ( z ) .
Revised: Variational Autoencoder The Variational Autoencoder (VAE) (Kingma & Welling, 2014; Rezende et al., 2014) introduces the constraint that the latent code z is a random variable distributed according to a prior distribution p ( z ) .

Original: The encoder f θ can then be considered an approximation to the posterior p ( z | x ) .
Revised: The encoder f θ can then be considered an approximation to the posterior p ( z | x ) .

Original: Then, the decoder g φ is taken to parametrize the likelihood p ( x | z ) ; in all of our experiments, we consider x to be Bernoulli distributed.
Revised: Then, the decoder g φ is taken to parametrize the likelihood p ( x | z ) ; in all of our experiments, we consider x to be Bernoulli distributed.

Original: The latent distribution constraint is enforced by an additional loss term which measures the KL divergence between approximate posterior and prior.
Revised: The latent distribution constraint is enforced by an additional loss term which measures the KL divergence between approximate posterior and prior.

Original: VAEs then use log-likelihood for the reconstruction loss (cross-entropy in the case of Bernoulli-distributed data), which results in the following combined loss function: − E
Revised: VAEs then use log-likelihood for the reconstruction loss (cross-entropy in the case of Bernoulli-distributed data), which results in the following combined loss function: − E

Original: [log g φ ( z )]
Revised: [log g φ ( z )]

Original: + KL( f θ ( x ) || p ( z )) where the expectation is taken with respect to z ∼ f θ ( x ) and KL( ·||· ) is the KL divergence.
Revised: + KL( f θ ( x ) || p ( z )) where the expectation is taken with respect to z ∼ f θ ( x ) and KL( ·||· ) is the KL divergence.

Original: Minimizing this loss function can be considered maximizing a lower bound (the “ELBO”) on the likelihood of the training set, producing a likelihood-based generative model which allows novel data points to be sampled by ﬁrst sampling z ∼ p
Revised: Minimizing this loss function can be considered maximizing a lower bound (the “ELBO”) on the likelihood of the training set, producing a likelihood-based generative model which allows novel data points to be sampled by ﬁrst sampling z ∼ p

Original: ( z ) and then computing g φ ( z ) .
Revised: ( z ) and then computing g φ ( z ) .

Original: A common choice is to let q ( z | x ) be a diagonalcovariance Gaussian, in which case backpropagation through sampling from q ( z | x ) is feasible via the “reparametrization trick” which replaces z ∼ N ( µ, σI ) with (cid:15) ∼ N (0 , I ) , z = µ + σ (cid:12) (cid:15) where µ, σ ∈ R
Revised: A common choice is to let q ( z | x ) be a diagonalcovariance Gaussian, in which case backpropagation through sampling from q ( z | x ) is feasible via the “reparametrization trick” which replaces z ∼ N ( µ, σI ) with (cid:15) ∼ N (0 , I ) , z = µ + σ (cid:12) (cid:15) where µ, σ ∈ R

Original: d
Revised: d

Original: z are the predicted mean and standard deviation produced by f θ .
Revised: z are the predicted mean and standard deviation produced by f θ .

Original: Various modiﬁed objectives (Higgins et al., 2016; Zhao et al., 2017), improved prior distributions (Kingma et al., 2016; Tomczak & Welling, 2016; 2017) and improved model architectures (Sønderby et al., 2016;
Revised: Various modiﬁed objectives (Higgins et al., 2017; Zhao et al., 2017), improved prior distributions (Kingma et al., 2016; Tomczak & Welling, 2016; 2017) and improved model architectures (Sønderby et al., 2016;

Original: Chen et al., 2016b; Gulrajani et al., 2016) have been proposed to better the VAE’s performance on downstream tasks, but in this paper we solely consider the “vanilla” VAE objective and prior described above applied to our baseline autoencoder structure.
Revised: Chen et al., 2016b; Gulrajani et al., 2016) have been proposed to better the VAE’s performance on downstream tasks, but in this paper we solely consider the “vanilla” VAE objective and prior described above applied to our baseline autoencoder structure.

Original: When trained on the lines benchmark, we found the VAE was able to effectively model the data distribution (see samples, ﬁg. 5 in appendix C) and accurately reconstruct inputs.
Revised: When trained on the lines benchmark, we found the VAE was able to effectively model the data distribution (see samples, ﬁg. 5 in appendix C) and accurately reconstruct inputs.

Original: In interpolations produced by the VAE, intermediate points tend to look realistic, but the angle of the lines do not follow a smooth or short path (ﬁg. 3c).
Revised: In interpolations produced by the VAE, intermediate points tend to look realistic, but the angle of the lines do not follow a smooth or short path (ﬁg. 3c).

Original: This resulted in a very good Mean Distance score but a very poor Smoothness score.
Revised: This resulted in a very good Mean Distance score but a very poor Smoothness score.

Original: Contrary to expectations, this suggests that desirable interpolation behavior may not follow from an effective generative model of the data distribution.
Revised: Contrary to expectations, this suggests that desirable interpolation behavior may not follow from an effective generative model of the data distribution.

Original: Adversarial 
Revised: Adversarial Autoencoder

Original: Autoencoder The Adversarial Autoencoder (AAE) (Makhzani et al., 2015) proposes an alternative way of enforcing structure on the latent code.
Revised: The Adversarial Autoencoder (AAE) (Makhzani et al., 2015) proposes an alternative way of enforcing structure on the latent code.

Original: Instead of minimizing a KL divergence between the distribution of latent codes and a prior distribution, a critic network is trained in tandem with the autoencoder to predict whether a latent code comes from f θ or from the prior p ( z ) .
Revised: Instead of minimizing a KL divergence between the distribution of latent codes and a prior distribution, a critic network is trained in tandem with the autoencoder to predict whether a latent code comes from f θ or from the prior p ( z ) .

Original: The autoencoder is simultaneously trained to reconstruct inputs (via a standard reconstruction loss) and to “fool” the critic.
Revised: The autoencoder is simultaneously trained to reconstruct inputs (via a standard reconstruction loss) and to “fool” the critic.

Original: The autoencoder is allowed to backpropagate gradients through the critic’s loss function, but the autoencoder and critic parameters are optimized separately.
Revised: The autoencoder is allowed to backpropagate gradients through the critic’s loss function, but the autoencoder and critic parameters are optimized separately.

Original: This effectively computes an “adversarial divergence” between the latent code distribution and the chosen prior.
Revised: This effectively computes an “adversarial divergence” between the latent code distribution and the chosen prior.

Revised: This framework was later generalized and referred to as the “Wasserstein Autoencoder” (Tolstikhin et al., 2017)

Original: One advantage of this approach is that it allows for an arbitrary prior (as opposed to those which have a tractable KL divergence).
Revised: One advantage of this approach is that it allows for an arbitrary prior (as opposed to those which have a tractable KL divergence).

Original: The disadvantages are that the AAE no longer has a probabilistic interpretation and involves optimizing a minimax game, which can cause instabilities.
Revised: The disadvantages are that the AAE no longer has a probabilistic interpretation and involves optimizing a minimax game, which can cause instabilities.

Original: Using the AAE requires choosing a prior, a critic structure, and a training scheme for the critic.
Revised: Using the AAE requires choosing a prior, a critic structure, and a training scheme for the critic.

Original: For simplicity, we also used a spherical Gaussian prior for the AAE.
Revised: For simplicity, we also used a spherical Gaussian prior for the AAE.

Original: We experimented with various architectures for the critic, and found the best performance with a critic which consisted of two dense layers, each with 100 units and a leaky ReLU nonlinearity.
Revised: We experimented with various architectures for the critic, and found the best performance with a critic which consisted of two dense layers, each with 100 units and a leaky ReLU nonlinearity.

Original: We found it satisfactory to simply use the same optimizer and learning rate for the critic as was used for the autoencoder.
Revised: We found it satisfactory to simply use the same optimizer and learning rate for the critic as was used for the autoencoder.

Original: On our lines benchmark, the AAE typically produced smooth interpolations, but exhibited degraded quality in the middle of interpolations (ﬁg. 3d).
Revised: On our lines benchmark, the AAE typically produced smooth interpolations, but exhibited degraded quality in the middle of interpolations (ﬁg. 3d).

Original: This behavior produced the best Smoothness score among existing autoencoders, but a relatively poor Mean Distance score.
Revised: This behavior produced the best Smoothness score among existing autoencoders, but a relatively poor Mean Distance score.

Original: Vector Quantized Variational Autoencoder (VQ-VAE)
Revised: Vector Quantized Variational Autoencoder (VQ-VAE)

Original: The Vector Quantized Variational Autoencoder (VQ-VAE) was introduced by (van den Oord et al., 2017) as a way to train discrete-latent autoencoders using a learned codebook.
Revised: The Vector Quantized Variational Autoencoder (VQ-VAE) was introduced by (van den Oord et al., 2017) as a way to train discrete-latent autoencoders using a learned codebook.

Original: In the VQ-VAE, the encoder f θ ( x ) produces a continuous hidden representation z ∈ R dz which is then mapped to z q , its nearest neighbor in a “codebook” { e j ∈ R d z , j ∈ 1 , . .
Revised: In the VQ-VAE, the encoder f θ ( x ) produces a continuous hidden representation z ∈ R dz which is then mapped to z q , its nearest neighbor in a “codebook” { e j ∈ R d z , j ∈ 1 , . .

Original: . , K } .
Revised: . , K } .

Original: z q is then passed to the decoder for reconstruction.
Revised: z q is then passed to the decoder for reconstruction.

Original: The encoder is trained to minimize the reconstruction loss using the straight-through gradient estimator (Bengio et al., 2013a), together with a commitment loss term β (cid:107) z − sg( z q ) (cid:107) (where β is a scalar hyperparameter) which encourages encoder outputs to move closer to their nearest codebook entry.
Revised: The encoder is trained to minimize the reconstruction loss using the straight-through gradient estimator (Bengio et al., 2013a), together with a commitment loss term β (cid:107) z − sg( z q ) (cid:107) (where β is a scalar hyperparameter) which encourages encoder outputs to move closer to their nearest codebook entry.

Original: Here sg denotes the stop gradient operator, i.e. sg( x ) =
Revised: Here sg denotes the stop gradient operator, i.e. sg( x ) =

Original: x in the forward pass, and sg( x ) = 0 in the backward pass.
Revised: x in the forward pass, and sg( x ) = 0 in the backward pass.

Original: The codebook entries e j are updated as an exponential moving average (EMA) of the continuous latents z that map to them at each training iteration.
Revised: The codebook entries e j are updated as an exponential moving average (EMA) of the continuous latents z that map to them at each training iteration.

Original: The VQ-VAE training procedure using this EMA update rule can be seen as performing the K -means or the hard Expectation Maximization (EM) algorithm on the latent codes (Roy et al., 2018).
Revised: The VQ-VAE training procedure using this EMA update rule can be seen as performing the K -means or the hard Expectation Maximization (EM) algorithm on the latent codes (Roy et al., 2018).

Original: We perform interpolation in the VQ-VAE by interpolating continuous latents, mapping them to their nearest codebook entries, and decoding the result.
Revised: We perform interpolation in the VQ-VAE by interpolating continuous latents, mapping them to their nearest codebook entries, and decoding the result.

Original: Assuming sufﬁciently large codebook, a semantically “smooth” interpolation may be possible.
Revised: Assuming sufﬁciently large codebook, a semantically “smooth” interpolation may be possible.

Original: On the lines task, we found that this procedure produced poor interpolations.
Revised: On the lines task, we found that this procedure produced poor interpolations.

Original: Ultimately, many entries of the codebook were mapped to unrealistic datapoints, and the interpolations resembled those of the baseline autoencoder.
Revised: Ultimately, many entries of the codebook were mapped to unrealistic datapoints, and the interpolations resembled those of the baseline autoencoder.

Original: Adversarially Constrained Autoencoder Interpolation Finally, we turn to evaluating our proposed adversarial regularizer for improving interpolations.
Revised: Adversarially Constrained Autoencoder Interpolation Finally, we turn to evaluating our proposed adversarial regularizer for improving interpolations.

Original: For simplicity, on the lines benchmark we found it sufﬁcient to use a critic architecture which was equivalent to the encoder (as described in appendix B).
Revised: For simplicity, on the lines benchmark we found it sufﬁcient to use a critic architecture which was equivalent to the encoder (as described in appendix B).

Original: To produce a single scalar value from its output, we computed the mean of its ﬁnal layer activations.
Revised: To produce a single scalar value from its output, we computed the mean of its ﬁnal layer activations.

Original: For the hyperparameters λ and γ we found values of 0 .
Revised: For the hyperparameters λ and γ we found values of 0 .

Original: 5 and 0 .
Revised: 5 and 0 .

Original: 2 to achieve good results, though the performance was not very sensitive to their values.
Revised: 2 to achieve good results, though the performance was not very sensitive to their values.

Original: We use these values for the coefﬁcients for all of our experiments.
Revised: We use these values for the coefﬁcients for all of our experiments.

Original: Finally, we trained the critic using the same optimizer and hyperparameters as the autoencoder.
Revised: Finally, we trained the critic using the same optimizer and hyperparameters as the autoencoder.

Original: [Table]

Original: We found dramatically improved performance on the lines benchmark when using ACAI – it achieved the best Mean Distance and Smoothness score among the autoencoders we considered.
Revised: We found dramatically improved performance on the lines benchmark when using ACAI – it achieved the best Mean Distance and Smoothness score among the autoencoders we considered.

Original: When inspecting the resulting interpolations, we found it occasionally chose a longer path than necessary but typically produced “perfect” interpolation behavior as seen in ﬁg. 3f.
Revised: When inspecting the resulting interpolations, we found it occasionally chose a longer path than necessary but typically produced “perfect” interpolation behavior as seen in ﬁg. 3f.

Original: This provides quantitative evidence ACAI is successful at encouraging realistic and smooth interpolations
Revised: This provides quantitative evidence ACAI is successful at encouraging realistic and smooth interpolations.

Original: [Table]

Original: 3. I NTERPOLATIONS ON R EAL D ATA
Revised: 3. I NTERPOLATIONS ON R EAL D ATA

Original: We have so far only discussed results on our synthetic lines benchmark.
Revised: We have so far only discussed results on our synthetic lines benchmark.

Original: We also provide example reconstructions and interpolations produced by each autoencoder for MNIST (LeCun, 1998), SVHN (Netzer et al., 2011), and CelebA (Liu et al., 2015) in appendix D. For each dataset, we trained autoencoders with latent dimensionalities of 32 and 256 .
Revised: We also provide example reconstructions and interpolations produced by each autoencoder for MNIST (LeCun, 1998), SVHN (Netzer et al., 2011), and CelebA (Liu et al., 2015) in appendix D. For each dataset, we trained autoencoders with latent dimensionalities of 32 and 256 .

Original: Since we do not know the underlying data manifold for these datasets, no metrics are available to evaluate performance and we can only make qualitative judgments as to the reconstruction and interpolation quality.
Revised: Since we do not know the underlying data manifold for these datasets, no metrics are available to evaluate performance and we can only make qualitative judgments as to the reconstruction and interpolation quality.

Original: We ﬁnd that most autoencoders produce “blurrier” images with d z = 32 but generally give smooth interpolations regardless of the latent dimensionality.
Revised: We ﬁnd that most autoencoders produce “blurrier” images with d z = 32 but generally give smooth interpolations regardless of the latent dimensionality.

Original: The exception to this observation was the VQ-VAE which seems generally to work better with d z = 32 and occasionally even diverged for d z = 256 (see e.g. ﬁg. 9e).
Revised: The exception to this observation was the VQ-VAE which seems generally to work better with d z = 32 and occasionally even diverged for d z = 256 (see e.g. ﬁg. 9e).

Original: This may be due to the nearest-neighbor discretization (and gradient estimator) failing in high dimensions.
Revised: This may be due to the nearest-neighbor discretization (and gradient estimator) failing in high dimensions.

Original: Across datasets, we found the VAE and denoising autoencoder typically produced more blurry interpolations, whereas AAE and ACAI usually produced realistic interpolations.
Revised: Across datasets, we found the VAE and denoising autoencoder typically produced more blurry interpolations. AAE and ACAI generally produced realistic interpolations, even between dissimilar datapoints (for example, in ﬁg. 7 bottom).

Original: The baseline model often effectively interpolated in data space.
Revised: The baseline model often effectively interpolated in data space.

Original:  I MPROVED R EPRESENTATION L EARNING
Revised:  I MPROVED R EPRESENTATION L EARNING

Original: We have so far solely focused on measuring the interpolation abilities of different autoencoders.
Revised: We have so far solely focused on measuring the interpolation abilities of different autoencoders.

Original: Now, we turn to the question of whether improved interpolation is associated with improved performance on downstream tasks.
Revised: Now, we turn to the question of whether improved interpolation is associated with improved performance on downstream tasks.

Original: Speciﬁcally, we will evaluate whether using our proposed regularizer results in latent space representations which provide better performance in supervised learning and clustering.
Revised: Speciﬁcally, we will evaluate whether using our proposed regularizer results in latent space representations which provide better performance in supervised learning and clustering.

Original: Put another way, we seek to test whether improving interpolation results in a latent representation which has disentangled important factors of variation (such as class identity) in the dataset.
Revised: Put another way, we seek to test whether improving interpolation results in a latent representation which has disentangled important factors of variation (such as class identity) in the dataset.

Original: To answer this question, we ran classiﬁcation and clustering experiments using the learned latent spaces of different autoencoders on the MNIST (LeCun, 1998), SVHN (Netzer et al., 2011), and CIFAR-
Revised: To answer this question, we ran classiﬁcation and clustering experiments using the learned latent spaces of different autoencoders on the MNIST (LeCun, 1998), SVHN (Netzer et al., 2011), and CIFAR-

Original: (Krizhevsky, 2009) datasets.
Revised: (Krizhevsky, 2009) datasets.

Original: Single-Layer Classiﬁer
Revised: Single-Layer Classiﬁer

Original: A common method for evaluating the quality of a learned representation (such as the latent space of an autoencoder) is to use it as a feature representation for a simple, one-layer classiﬁer (i.e. logistic regression) trained on a supervised learning task (Coates et al., 2011).
Revised: A common method for evaluating the quality of a learned representation (such as the latent space of an autoencoder) is to use it as a feature representation for a simple, one-layer classiﬁer (i.e. logistic regression) trained on a supervised learning task (Coates et al., 2011).

Original: The justiﬁcation for this evaluation procedure is that a learned representation which has effectively disentangled class identity will allow the classiﬁer to obtain reasonable performance despite its simplicity.
Revised: The justiﬁcation for this evaluation procedure is that a learned representation which has effectively disentangled class identity will allow the classiﬁer to obtain reasonable performance despite its simplicity.

Original: To test different autoencoders in this setting, we trained a separate single-layer classiﬁer in tandem with the autoencoder using the latent representation as input.
Revised: To test different autoencoders in this setting, we trained a separate single-layer classiﬁer in tandem with the autoencoder using the latent representation as input.

Original: We did not optimize autoencoder parameters with respect to the classiﬁer’s loss, which ensures that we are measuring unsupervised  representation learning performance.
Revised: We did not optimize autoencoder parameters with respect to the classiﬁer’s loss, which ensures that we are measuring unsupervised [Table] [Table] representation learning performance.

Original: We repeated this procedure for latent dimensionalities ofand 256 (MNIST and SVHN) and 256 and 1024 (CIFAR-10).
Revised: We repeated this procedure for latent dimensionalities ofand 256 (MNIST and SVHN) and 256 and 1024 (CIFAR-10).

Original: Our results are shown in table 2.
Revised: Our results are shown in table 2.

Original: In all settings, using ACAI instead of the baseline autoencoder upon which it is based produced signiﬁcant gains – most notably, on SVHN with a latent dimensionality of 256 , the baseline achieved an accuracy of only 22.74% whereas ACAI achieved 85.14%.In general,
Revised: In all settings, using ACAI instead of the baseline autoencoder upon which it is based produced signiﬁcant gains – most notably, on SVHN with a latent dimensionality of 256 , the baseline achieved an accuracy of only 22.74% whereas ACAI achieved 85.14%.

Original: [Table] we found the denoising autoencoder, VAE [Table] and ACAI obtained signiﬁcantly higher performance compared to the remaining models.
Revised: In general, we found the denoising autoencoder, VAE, and ACAI obtained signiﬁcantly higher performance compared to the remaining models.

Original: On MNIST and SVHN, ACAI achieved the best accuracy by a signiﬁcant margin; on CIFAR-10, the performance of ACAI and the denoising autoencoder was similar.
Revised: On MNIST and SVHN, ACAI achieved the best accuracy by a signiﬁcant margin; on CIFAR-10, the performance of ACAI and the denoising autoencoder was similar.

Original: By way of comparison, we found a single-layer classiﬁer applied directly to (ﬂattened) image pixels achieved an accuracy of 92.31%, 23.48%, and 39.70% on MNIST, SVHN, and CIFARrespectively, so classifying using the representation learned by ACAI provides a huge beneﬁt.
Revised: By way of comparison, we found a single-layer classiﬁer applied directly to (ﬂattened) image pixels achieved an accuracy of 92.31%, 23.48%, and 39.70% on MNIST, SVHN, and CIFARrespectively, so classifying using the representation learned by ACAI provides a huge beneﬁt.

Original: Clustering If an autoencoder groups points with common salient characteristics close together in latent space without observing any labels, it arguably has uncovered some important structure in the data in an unsupervised fashion.
Revised: Clustering If an autoencoder groups points with common salient characteristics close together in latent space without observing any labels, it arguably has uncovered some important structure in the data in an unsupervised fashion.

Original: A more difﬁcult test of an autoencoder is therefore clustering its latent space, i.e. separating the latent codes for a dataset into distinct groups without using any labels.
Revised: A more difﬁcult test of an autoencoder is therefore clustering its latent space, i.e. separating the latent codes for a dataset into distinct groups without using any labels.

Original: To test the clusterability of the latent spaces learned by different autoencoders, we simply apply K-Means clustering (MacQueen, 1967) to the latent codes for a given dataset.
Revised: To test the clusterability of the latent spaces learned by different autoencoders, we simply apply K-Means clustering (MacQueen, 1967) to the latent codes for a given dataset.

Original: Since K-Means uses Euclidean distance, it is sensitive to each dimension’s relative variance.
Revised: Since K-Means uses Euclidean distance, it is sensitive to each dimension’s relative variance.

Original: We therefore used PCA whitening on the latent space learned by each autoencoder to normalize the variance of its dimensions prior to clustering.
Revised: We therefore used PCA whitening on the latent space learned by each autoencoder to normalize the variance of its dimensions prior to clustering.

Original: K-Means can exhibit highly variable results depending on how it is initialized, so for each autoencoder we ran K-Means 1,000 times from different random initializations and chose the clustering with the best objective value on the training set.
Revised: K-Means can exhibit highly variable results depending on how it is initialized, so for each autoencoder we ran K-Means 1,000 times from different random initializations and chose the clustering with the best objective value on the training set.

Original: For evaluation, we adopt the methodology of Xie et al.
Revised: For evaluation, we adopt the methodology of Xie et al.

Original: ; Hu et al.
Revised: ; Hu et al.

Original: Given that the dataset in question has labels (which are not used for training the model, the clustering algorithm, or choice of random initialization), we can cluster the data into C distinct groups where C is the number of classes in the dataset.
Revised: Given that the dataset in question has labels (which are not used for training the model, the clustering algorithm, or choice of random initialization), we can cluster the data into C distinct groups where C is the number of classes in the dataset.

Original: We then compute the “clustering accuracy”, which is simply the accuracy corresponding to the optimal one-to-one mapping of cluster IDs and classes (Xie et al., 2016).
Revised: We then compute the “clustering accuracy”, which is simply the accuracy corresponding to the optimal one-to-one mapping of cluster IDs and classes (Xie et al., 2016).

Original: Our results are shown in table 3.
Revised: Our results are shown in table 3.

Original: On both MNIST and SVHN, ACAI achieved the best or second-best performance for both d z = 32 and d z = 256 .
Revised: On both MNIST and SVHN, ACAI achieved the best or second-best performance for both d z = 32 and d z = 256 .

Original: We do not report results on CIFAR-10 because all of the autoencoders we studied achieved a near-random clustering accuracy.
Revised: We do not report results on CIFAR-10 because all of the autoencoders we studied achieved a near-random clustering accuracy.

Original: Previous efforts to evaluate clustering performance on CIFAR-10 use learned feature representations from a convolutional network trained on ImageNet (Hu et al., 2017) which we believe only indirectly measures unsupervised learning capabilities.
Revised: Previous efforts to evaluate clustering performance on CIFAR-10 use learned feature representations from a convolutional network trained on ImageNet (Hu et al., 2017) which we believe only indirectly measures unsupervised learning capabilities.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we have provided an in-depth perspective on interpolation in autoencoders.
Revised: In this paper, we have provided an in-depth perspective on interpolation in autoencoders.

Original: We proposed
Revised: We proposed

Original: Adversarially Constrained Autoencoder Interpolation (ACAI), which uses a critic to encourage interpolated datapoints to be more realistic.
Revised: Adversarially Constrained Autoencoder Interpolation (ACAI), which uses a critic to encourage interpolated datapoints to be more realistic.

Original: To make interpolation a quantiﬁable concept, we proposed a synthetic benchmark and showed that ACAI substantially outperformed common autoencoder models.
Revised: To make interpolation a quantiﬁable concept, we proposed a synthetic benchmark and showed that ACAI substantially outperformed common autoencoder models.

Original: This task also yielded unexpected insights, such as that a VAE which has effectively learned the data distribution might not interpolate.
Revised: This task also yielded unexpected insights, such as that a VAE which has effectively learned the data distribution might not interpolate.

Original: We also studied the effect of improved interpolation on downstream tasks, and showed that ACAI led to improved performance for feature learning and unsupervised clustering.
Revised: We also studied the effect of improved interpolation on downstream tasks and showed that ACAI led to improved performance for feature learning and unsupervised clustering.

Original: In future work, we are interested in investigating whether our regularizer improves the performance of autoencoders other than the standard “vanilla” autoencoder we applied it to.
Revised: In future work we are interested in investigating whether our regularizer improves the performance of autoencoders other than the standard “vanilla” autoencoder we applied it to.

Original: In this paper, we primarily focused on image datasets due to the ease of visualizing interpolations, but we are also interested in applying these ideas to non-image datasets.
Revised: In this paper, we primarily focused on image datasets due to the ease of visualizing interpolations, but we are also interested in applying these ideas to non-image datasets

Original: A L INE B ENCHMARK A. E XAMPLE I NTERPOLATIONS
Revised: A L INE B ENCHMARK A. E XAMPLE I NTERPOLATIONS

Original: Some example data and interpolations for our synthetic lines benchmark are shown in ﬁg. 4.
Revised: Some example data and interpolations for our synthetic lines benchmark are shown in ﬁg. 4.

Original: Full discussion of this benchmark is available in section 3.
Revised: Full discussion of this benchmark is available in section 3.

Original: . A. E VALUATION M ETRICS
Revised: . A. E VALUATION M ETRICS

Original: We deﬁne our Mean Distance and Smoothness metrics as follows: Let x 1 and x 2 be two input images we are interpolating between and
Revised: We deﬁne our Mean Distance and Smoothness metrics as follows: Let x 1 and x 2 be two input images we are interpolating between and

Original: [Equation]
Revised: [Equation]

Original: be the decoded point corresponding to mixing x 1 and x 2 ’s latent codes using coefﬁcient α = n − 1 / N − 1 .
Revised: be the decoded point corresponding to mixing x 1 and x 2 ’s latent codes using coefﬁcient α = n − 1 / N − 1 .

Original: The images ˆ x n , n ∈ { 1 , . . .
Revised: The images ˆ x n , n ∈ { 1 , . . .

Original: , N } then comprise a length- N interpolation between x 1 and x 2 .
Revised: , N } then comprise a length- N interpolation between x 1 and x 2 .

Original: To produce our evaluation metrics, we ﬁrst ﬁnd the closest true datapoint (according to cosine distance) for each of the N intermediate images along the interpolation.
Revised: To produce our evaluation metrics, we ﬁrst ﬁnd the closest true datapoint (according to cosine distance) for each of the N intermediate images along the interpolation.

Original: Finding the closest image among all possible line images is infeasible; instead we ﬁrst generate a size- D collection of line images D with corresponding angles Λ q , q ∈ { 1 , . . .
Revised: Finding the closest image among all possible line images is infeasible; instead we ﬁrst generate a size- D collection of line images D with corresponding angles Λ q , q ∈ { 1 , . . .

Original: , D } spaced evenly between 0 and 2 π .
Revised: , D } spaced evenly between 0 and 2 π .

Original: Then, we match each image in the interpolation to a real datapoint by ﬁnding
Revised: Then, we match each image in the interpolation to a real datapoint by ﬁnding

Original: [Equation]
Revised: [Equation]

Original: for n ∈ { 1 , . . .
Revised: for n ∈ { 1 , . . .

Original: , N } , where C n,q is the cosine distance between ˆ x n and the q th entry of D .
Revised: , N } , where C n,q is the cosine distance between ˆ x n and the q th entry of D .

Original: To capture the notion of “intermediate points look realistic”, we compute
Revised: To capture the notion of “intermediate points look realistic”, we compute

Original: [Equation]
Revised: [Equation]

Original: We now deﬁne a perfectly smooth interpolation to be one which consists of lines with angles which linearly move from the angle of D q (cid:63) 1 to that of D q (cid:63)N .
Revised: We now deﬁne a perfectly smooth interpolation to be one which consists of lines with angles which linearly move from the angle of D q (cid:63) 1 to that of D q (cid:63)N .

Original: Note that if, for example, the interpolated lines go from Λ q (cid:63) 

Revised: Note that if, for example, the interpolated lines go from Λ q (cid:63)

Original: = π / 10 to Λ q (cid:63)N = 19 π / 10 then the angles corresponding to the shortest path will have a discontinuity from 0 to 2 π .
Revised: = π / 10 to Λ q (cid:63)N = 19 π / 10 then the angles corresponding to the shortest path will have a discontinuity from 0 to 2 π .

Original: To avoid this, we ﬁrst “unwrap” the angles { Λ q (cid:63) 1 , . . .
Revised: To avoid this, we ﬁrst “unwrap” the angles { Λ q (cid:63) 1 , . . .

Original: , Λ q (cid:63)N } by removing discontinuities larger than π by adding multiples of ± 2 π when the absolute difference between Λ q (cid:63)n − 1 and Λ q (cid:63)n is greater than π to produce the angle sequence { ˜Λ q (cid:63) 1 , . . .
Revised: , Λ q (cid:63)N } by removing discontinuities larger than π by adding multiples of ± 2 π when the absolute difference between Λ q (cid:63)n − 1 and Λ q (cid:63)n is greater than π to produce the angle sequence { ˜Λ q (cid:63) 1 , . . .

Original: ˜Λ q (cid:63)N } .
Revised: ˜Λ q (cid:63)N } .

Original: Then, we deﬁne a measure of smoothness as
Revised: Then, we deﬁne a measure of smoothness as

Original: [Equation]
Revised: [Equation]

Original: In other words, we measure the how much larger the largest change in (normalized) angle is compared to the minimum possible value ( 1 / ( N − 1) ).
Revised: In other words, we measure the how much larger the largest change in (normalized) angle is compared to the minimum possible value ( 1 / ( N − 1) ).

Original: By way of example, ﬁgs. 4b, 4d and 4e would all achieve Mean Distance scores near zero and ﬁgs.
Revised: By way of example, ﬁgs. 4b, 4d and 4e would all achieve Mean Distance scores near zero and ﬁgs.

Original: 4c and 4f would achieve larger Mean Distance scores.
Revised: 4c and 4f would achieve larger Mean Distance scores.

Original: Figures 4b and 4f would achieve Smoothness scores near zero, ﬁgs.
Revised: Figures 4b and 4f would achieve Smoothness scores near zero, ﬁgs.

Original: 4c and 4d have poor Smoothness, and ﬁg. 4e is in between.
Revised: 4c and 4d have poor Smoothness, and ﬁg. 4e is in between.

Original: B B ASE M ODEL A RCHITECTURE AND T RAINING P ROCEDURE
Revised: B B ASE M ODEL A RCHITECTURE AND T RAINING P ROCEDURE

Original: All of the autoencoder models we studied in this paper used the following architecture and training procedure: The encoder consists of blocks of two consecutive 3 × 3 convolutional layers followed by 2 × 2 average pooling.
Revised: All of the autoencoder models we studied in this paper used the following architecture and training procedure: The encoder consists of blocks of two consecutive 3 × 3 convolutional layers followed by 2 × 2 average pooling.

Original: All convolutions (in the encoder and decoder) are zero-padded so that the input and output height and width are equal.
Revised: All convolutions (in the encoder and decoder) are zero-padded so that the input and output height and width are equal.

Original: The number of channels is doubled before each average pooling layer.
Revised: The number of channels is doubled before each average pooling layer.

Original: Two more 3 × 3 convolutions are then performed, the last one without activation and the ﬁnal output is used as the latent representation.
Revised: Two more 3 × 3 convolutions are then performed, the last one without activation and the ﬁnal output is used as the latent representation.

Original: All convolutional layers except for the ﬁnal use a leaky ReLU nonlinearity (Maas et al., 2013).
Revised: All convolutional layers except for the ﬁnal use a leaky ReLU nonlinearity (Maas et al., 2013).

Original: For experiments on the synthetic “lines” task, the convolution-average pool blocks are repeated 4 times until we reach a latent dimensionality of 64.
Revised: For experiments on the synthetic “lines” task, the convolution-average pool blocks are repeated 4 times until we reach a latent dimensionality of 64.

Original: For subsequent experiments on real datasets (section 4), we repeat the blocks 3 times, resulting in a latent dimensionality of 256.
Revised: For subsequent experiments on real datasets (section 4), we repeat the blocks 3 times, resulting in a latent dimensionality of 256.

Original: The decoder consists of blocks of two consecutive 3 × 3 convolutional layers with leaky ReLU nonlinearities followed by 2 × 2 nearest neighbor
Revised: The decoder consists of blocks of two consecutive 3 × 3 convolutional layers with leaky ReLU nonlinearities followed by 2 × 2 nearest neighbor

Original: upsampling (Odena et al., 2016).
Revised: upsampling (Odena et al., 2016).

Original: The number of channels is halved after each upsampling layer.
Revised: The number of channels is halved after each upsampling layer.

Original: These blocks are repeated until we reach the target resolution ( 32 × 32 in all experiments).
Revised: These blocks are repeated until we reach the target resolution ( 32 × 32 in all experiments).

Original: Two more 3 × 3 convolutions are then performed, the last one without activation and with a number of channels equal to the number of desired colors.
Revised: Two more 3 × 3 convolutions are then performed, the last one without activation and with a number of channels equal to the number of desired colors.

Original: All parameters are initialized as zero-mean Gaussian random variables with a standard deviation of/ √ fan_in
Revised: All parameters are initialized as zero-mean Gaussian random variables with a standard deviation of/ √ fan_in

Original: (1+0 .
Revised: (1+0 .

Original: 2 2 ) set in accordance with the leaky ReLU slope of 0 . 2 .
Revised: 2 2 ) set in accordance with the leaky ReLU slope of 0 . 2 .

Original: Models are trained on 2samples in batches of size 64 .
Revised: Models are trained on 2samples in batches of size 64 .

Original: Parameters are optimized with Adam (Kingma & Welling, 2014) with a learning rate of 0.0001 and default values for β 1 , β 2 , and (cid:15) .
Revised: Parameters are optimized with Adam (Kingma & Welling, 2014) with a learning rate of 0.0001 and default values for β 1 , β 2 , and (cid:15) .

Original: C VAE S AMPLES ON THE L INE B ENCHMARK
Revised: C VAE S AMPLES ON THE L INE B ENCHMARK

Original: In ﬁg. 5, we show some samples from our VAE trained on the synthetic line benchmark.
Revised: In ﬁg. 5, we show some samples from our VAE trained on the synthetic line benchmark.

Original: The VAE generally produces realistic samples and seems to cover the data distribution well, despite the fact that it does not produce high-quality interpolations (ﬁg. 3c).
Revised: The VAE generally produces realistic samples and seems to cover the data distribution well, despite the fact that it does not produce high-quality interpolations (ﬁg. 3c).

Original: D I NTERPOLATION E XAMPLES ON R EAL D ATA
Revised: D I NTERPOLATION E XAMPLES ON R EAL D ATA

Original: In this section, we provide a series of ﬁgures (ﬁgs. 6 to 11) showing interpolation behavior for the different autoencoders we studied.
Revised: In this section, we provide a series of ﬁgures (ﬁgs. 6 to 11) showing interpolation behavior for the different autoencoders we studied.

Original: Further discussion of these results is available in section 3.
Revised: Further discussion of these results is available in section 3.

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: ) (e) (f) (a) (
Revised: ) (e) (f) (a) (

Original: [Figure]
Revised: [Figure]

Original: ) (c) (
Revised: ) (c) (

Original: [Figure]
Revised: [Figure]

Original: ) (e) (f) (a) (
Revised: ) (e) (f) (a) (

Original: [Figure]
Revised: [Figure]

Original: ) (c) (d) (e) (
Revised: ) (c) (d) (e) (

Original: [Figure]
Revised: [Figure]

Original: ) (a)
Revised: ) (a)

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: (e) (f) (a) (
Revised: (e) (f) (a) (

Original: [Figure]
Revised: [Figure]

Original: ) (c) (
Revised: ) (c) (

Original: [Figure]
Revised: [Figure]

Original: ) (e) (f) (a) (
Revised: ) (e) (f) (a) (

Original: [Figure]
Revised: [Figure]

Original: ) (c) (d) (e) (
Revised: ) (c) (d) (e) (

Original: [Figure]
Revised: [Figure]

Original: ) (a)
Revised: ) (a)

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]