T RANSFORMATION A UTOREGRESSIVE N ETWORKS
Anonymous authors
Paper under double-blind review
A BSTRACT
The fundamental task of general density estimation has been of keen interest to machine learning.
Recent advances in density estimation have either: a ) proposed a ﬂexible model to estimate the conditional factors of the chain rule, p ( x i ∣ x i − 1 ,... ) ; or b ) used ﬂexible, non-linear transformations of variables of a simple base distribution.
Instead, this work jointly leverages transformations of variables and autoregressive conditional models, and proposes novel methods for both.
We provide a deeper understanding of our methods, showing a considerable improvement through a comprehensive study over both real world and synthetic data.
Moreover, we illustrate the use of our models in outlier detection and image modeling tasks.
I NTRODUCTION
Density estimation is at the core of a multitude of machine learning applications.
However, this fundamental task, which encapsulates the understanding of data, is difﬁcult in the general setting due to issues like the curse of dimensionality.
Furthermore, general data, unlike spatial/temporal data, does not contain a priori known correlations among covariates that may be exploited and engineered with.
For example, image data has known correlations among neighboring pixels that may be hard-coded into a model, whereas one must ﬁnd such correlations in a data-driven fashion with general data.
In order to model high dimensional data, a large number of methods have considered auto-regressive models, which model the conditional factors of the chain rule (Uria et al., 2013; 2016; Germain et al., 2015; Gregor et al., 2014).
I.e these models estimate the conditionals: p ( x i ∣ x i − 1 ,...,x 1 ) , for i ∈ { 1 ,...,d } .
While some methods directly model the conditionals p ( x i ∣ x i − 1 ,... ) using sophisticated semiparametric density estimates, other methods apply sophisticated transformations of variables x ↦ z and take the conditionals over z to be a restricted, often independent base distribution p ( z i ∣ z
i − 1 ,... )
≈ f
( z i ) (Dinh et al., 2014; 2016; Goodfellow, 2016).
In this paper we take a step back from these previous approaches that have considered either: a ) a ﬂexible autoregressive scheme with simple or no transformations of variables (Figure 1a); or b ) a simple autoregressive scheme with ﬂexible transformations of variables (Figure 1b) .
We leverage both of these approaches (Figure 1c), develop novel methods for each, and show a considerable improvement with their combination.
Contributions:
The following are our contributions.
First, we propose two ﬂexible autoreggressive models for modeling conditional distributions: the linear autoregressive model (LAM), and the recurrent autoregressive model (RAM).
LAM employs a simple linear form to condition on previously seen covariates in a ﬂexible fashion.
RAM uses a recurrent neural network (RNN) to evolve conditioning features as the set of conditioning covariates expands.
Furthermore, this paper proposes several novel transformations of variables: 1) we propose an efﬁcient method for learning a linear transfromations on covariates; 2) we develop an invertible RNN-based transformation that directly acts on covariates; 3) we also propose an additive RNN-base transformation.
To better capture correlations in general data, we combine our novel autoreggresive models and transformations of variables.
That is, we propose transformation autogressive networks (TANs) that use a ﬂexible transformation over the original covariates and a powerful autoregressive model to estimate the conditional probabilities over the transformed space.
To assess the efﬁcacy of models, we performed a comprehensive evaluation of autoregressive models and transformations.
Extensive experiments on both real and synthetic datasets show the power of TANs for capturing complex dependencies between the covariates.
Moreover, we show that the learned model can be used for outlier detection, and image modeling.
The remainder of the paper is structured as follows.
First, in Section 2.1 we present two novel methods for modeling condition distributions across covariates.
Next, in Section 2.2, we describe several transformations to use in conjunction with our proposed conditional models.
After, we discuss related work and contrast our approach to previous methods.
We then illustrate the efﬁcacy of our methods with both synthetic and real-world data.
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation] [Figure] [Equation] [Figure]
[Equation]
[Figure]
T RANSFORMATION A UTOREGRESSIVE N ETWORKS
First, we propose two autoregressive models to estimate the conditional distribution of input covariates x ∈ R d .
After, we shall show how we may use such models over a transformation z
= q ( x ) , while renormalizing to obtain density values for x .
2. A UTOREGRESSIVE M ODELS
Autoregressive models decompose density estimation of a multivariate variable x ∈ R d into multiple conditional tasks on a growing set of inputs through the chain rule:
[Equation]
That is, autoregressive models will look to estimate the d conditional distributions p ( x i ∣ x i − 1 ,... ) .
A particular class of autoregressive models can be deﬁned by approximating conditional distributions through a mixture model, MM ( θ ( x i − 1 ,...,x 1 )) , with parameters depending on x i − 1 ,...,x 1 :
[Equation]
where f (⋅) is a fully connected network that may use a element-wise non-linearity on inputs, and g i (⋅) is some general mapping that computes a hidden state of features, h i ∈ R p , which help in modeling the conditional distribution of x i ∣ x
i − 1 ,...,x 1 .
One can control the ﬂexibility of the autoregressive model through g i .
It is important to be powerful enough to model our covariates while still generalizing.
In order to achieve this we propose two methods for modeling g i .
First, we propose the linear autoregressive model (LAM), using a straightforward linear map as g i (eq. 4):
[Equation]
where W ( i ) ∈
R p ×( i − 1 ) , b ∈ R p , and x < i =
( x i − 1 ,...,x 1 ) T .
Notwithstanding the simple form of (eq. 5), the resulting model is quite ﬂexible as it may model consecutive conditional problems p ( x i ∣ x i − 1 ,...,x 1 ) and p ( x i + 1 ∣ x i ,...,x 1 ) very differently.
Next we propose the recurrent autoregressive model (RAM), which features a recurrent relation between g i ’s. Given the expanding set of covariates progressively fed into g
i ’s, it is natural to consider a hidden state that evolves according to an RNN recurrence relationship:
[Equation]
where g ( x,s ) is a RNN function for updating one’s state based on an input x and prior state s (see Figure 2).
In the case of gated-RNNs, the model will be able to scan through previously seen dimensions remembering and forgetting information as needed for conditional densities without making any strong Markovian assumptions.
[Equation]
↘
[Equation]
↘ ⋮ ⋮
[Equation]
∅ → h 1 = g (∅ ,h 0 ) → ↓ x→ h 2 = g ( x 1 ,h 1 ) → ↓ x→ h 3 = g ( x 3 ,h 2 ) → ↓ ⋮ x d
[Equation]
→ h d = [Equation]
[Equation]
→
[Figure]
Both LAM and RAM are ﬂexible and able to adjust the hidden states, h i (eq. 4), to model the distinct conditional tasks p ( x i ∣ x i − 1 ,... ) .
However, there is a trade-off of added ﬂexibility and transferred information between the two models (see Figure 2).
LAM treats the conditional tasks for p ( x i ∣ x
i − 1 ,... ) and p ( x i + 1 ∣ x i ,... ) in a largely independent fashion.
This makes for a very ﬂexible model, however the parameter size is also large and there is no sharing of information among the conditional tasks.
On the other hand, RAM provides a framework for transfer learning among the conditional tasks by allowing the hidden state h i to evolve through the distinct conditional tasks.
This leads to fewer parameters and more sharing of information in respective tasks, but also yields less ﬂexibility since conditional estimates are tied, and may only change in a smooth fashion.
2. T RANSFORMATIONS
Several methods (Dinh et al., 2014; 2016; Goodfellow, 2016) have shown that the expressive power of very simple conditional densities (eq. 1) (such as independent Gaussians) can be greatly improved with transformations of variables.
Although the chain rule holds for arbitrary distributions, a limited amount of data and parameters limits the expressive power of models.
Hence, we expect that combining our conditional models with transformations of variables will also further increase ﬂexibility.
When using an invertible transformation of variables z = ( q 1 ( x ) ,...,q d ( x ))
∈ R d , one renormalizes the pdf of x as:
[Equation]
where ∣ det d q d x ∣ is a normalizing factor of the Jacobian of the transformation.
For analytical and computational considerations, we require transformations that are invertible, efﬁcient to compute and invert, and have a structured Jacobian matrix.
In order to meet these criteria we consider the following transformations.
Linear Transformation: First, we propose a linear transformation:
[Equation]
where we take A to be invertable.
Note that even though this linear transformation is simple, it includes permutations, and may also perform a PCA-like transformation, capturing coarse and highly varied features of the data before moving to more ﬁne grained details.
In order to not incur a high cost for updates, we wish to compute the determinant of the Jacobian efﬁciently.
We do so by directly working over an LU decomposition A = LU where L is a lower triangular matrix with unit diagonals and U is a upper triangular matrix with arbitrary diagonals.
As a function of L , U we have that det d z d x = ∏ di = 1 U ii ; hence we may efﬁciently optimize the parameters of the linear map.
Furthermore, inverting our mapping is also efﬁcient through solving two triangular matrix equations.
Recurrent Transformation: Recurrent neural networks are also a natural choice for variable transformations.
Due to their dependence on only previously seen dimensions, RNN transformations have triangular Jacobians, leading to simple determinants.
Furthermore, with an invertible output unit, their inversion is also straight-forward.
We consider the following form to an RNN transformation:
[Equation]
where r α is a leaky ReLU unit r α ( t )
=
I { t < 0 } αt
+ I { t ≥ 0 } t , r is a standard ReLU unit, s ∈ R ρ is the hidden state y , u , b
a are scalars, and w,v ∈ R ρ are vectors.
As compared to the linear transformation, the recurrent transformation is able to transform the input with different dynamics depending on its values.
Inverting (eq. 9) is a matter of inverting outputs and updating the hidden state (where the initial state s 0 is known and constant):
[Equation]
Furthermore, the determinant of the Jacobian for (eq. 9) is the product of diagonal terms:
[Equation]
where r ′ α ( t ) =
I { t > 0 } + α I { t < 0 } .
Recurrent Shift Transformation: It is worth noting that the rescaling brought on by the recurrent transformation effectively incurs a penalty through the log of the determinant (eq. 11).
However, one can still perform a transformation that depends on the values of covariates through a shift operation.
In particular, we propose an additive shift based on a recurrent function on prior dimensions:
[Equation]
where g is some recurrent function for updating states, and m is a fully connected network.
Inversion proceeds as before:
[Equation]
The Jacobian is again lower triangular, however due to the additive nature of (eq. 12), we have a unit diagonal.
Thus, det d z d x = 1 .
One interpretation of this transformation is that one can shift the value of x k based on x k − 1 ,x k − 2 ,... for better conditional density estimation without any penalty coming from the determinant term in (eq. 7).
Composing Transformations: Lastly, we considering stacking (i.e. composing) several transformations q = q ( 1 )
○ ...
○ q ( T ) and renormalizing:
[Equation]
where we take q ( 0 ) to be x .
We note that composing several transformations together allows one to leverage the respective strengths of each transformation.
Moreover, inserting a reversal transformation ( x 1 ,...,x d ↦ x d ,...,x 1 ) in between transformations yields biderectional relationships for several transformations.
2. C OMBINED A PPROACH
We combine the use of both transformations of variables and rich autoregressive models by: 1) writing the density of inputs, p ( x ) , as a normalized density of a transformation: p ( q ( x ))
(eq. 14).
Then we estimate the conditionals of p ( q ( x )) using an autoregressive model.
I.e. to learn our model we minimize the negative log likelihood:
[Equation]
which is obtained by substituting (eq. 2) into (eq. 14) with h i as deﬁned in (eq. 4).
R ELATED W ORK
Nonparametric density estimation has been a well studied problem in statistics and machine learning (Wasserman, 2007).
Unfortunately, nonparametric approaches like kernel density estimation suffer greatly from the curse of dimensionality and do not perform well when data does not have a small number of dimensions ( d ≲ 3 ).
To alleviate this, several semiparametric approaches have been explored.
Such approaches include forest density estimation (Liu et al., 2011), which assumes that the data has a forest (i.e. a collection of trees) structured graph.
This assumption leads to a density which factorizes in a ﬁrst order Markovian fashion through a tree traversal of the graph.
Another common semiparametric approach is to use a nonparanormal type model (Liu et al., 2009).
This approach uses a Gaussian copula with a rank-based transformation and a sparse precision matrix.
While both approaches are well-understood theoretically, their strong assumptions often lead to inﬂexible models.
In order to provide greater ﬂexibility with semiparametric models, recent work has employed deep learning for density estimation.
The use of neural networks for density estimation dates back to early work by Bishop (1994) and has seen success in areas like speech (Zen & Senior, 2014; Uria, 2015), music (Boulanger-Lewandowski et al., 2012), etc.. Typically such approaches use a network to learn to parameters of a parametric model for data.
Recent work has also explored the application of deep learning to build density estimates in image data (Oord et al., 2016; Dinh et al., 2016).
However, such approaches are heavily reliant on exploiting structure in neighboring pixels, often subsampling, reshaping or re-ordering data, and using convolutions to take advantage of neighboring correlations.
Modern approaches for general density estimation in real-valued data include Uria et al. Germain et al. (2015); Gregor et al.
Dinh et al. NADE (Uria et al., 2013) is an RBM-inspired density estimator with a weight-sharing scheme across conditional densities on covariates.
It may be written as a special case of LAM (eq. 5) with:
[Equation]
where W < i ∈ R p
×
i − 1 is the weight matrix compose of the ﬁrst i − 1 columns of a shared matrix
W = ( w 1 ,...w d ) .
We note also that LAM and NADE models are both related to fully visible sigmoid belief networks (Frey, 1998; Neal, 1992).
Even though the weight-sharing scheme in (eq. 16) reduces the number of parameters, it also greatly limits the types of distributions one can model.
Roughly speaking, the NADE weight-sharing scheme makes it difﬁcult to adjust conditional distributions when expanding the conditioning set with a covariate that has a small information gain.
We illustrate these kinds of limitations with a simple example.
Consider the following 3-dimensional distribution:
[Equation]
where C 0 .
5 is the 50% conﬁdence interval of a standard Gaussian distribution, and (cid:15) > 0 is some small constant.
That is, x 2 , and x 3 are marginally distributed as an equi-weighted bimodel mixture of Gaussian with means − 1 , 1 and 0 , 1 , respectively.
Due to NADE’s weight-sharing linear model, it will be difﬁcult to adjust h 2 and h 3 jointly to correctly model x 2 and x 3 respectively.
However, given their additional ﬂexibility, both LAM and RAM are able to adjust hidden states to remember and transform features as needed.
NICE models assume that data is drawn from a latent independent Gaussian space and transformed (Dinh et al., 2014).
The transformation uses several “additive coupling” shifting transformations on the second half of dimensions, using the ﬁrst half of dimensions.
That is, additive coupling proceeds by splitting inputs into halves x = ( x < d / 2 , x ≥ d / 2 ) , and transforming the second half as an additive function of the ﬁrst half:
[Equation]
18) where m (⋅) is the output of a fully connected network.
Inversion is simply a matter of subtraction x
=
( z < d / 2 , z ≥ d / 2 − m ( z < d / 2 )) .
The full transformation is the result of stacking several of these additive coupling layers together followed by a ﬁnal rescaling operation.
Furthermore, as with the RNN shift transformation, the additive nature of (eq. 18) yields a simple determinant, det d z d x = 1 .
We also note that are several methods for obtaining samples from an unknown distribution that by-pass density estimation.
For instance, generative adversarial networks (GANs) apply a (typically noninvertible) transformation of variables to a base distribution by optimizing a minimax loss over a discrimator and the transformation (Goodfellow, 2016).
Furthermore, one can also obtain samples with only limited information about the density of interest.
For example, if one has an unnormalized pdf, one may use Markov chain Monte Carlo (MCMC) methods to obtain samples (Neal, 1993).
E XPERIMENTS
We compare models using several experiments on synthetic and real-world datasets.
First, we compute the average log likelihoods on test data.
Then, to gain further context of the efﬁcacy of models, we also use their density estimates for anomaly detection, where we take low density instances to be outliers.
Moreover, we look at an illustrative MNIST image modeling task.
We study the performance of various combinations of conditional models and transformation.
That is, we consider various models for the conditionals p ( q i ( x )∣ h i ) and various transformations q (⋅) (eq. 15).
In particular the following conditional models were considered: LAM , RAM , Tied , MultiInd , and SingleInd .
Here, LAM , RAM , and Tied are as described in equations (eq. 5), (eq. 6), and (eq. 16), respectively.
MultiInd takes p ( q i ( x )∣ h i ) to be p ( q i ( x )∣ MM ( θ i )) , that is we shall use d distinct independent mixtures to model the transformed covariates.
Similarly, SingleInd takes p ( q i ( x )∣ h i ) to be p ( q i ( x )) , the density of a standard single component.
Moreover, we considered the following transformations:
None , RNN , 2xRNN , 4xAdd+Re , 4xSRNN+Re , RNN+4xAdd+Re , and RNN+4xSRNN+Re .
None indicates that no transformation of variables was performed.
RNN and 2xRNN performs a single recurrent transformation (eq. 9), and two recurrent transformations with a reversal permutation in between, respectively.
Following (Dinh et al., 2014), 4xAdd+Re performs four additive coupling transformations (eq. 18) with reversal permutations in between followed by a ﬁnal element-wise rescaling: x ↦ x ∗ exp ( s ) , where s is a learned variable.
Similarly, 4xSRNN+Re , performs four recurrent shift transformations (eq. 12) with reversal permutations in between, followed by an element-wise rescaling.
RNN+4xAdd+Re , and RNN+4xSRNN+Re are as before, but performing an initial recurrent transformation.
Furthermore, we also considered performing an initial linear transformation (eq. 8).
We ﬂag this by prepending an L to the transformation; e.g. L RNN denotes a linear transformation followed by a recurrent transformation.
Models were implemented in Tensorﬂow (Abadi et al., 2016).
Both RAM conditional models as well as the RNN shift transformation make use of the standard GRUCell GRU implementation 1 .
We take the mixture models of conditionals (eq. 2) to be mixtures of 40 Gaussians.
We optimize all models using the AdamOptimizer (Kingma & Ba, 2014) with an initial learning rate of 0 .
Training consisted of 30000 iterations, with mini-batches of size 256 .
The learning rate was decreased by a factor of 0 . 1 , or 0 .
5 (chosen via a validation set)
every 5000 iterations.
Gradient clipping with a norm of 1 was used.
After training, the best iteration according to the validation set loss was used to produce the test set mean log likelihoods.
4. S YNTHETIC
We perform a thorough empirical analysis over synthetic data.
By carefully constructing data we will be able to pinpoint strengths and short-comings of conditional models and transformations.
We study a dataset with a ﬁrst-order Markovian structure, and one with a star-shaped structure; they are described below.
4.1. M ARKOVIAN D ATA
First, we describe experiments performed on a synthetic dataset with a Markovian structure that features several exploitable correlations among covariates.
The dataset is sampled as follows:
y 1 ,y 2 ,y 3 ∼ N( 0 , 1 ) and y i ∣ y
i − 1 ,...,y 1 ∼ f
( i,y 1 ,y 2 ,y 3 )
+ (cid:15)
i for i > 3 where (cid:15) i ∼ N( (cid:15)
i − 1 ,σ ) , f
( i,y 1 ,y 2 ,x 3 ) =
y 1 sin ( y 2 g i + y 3 ) , and g i ’s are equi-spaced points on the unit interval.
That is, instances are sampled using random draws of amplitude, frequency, and shift covariates y 1 ,y 2 ,y 3 , which determine the mean of the other covariates, y 1 sin ( y 2 g i + y 3 ) , stemming from function evaluations on a grid, and random noise (cid:15)
i with a Gaussian random walk.
The resulting instances are easy to visualize, and contain many correlations among covariates (for instance, y 4 is highly informative of y 5 ).
To test robustness to correlations from distant (by index) covariates, we observe covariates that are shufﬂed using a ﬁxed permutation π chosen ahead of time: x = ( y π 1 ,...,y π d ) .
We take d = 32 , and the number of training instances to be 100000 .
We detail the mean log-likelihoods on a test set for TANs using various combinations of conditional models and transformations in Appendix, Table 4.
Note that the performance of previous oneprong approaches that considered a complex conditional model with simple or no transformation and vice-versa are illustrated by None & Tied (NADE), 4xAdd+Re & SingleInd (NICE) models, as well as by the entire row corresponding to None transformation and the MultiInd and SingleInd columns.
We see that both LAM and RAM conditionals are providing most of the top models.
We observe good samples from the best performing model (picked on validation dataset) as shown in Figure 3.
Here we also observe relatively good performance stemming from MultiInd conditionals with more complex transformations.
[Figure] [Figure]
4.1. S TAR D ATA
Next [Figure] we consider a dataset with a star-structured graphical model where fringe nodes are very uninformative with each-other.
We divide the covariates into disjoint center and vertex sets C = { 1 ,..., 4 } , V = { 5 ,...,d } respectively.
For center nodes j ∈ C , y j ∼ N( 0 , 1 ) .
Then, for j ∈ V , y j ∼ N( f j ( w Tj y C ) ,σ ) where f j is a ﬁxed step function with 32 intervals, w j ∈ R 4 is a ﬁxed vector, and y C
= ( y 1 ,y 2 ,y 3 ,y 4 ) .
We note that this dataset poses a difﬁcult density estimation problem since the distribution of each of the fringe vertices will be considerably different from each other, the fringe vertices are largely uninformative from one another, and the distribution of the fringe vertices are difﬁcult to estimate without conditioning on all the center nodes.
As before we observe covariates that are shufﬂed using a ﬁxed permutation π chosen ahead of time: x = ( y π 1 ,...,y π d ) , with d = 32 .
We detail the mean log-likelihoods on a test set for TANs using various combinations of conditional models and transformations in the Appendix, Table 5.
Once more we observe that both LAM and RAM conditionals are providing most of the top models.
In this dataset, however, simpler conditional methods are unable to model the data well, suggesting that the complicated dependencies need a two-prong TAN approach.
We observe a similar pattern when learning over data with d = 128 (see Appendix, Table 6). [Table] N=
[Table] L4xAdd+Re L4xSRNN+Re L4xAdd+Re
L2xRNN
L4xSRNN+Re
LRNN+4xAdd+Re
LRNN+4xSRNN+Re L4xSRNN+Re 4. T EST -D ATA L OG L IKELIHOODS We used multiple datasets from the UCI machine learning repository 2 and Stony Brook outlier detection datasets collection (ODDS) 3 to evaluate log-likelihoods on test data.
Broadly, the datasets can be divided into: Particle acceleration : higgs , hepmass , and susy datasets where generated for high-energy physics experiments using Monte Carlo simulations; Music : The music dataset contains timbre features from the million song dataset of mostly commercial western song tracks from the year 1922 to 2011; (Bertin-Mahieux et al., 2011).
Word2Vec : wordvecs consists of 3 million words from a Google News corpus.
Each word represented as a 300 dimensional vector trained using a word2vec model 4 .
ODDS datasets : We used several ODDS datasets– forest , pendigits , satimage2 .
These are multivariate datasets from varied set of sources meant to provide a broad picture of performance across anomaly detection tasks.
To not penalize models for low likelihoods on outliers in ODDS, we removed anomalies from test sets when reporting log-likelihoods.
As noted in (Dinh et al., 2014), data degeneracies and other corner-cases may lead to arbitrarily low negative log-likelihoods.
In order to avoid such complications, we remove discrete features, standardized all datasets, and add independent Gaussian noise with a standard deviation of 0 .
01 to training sets.
We report average test log-likelihoods in Table 1.
For each dataset and conditional model, we report the test log-likelihood of the top-2 transformations (picked on a validation dataset).
We note that the best performing model on each dataset had either LAM or RAM conditionals.
The tables detailing test log-likelihoods for all combinations of conditional models and transformations for each dataset may be found in the Appendix (see Tables 8-14).
We also observe that L RNN+4xAdd+Re & LAM and L RNN & RAM are consistently among the top-10 picked models.
4. A NOMALY D ETECTION
Next, we apply density estimates to anomaly detection.
Typically anomalies or outliers are datapoints that are unlikely given a dataset.
In terms of density estimations, such a task is framed by identifying which instances in a dataset have a low corresponding density.
That is, we shall label an instance x , as an anomaly if ˆ p ( x ) ≤ t , where t ≥ 0 is some threshold and ˆ p is the density estimate based on training data.
Note that this approach is trained in an unsupervised fashion.
However, each methods’ density estimates were evaluated on test data with anomaly/non-anomaly labels on instances.
We used thresholded log-likelihoods on the test set to compute precision and
[Table]
recall.
We use the average-precision metric: avg-prec =
∑ N test k = 1 precision r ( recall r − recall r − 1 ) where precision r = tp r tp r + fp r , recall r = tp r tp r + fn r , and tp r ,fp r ,fn r are true positive anomalies, false positives and false negative respectively among the bottom r log-likelihood instances in test data.
Our results are shown in Table 2.
We see that RAM performs the best on all three datasets.
Beyond providing another interesting use for our density estimates, seeing good performance in these outlier detection tasks further demonstrates that our models are learning semantically meaningful patterns.
4. D IGIT M ODELING
For illustrative purposes, we consider modeling MNIST digits.
In keeping with our focus of general data modeling, we treat each image as a ﬂattened vector of 784 dimensions.
Here we demonstrate that our proposed models can be used to model high dimensional data and produce coherent samples.
First, we model dequantized pixel values rescaled to the unit interval as described in (Dinh et al., 2014).
Moreover, we also model the MNIST digits through a logit transformation of pixel values.
That is, we take the dequantized pixel values in the range [ 0 , 256 ] , y and model: x = logit ( λ + ( 1 − λ ) y 256 ) , with λ = 0 .
This transformation will lessen boundary effects and keep pixel values inside a valid range.
We ran experiments on MNIST using the two models ( L RNN+4xAdd+Re & LAM and
L RNN & RAM ) that consistently appear in the top-10 in our previous experiments (see Tables 8in Appendix).
We observe test set average log-likelihoods and samples reported in bits per pixel in Table 3.
Furthermore, we plot samples in Figure 4.
We see that our models are able to capture the structure of MNIST digits, with very few artifacts in samples.
This is also reﬂected in the likelihoods, which are comparable or better than state-of-the-art.
[Table]
D ISCUSSION
We begin by noting the breadth of our proposed methods.
As mentioned above, previous approaches considered a complex conditional model with a simple or no transformation and vice-versa.
As such, some previous works have proposed a single new type of transformation, or a single new conditional model.
Here, we propose multiple methods for transformations (linear, recurrent, and shift recurrent) and multiple autoregressive conditional models (LAM, and RAM).
Furthermore, we consider the various combinations of transformations and autoregressive models, most of which constitute a novel TAN.
We draw several conclusions through our comprehensive empirical study.
First, we consider our experiments on synthetic data.
Methods that only consider complex transformations or condition models are illustrated in the entire row corresponding to the None transformation and the MultiInd [Figure] and SingleInd columns, respectively.
The performance of some of these models, which include None & Tied (NADE), 4xAdd+Re & SingleInd (NICE), was moderate on the Markovian data, however these one-prong approaches fail in the star dataset.
Overall LAM and RAM methods provided considerable improvements, especially in the star dataset, where the ﬂexibility of LAM made it possible to learn the widely different conditional probabilities present in the data.
Similarly, we observe that the best performing models in real-world datasets are those that incorporate a ﬂexible transformation and conditional model.
In fact, the best model (according to validation dataset) always has LAM or RAM autoregressive components.
Hence, validation across models would always select one of these methods.
In fact, 95% of top-10 models (aggregated across all datasets) have a LAM and RAM conditional model (see Tables 8-14).
It is interesting to see that many of these top models also contain a linear transformation.
Of course, linear transformations of variables are common to most parametric models, however they have been under-explored in the context of autoregressive density estimation.
Our methodology for efﬁciently learning linear transformations coupled with their strong empirical performance encourages their inclusion in autoregressive models.
Finally, we digest results over the real-world datasets by computing the percentage of the top likelihood achieved by each transformation t , and conditional model m , in dataset D : s ( t,m,D ) =
exp ( l t,m,D )/ max a,b exp ( l a,b,D ) , where l t,m,D is the test log-likelihood for t,m on D .
We then average S over the datasets: S ( t,m ) =
1 T ∑ D S ( t,m,D ) , where T is the total number of datasets.
We show this score in the Appendix, Table 7.
This table gives a summary of which models performed better (closer to the best performing model per dataset) over multiple datasets.
We see that RAM conditional with L RNN transformation, and LAM conditional with L RNN+4xAdd+Re were the two best performers.
C ONCLUSION
In conclusion this work jointly leverages transformations of variables and autoregressive models and proposes novel methods for both.
We show a considerable improvement with our methods through a comprehensive study over both real world and synthetic data.
Also, we illustrate the utility of our models in outlier detection and digit modeling tasks
A PPENDIX
[Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] 4xAdd+Re − 19 .−
11 .− 12 .− 21 .− 22 .
L RNN+4xAdd+Re − 20 .− 12 .− 12 .− 26 .− 28 .
RNN+4xSRNN+Re − 13 .− 3 .− 6 .− 15 .− 16 . L RNN+4xSRNN+Re − 20 .− 12 .− 7 .− 18 .− 18 .
[Table] [Table]