Transformation Autoregressive Networks
Anonymous authors
Paper under double-blind review
Abstract
The fundamental task of general density estimation has been of keen interest to machine learning.
Recent advances in density estimation have either: a ) proposed a ﬂexible model to estimate the conditional factors of the chain rule, p ( x i ∣ x i − 1 ,... ) ; or b ) used ﬂexible, non-linear transformations of variables of a simple base distribution.
Instead, this work jointly leverages transformations of variables and autoregressive models, and proposes novel methods for both.
We provide a deeper understanding of our methods, showing a considerable improvement through a comprehensive study over both real world and synthetic data.
Moreover, we illustrate the use of our models in outlier detection and image modeling tasks
Introduction
Density estimation is at the core of a multitude of machine learning applications.
However, this fundamental task, which encapsulates the understanding of data, is diﬃcult in the general setting due to issues like the curse of dimensionality.
Furthermore, general data, unlike spatial/temporal data, does not contain a priori known correlations among covariates that may be exploited and engineered with.
For example, image data has known correlations among neighboring pixels that may be hard-coded into a model, whereas one must ﬁnd such correlations in a data-driven fashion with general data.
In order to model high dimensional data, a large number of methods have considered autoregressive models, which model the conditional factors of the chain rule (23; 22; 11; 13).
I.e these models estimate the conditionals: p ( x i ∣ x i − 1 ,...,x 1 ) , for i ∈ { 1 ,...,d } .
While some methods directly model the conditionals, p ( x i ∣ x i − 1 ,... ) using sophisticated semiparametric density estimates, other methods apply sophisticated transformations of variables x ↦ z and take the conditionals over z to be a restricted, often independent base distribution p ( z i ∣ z
i − 1 ,... )
≈ f
( z i ) (8; 9; 12).
In this paper we take a step back from these previous approaches that have considered either: a ) a ﬂexible autoregressive scheme with simple or no transformations of variables (Figure 1a); or b ) a simple autoregressive scheme with ﬂexible transformations of variables.
(Figure 1b)
We leverage both of these approaches (Figure 1c), develop novel methods for each, and show a considerable improvement with their combination.
Contributions:
The following are our contributions.
First, we propose two ﬂexible autoreggressive models for modeling conditional distributions: the linear autoregressive model (LAM), and the recurrent autoregressive model (RAM).
LAM employs a simple linear form to condition on previously seen covariates in a ﬂexible fashion.
RAM uses a recurrent neural network (RNN) to evolve conditioning features as the set of conditioning covariates expands.
Furthermore, this paper proposes several novel transformations of variables: 1) we propose an eﬃcient method for learning a linear transfromations on covariates; 2) we develop an invertible RNN-based transformation that directly acts on covariates; 3) we also propose an additive RNN-base transformation.
To better capture correlations in general data, we combine our novel autoreggresive models and transformations of variables.
That is, we propose transformation autogressive networks (TANs) that use a ﬂexible transformation over the original covariates and a powerful autoregressive model to estimate the conditional probabilities over the transformed space.
To assess the eﬃcacy of models, we performed a comprehensive evaluation of autoregressive models and transformations.
Extensive experiments on both real and synthetic datasets show the power of TANs for capturing complex dependencies between the covariates.
Moreover, we show that the learned model can be used for outlier detection, and image modeling.
The remainder of the paper is structured as follows.
First, in Section 2.1 we present two novel methods for modeling condition distributions across covariates.
Next, in Section 2.2, we describe several transformations to use in conjunction with our proposed conditional models.
After, we discuss related work and contrast our approach to previous methods.
We then illustrate the eﬃcacy of our methods with both synthetic and real-world data.
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation] [Figure] [Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
Transformation Autoregressive Networks
First, we propose two autoregressive models to estimate the conditional distribution of input covariates x ∈ R d .
After, we shall show how we may use such models over a transformation z
= q ( x ) , while renormalizing to obtain density values for x .
2. Autoregressive Models
Autoregressive models decompose density estimation of a multivariate variable x ∈ R d into multiple conditional tasks on a growing set of inputs through the chain rule:
[Equation]
That is, autoregressive models will look to estimate the d conditional distributions p ( x i ∣ x i − 1 ,... ) .
A particular class of autoregressive models can be deﬁned by approximating conditional distributions through a mixture model, MM ( θ ( x i − 1 ,...,x 1 )) , with parameters depending on x i − 1 ,...,x 1 :
[Equation]
where f (⋅) is a fully connected network that may use a element-wise non-linearity on inputs, and g i (⋅) is some general mapping that computes a hidden state of features, h i , which help in modeling the conditional distribution of x i ∣ x
i − 1 ,...,x 1 .
One can control the ﬂexibility of the autoregressive model through g i .
It is important to be powerful enough to model our covariates while still generalizing.
In order to achieve this we propose two methods for modeling g i .
First, we propose the linear autoregressive model (LAM), using a straightforward linear map as g i (eq. 4):
[Equation]
where W ( i ) ∈
R p ×( i − 1 ) , b ∈ R p , and x < i =
( x i − 1 ,...,x 1 ) T .
Notwithstanding the simple form of (eq. 5), the resulting model is quite ﬂexible as it may model consecutive conditional problems p ( x i ∣ x i − 1 ,...,x 1 ) and p ( x i + 1 ∣ x i ,...,x 1 ) very diﬀerently.
Next we propose the recurrent autoregressive model (RAM), which features a recurrent relation between g i ’s. Given the expanding set of covariates progressively fed into g
i ’s, it is natural to consider a hidden state that evolves according to an RNN recurrence relationship:
[Equation]
where g ( x,s ) is a RNN function for updating one’s state based on an input x and prior state s (see Figure 2).
In the case of gated-RNNs, the model will be able to scan through previously seen dimensions remembering and forgetting information as needed for conditional densities without making any strong Markovian assumptions.
[Equation]
x
[Equation]
h
[Equation]
h
[Equation]
→ ↓ ⋮ x d −→ h d = g ( x d −
[Equation]
h d − 1 )
[Figure]
are updated and then used to compute the parameters of the next conditional density for x k .
Note that in LAM the hidden states h j ’s are not tied together, where in RAM the hidden state h j along with x j are used to compute the hidden state h j + 1 which determines the parameters of p ( x j + 1 ∣ h j + 1 ) .
Both LAM and RAM are ﬂexible and able to adjust the hidden states, h i (eq. 4), to model the distinct conditional tasks p ( x i ∣ x i − 1 ,... ) .
However, there is a trade-oﬀ of added ﬂexibility and transferred information between the two models (see Figure 2).
LAM treats the conditional tasks for p ( x i ∣ x
i − 1 ,... ) and p ( x i + 1 ∣ x i ,... ) in a largely independent fashion.
This makes for a very ﬂexible model, however the parameter size is also large and there is no sharing of information among the conditional tasks.
On the other hand, RAM provides a framework for transfer learning among the conditional tasks by allowing the hidden state h i to evolve through the distinct conditional tasks.
This leads to fewer parameters and more sharing of information in respective tasks, but also yields less ﬂexibility since conditional estimates are tied, and may only change in a smooth fashion.
2. Transformations
Several methods (8; 9; 12) have shown that the expressive power of very simple conditional densities (eq. 1) (such as independent Gaussians) can be greatly improved with transformations of variables.
Although the chain rule holds for arbitrary distributions, a limited amount of data and parameters limits the expressive power of models.
Hence, we expect that combining our conditional models with transformations of variables will also further increase ﬂexibility.
When using an invertible transformation of variables z = ( q 1 ( x ) ,...,q d ( x ))
∈ R d , one renormalizes the pdf of x as:
[Equation]
where ∣ det d q d x ∣ is a normalizing factor of the Jacobian of the transformation.
For analytical and computational considerations, we require transformations that are invertible, eﬃcient to compute and invert, and have a structured Jacobian matrix.
In order to meet these criteria we consider the following transformations.
Linear Transformation: First, we propose a linear transformation:
[Equation]
where we take A to be invertable.
Note that even though this linear transformation is simple, it includes permutations, and may also perform a PCA-like transformation, capturing coarse and highly varied features of the data before moving to more ﬁne grained details.
In order to not incur a high cost for updates, we wish to compute the determinant of the Jacobian eﬃciently.
We do so by directly working over an LU decomposition A = LU where L is a lower triangular matrix with unit diagonals and U is a upper triangular matrix with arbitrary diagonals.
As a function of L , U we have that det d z d x = ∏ di = 1 U ii ; hence we may eﬃciently optimize the parameters of the linear map.
Furthermore, inverting our mapping is also eﬃcient through solving two triangular matrix equations.
Recurrent Transformation: Recurrent neural networks are also a natural choice for variable transformations.
Due to their dependence on only previously seen dimensions, RNN transformations have triangular Jacobians, leading to simple determinants.
Furthermore, with an invertible output unit, their inversion is also straight-forward.
We consider the following form to an RNN transformation:
[Equation]
where r α is a leaky ReLU unit r α ( t )
=
I { t < 0 } αt
+ I { t ≥ 0 } t , r is a standard ReLU unit, s ∈ R ρ is the hidden state y , u , b
a are scalars, and w,v ∈ R ρ are vectors.
As compared to the linear transformation, the recurrent transformation is able to transform the input with diﬀerent dynamics depending on its values.
Inverting (eq. 9) is a matter of inverting outputs and updating the hidden state (where the initial state s 0 is known and constant):
[Equation]
Furthermore, the determinant of the Jacobian for (eq. 9) is the product of diagonal terms:
[Equation]
where r ′ α ( t ) =
I { t > 0 } + α I { t < 0 } .
Recurrent Shift Transformation: It is worth noting that the rescaling brought on by the recurrent transformation eﬀectively incurs a penalty through the log of the determinant (eq. 11).
However, one can still perform a transformation that depends on the values of covariates through a shift operation.
In particular, we propose an additive shift based on a recurrent function on prior dimensions:
[Equation]
where g is some recurrent function for updating states, and m is a fully connected network.
Inversion proceeds as before:
[Equation]
The Jacobian is again lower triangular, however due to the additive nature of (eq. 12), we have a unit diagonal.
Thus, det d z d x = 1 .
One interpretation of this transformation is that one can shift the value of x k based on x k − 1 ,x k − 2 ,... for better conditional density estimation without any penalty coming from the determinant term in (eq. 7).
Composing Transformations: Lastly, we considering stacking (i.e. composing) several transformations q = q ( 1 )
○ ...
○ q ( T ) and renormalizing:
[Equation]
where we take q ( 0 ) to be x .
We note that composing several transformations together allows one to leverage the respective strengths of each transformation.
Moreover, inserting a reversal transformation ( x 1 ,...,x d ↦ x d ,...,x 1 ) in between transformations yields biderectional relationships for several transformations.
2. Combined Approach
We combine the use of both transformations of variables and rich autoregressive models by: 1) writing the density of inputs, p ( x ) , as a normalized density of a transformation: p ( q ( x ))
(eq. 14).
Then we estimate the conditionals of p ( q ( x )) using an autoregressive model.
I.e. to learn our model we minimize the negative log likelihood:
[Equation]
which is obtained by substituting (eq. 2) into (eq. 14) with h i as deﬁned in (eq. 4).
Related Work
Nonparametric density estimation has been a well studied problem in statistics and machine learning (24).
Unfortunately, nonparametric approaches like kernel density estimation suﬀer greatly from the curse of dimensionality and do not perform well when data does not have a small number of dimensions ( d ≲ 3 ).
To alleviate this, several semiparametric approaches have been explored.
Such approaches include forest density estimation (16), which assumes that the data has a forest (i.e. a collection of trees) structured graph.
This assumption leads to a density which factorizes in a ﬁrst order Markovian fashion through a tree traversal of the graph.
Another common semiparametric approach is to use a nonparanormal type model (15).
This approach uses a Gaussian copula with a rank-based transformation and a sparse precision matrix.
While both approaches are well-understood theoretically, their strong assumptions often lead to inﬂexible models.
In order to provide greater ﬂexibility with semiparametric models, recent work has employed deep learning for density estimation.
The use of neural networks for density estimation dates back to early work by (6) and has seen success in areas like speech (25; 21), music (7), etc.. Typically such approaches use a network to learn to parameters of a parametric model for data.
Recent work has also explored the application of deep learning to build density estimates in image data (19; 9).
However, such approaches are heavily reliant on exploiting structure in neighboring pixels, often subsampling, reshaping or re-ordering data, and using convolutions to take advantage of neighboring correlations.
Modern approaches for general density estimation in real-valued data include (23; 22; 11; 13).
NADE (22) is an RBM-inspired density estimator with a weight-sharing scheme across conditional densities on covariates.
It may be written as a special case of LAM (eq. 5) with:
[Equation]
) where W < i ∈ R p × i − 1 is the weight matrix compose of the ﬁrst i − 1 columns of a shared matrix
W = ( w 1 ,...w d ) .
We note also that LAM and NADE models are both related to fully visible sigmoid belief networks (10; 17)
Even though the weight-sharing scheme in (eq. 16) reduces the number of parameters, it also greatly limits the types of distributions one can model.
Roughly speaking, the NADE weight-sharing scheme makes it diﬃcult to adjust conditional distributions when expanding the conditioning set with a covariate that has a small information gain.
We illustrate these kinds of limitations with a simple example.
Consider the following 3-dimensional distribution:
[Equation]
where C 0 .
5 is the 50% conﬁdence interval of a standard Gaussian distribution, and (cid:15) > 0 is some small constant.
That is, x 2 , and x 3 are marginally distributed as an equi-weighted bimodel mixture of Gaussian with means − 1 , 1 and 0 , 1 , respectively.
Due to NADE’s weight-sharing linear model, it will be diﬃcult to adjust h 2 and h 3 jointly to correctly model x 2 and x 3 respectively.
However, given their additional ﬂexibility, both LAM and RAM are able to adjust hidden states to remember and transform features as needed.
NICE models assume that data is drawn from a latent independent Gaussian space and transformed (8).
The transformation uses several “additive coupling” shifting transformations on the second half of dimensions, using the ﬁrst half of dimensions.
That is, additive coupling proceeds by splitting inputs into halves x = ( x <
[Equation]
x
[Equation]
2 ) , and transforming the second half as an additive function of the ﬁrst half:
[Equation]
where m (⋅) is the output of a fully connected network.
Inversion is simply a matter of subtraction x
=
( z < d / 2 , z ≥ d / 2 − m ( z < d / 2 )) .
The full transformation is the result of stacking several of these additive coupling layers together followed by a ﬁnal rescaling operation.
Furthermore, as with the RNN shift transformation, the additive nature of (eq. 18) yields a simple determinant, det d z d x = 1 .
We also note that are several methods for obtaining samples from an unknown distribution that by-pass density estimation.
For instance, generative adversarial networks (GANs) apply a (typically noninvertible) transformation of variables to a base distribution by optimizing a minimax loss over a discrimator and the transformation (12).
Furthermore, one can also obtain samples with only limited information about the density of interest.
For example, if one has an unnormalized pdf, one may use Markov chain Monte Carlo (MCMC) methods to obtain samples (18).
Experiments
We compare models using several experiments on synthetic and real-world datasets.
First, we compute the average log likelihoods on test data.
Then, to gain further context of the eﬃcacy of models, we also use their density estimates for anomaly detection, where we take low density instances to be outliers.
Moreover, we look at an illustrative MNIST image modeling task.
We study the performance of various combinations of conditional models and transformation.
That is, we consider various models for the conditionals p ( q i ( x )∣ h i ) and various transformations q (⋅) (eq. 15).
In particular the following conditional models were considered:
LAM , RAM , Tied , MultiInd , and SingleInd .
Here, LAM , RAM , and Tied are as described in equations (eq. 5), (eq. 6), and (eq. 16), respectively.
MultiInd takes p ( q i ( x )∣ h i ) to be p ( q i ( x )∣ MM ( θ i )) , that is we shall use d distinct independent mixtures to model the transformed covariates.
Similarly, SingleInd takes p ( q i ( x )∣ h i ) to be p ( q i ( x )) , the density of a standard single component.
Moreover, we considered the following transformations:
None ,
RNN , 2xRNN , 4xAdd+Re , 4xSRNN+Re , RNN+4xAdd+Re , and RNN+4xSRNN+Re .
None indicates that no transformation of variables was performed.
RNN and 2xRNN performs a single recurrent transformation (eq. 9), and two recurrent transformations with a reversal permutation in between, respectively.
Following (8), 4xAdd+Re performs four additive coupling transformations (eq. 18) with reversal permutations in between followed by a ﬁnal element-wise rescaling: x ↦ x ∗ exp ( s ) , where s is a learned variable.
Similarly, 4xSRNN+Re , performs four recurrent shift transformations (eq. 12) with reversal permutations in between, followed by an element-wise rescaling.
RNN+4xAdd+Re , and RNN+4xSRNN+Re are as before, but performing an initial recurrent transformation.
Furthermore, we also considered performing an initial linear transformation (eq. 8).
We ﬂag this by prepending an L to the transformation;
e.g. L
RNN denotes a linear transformation followed by a recurrent transformation.
Models were implemented in Tensorﬂow (4).
Both RAM conditional models as well as the
RNN shift transformation make use of the standard GRUCell GRU implementation 1 .
We take the mixture models of conditionals (eq. 2) to be mixtures of 40 Gaussians.
We optimize all models using the AdamOptimizer (14) with an initial learning rate of 0 .
Training consisted of 30000 iterations, with mini-batches of size 256 .
The learning rate was decreased by a factor of 0 . 1 , or 0 .
5 (chosen via a validation set)
every 5000 iterations.
Gradient clipping with a norm of 1 was used.
After training, the best iteration according to the validation set loss was used to produce the test set mean log likelihoods.
4. Synthetic
We perform a thorough empirical analysis over synthetic data.
By carefully constructing data we will be able to pinpoint strengths and short-comings of conditional models and transformations.
We study a dataset with a ﬁrst-order Markovian structure, and one with a star-shaped structure; they are described below.
4.1. Markovian Data
First, we describe experiments performed on a synthetic dataset with a Markovian structure that features several exploitable correlations among covariates.
The dataset is sampled as follows:
y 1 ,y 2 ,y 3 ∼ N( 0 , 1 ) and y i ∣ y
i − 1 ,...,y 1 ∼ f
( i,y 1 ,y 2 ,y 3 )
+ (cid:15)
i for i > 3 where (cid:15) i ∼ N( (cid:15)
i − 1 ,σ ) , f
( i,y 1 ,y 2 ,x 3 ) =
y 1 sin ( y 2 g i + y 3 ) , and g i ’s are equi-spaced points on the unit interval.
That is, instances are sampled using random draws of amplitude, frequency, and shift covariates y 1 ,y 2 ,y 3 , which determine the mean of the other covariates, y 1 sin ( y 2 g i + y 3 ) , stemming from function evaluations on a grid, and random noise (cid:15)
i with a Gaussian random walk.
The resulting instances are easy to visualize, and contain many correlations among covariates (for instance, y 4 is highly informative of y 5 ).
To test robustness to correlations from distant (by index) covariates, we observe covariates that are shuﬄed using a ﬁxed permutation π chosen ahead of time: x = ( y π 1 ,...,y π d ) .
We take d = 32 , and the number of training instances to be 100000 .
We detail the mean log-likelihoods on a test set for TANs using various combinations of conditional models and transformations in Appendix, Table 4.
Note that the performance of previous one-prong approaches that considered a complex conditional model with simple or no transformation and vice-versa are illustrated by None & Tied (NADE), 4xAdd+Re & SingleInd (NICE) models, as well as by the entire row corresponding to None transformation and the MultiInd and SingleInd columns.
We see that both LAM and
RAM conditionals are providing most of the top models.
We observe good samples from the best performing model (picked on validation dataset) as shown in Figure 3.
Here we also observe relatively good performance stemming from MultiInd conditionals with more complex transformations.
[Figure] [Figure]
4.1. Star Data
Next, we consider a dataset with a star-structured graphical model where fringe nodes are very uninformative with each-other.
We divide the covariates into disjoint center and vertex sets C = { 1 ,..., 4 } , V = { 5 ,...,d } respectively.
For center nodes j ∈ C , y j ∼ N( 0 , 1 ) .
Then, for j ∈ V , y j ∼ N( f j ( w T j y C ) ,σ ) where f j is a ﬁxed step function with 32 intervals, w j ∈ R 4 is a ﬁxed vector, and y C
= ( y 1 ,y 2 ,y 3 ,y 4 ) .
We note that this dataset poses a diﬃcult density estimation problem since the distribution of each of the fringe vertices will be considerably diﬀerent from each other, the fringe vertices are largely uninformative from one another, and the distribution of the fringe vertices are diﬃcult to estimate without conditioning on all the center nodes.
As before we observe covariates that are shuﬄed using a ﬁxed permutation π chosen ahead of time: x = ( y π 1 ,...,y π d ) , with d = 32 .
We detail the mean log-likelihoods on a test set for TANs using various combinations of conditional models and transformations in the Appendix, Table 5.
Once more we observe that both LAM and RAM conditionals are providing most of the top models.
In this dataset, however, simpler conditional methods are unable to model the data well, suggesting that the complicated dependencies need a two-prong TAN approach.
We observe a similar pattern when learning over data with d = 128 (see Appendix, Table 6).
4. Test-Data Log Likelihoods
We used multiple datasets from the UCI machine learning repository (2) and Stony Brook outlier detection datasets collection (ODDS) (1) to evaluate log-likelihoods on test data.
Broadly, the datasets can be divided into: Particle acceleration : higgs , hepmass , and susy datasets where generated for high-energy physics experiments using Monte Carlo simulations;
Music : The music dataset contains timbre features from the million song dataset of mostly commercial western song tracks from the year 1922 to 2011; (5).
Word2Vec : wordvecs consists of 3 million words from a Google News corpus.
Each word represented as adimensional vector trained using a word2vec model (3).
ODDS datasets : We used several
ODDS datasets– forest , pendigits , satimage2 .
These are multivariate datasets from varied set of sources meant to provide a broad picture of performance across anomaly detection tasks.
To not penalize models for low likelihoods on outliers in ODDS, we removed anomalies from test sets when reporting log-likelihoods.
As noted in (8), data degeneracies and other corner-cases may lead to arbitrarily low negative log-likelihoods.
In order to avoid such complications, we remove discrete features, standardized all datasets, and add independent Gaussian noise with a standard deviation of 0 .
01 to training sets.
We report average test log-likelihoods in Table 1.
For each dataset and conditional model, we report the test log-likelihood of the top-2 transformations (picked on a validation dataset).
We note that the best performing model on each dataset had either LAM or RAM conditionals.
The tables detailing test log-likelihoods for all combinations of conditional models and transformations for each dataset may be found in the Appendix (see Tables 8-14).
We also observe that L RNN+4xAdd+Re & LAM and L RNN & RAM are consistently among the toppicked models.
4. Anomaly Detection
Next, we apply density estimates to anomaly detection.
Typically anomalies or outliers are data-points that are unlikely given a dataset.
In terms of density estimations, such a task is framed by identifying which instances in a dataset have a low corresponding density.
That is, we shall label an instance x , as an anomaly if ˆ p ( x ) ≤ t , where t ≥ 0 is some threshold and ˆ p is the density estimate based on training data.
Note that this approach is trained in an unsupervised fashion.
However, each methods’ density estimates were evaluated on test data with anomaly/non-anomaly labels on instances.
We used thresholded log-likelihoods on the test set to compute precision and recall.
We use the average-precision metric: avg-prec =
∑ N test k = 1 precision r ( recall r − recall r − 1 ) where precision r =
[Equation]
r , recall
[Equation]
=
[Equation]
, and tp r ,fp r ,fn r are true positive anomalies, false positives and false negative respectively among the bottom r log-likelihood instances in test data.
Our results are shown in Table 2.
We see that RAM performs the best on all three datasets.
Beyond providing another interesting use for our density estimates, seeing good performance in these outlier detection tasks further demonstrates that our models are learning semantically meaningful patterns. [Table] [Table] None
RNN
[Table]
2xRNN
[Table]
RNN 2xRNN 4xSRNN+
[Table]
None
[Table]
4. Digit Modeling
For illustrative purposes, we consider modeling MNIST digits.
In keeping with our focus of general data modeling, we treat each image as a ﬂattened vector of 784 dimensions.
Here we demonstrate that our proposed models can be used to model high dimensional data and produce coherent samples.
First, we model dequantized pixel values rescaled to the unit interval as described in (8).
Moreover, we also model the MNIST digits through a logit transformation of pixel values.
That is, we take the dequantized pixel values in the range [ 0 , 256 ] , y and model: x = logit ( λ + ( 1 − λ ) y 256 ) , with λ = 0 .
This transformation will lessen boundary eﬀects and keep pixel values inside a valid range.
We ran experiments on MNIST using the two models ( L RNN+4xAdd+Re & LAM and
L RNN & RAM ) that consistently appear in the top-10 in our previous experiments (see
Tables 8-14 in Appendix).
We observe test set average log-likelihoods and samples reported in bits per pixel in Table 3.
Furthermore, we plot samples in Figure 4.
We see that our models are able to capture the structure of MNIST digits, with very few artifacts in samples.
This is also reﬂected in the likelihoods, which are comparable or better than state-of-the-art.
[Table]
[Figure]
Discussion
We begin by noting the breadth of our proposed methods.
As mentioned above, previous approaches considered a complex conditional model with simple or no transformation and vice-versa.
As such, some previous works have proposed a single new type of transformation, or a single new conditional model.
Here, we propose multiple methods for transformations (linear, recurrent, and shift recurrent) and multiple autoregressive conditional models (LAM, and RAM).
Furthermore, we consider the various combinations of transformations and autoregressive models, most of which constitute a novel TAN.
We draw several conclusions through our comprehensive empirical study.
First, we consider our experiments on synthetic data.
Methods that only consider complex transformations or condition models are illustrated in the entire row corresponding to None transformation and the MultiInd and SingleInd columns, respectively.
Performance of some of these models, which include None & Tied (NADE), 4xAdd+Re & SingleInd (NICE), can be moderately well on the Markovian data, however these one-prong approaches fail in the star dataset.
Overall, LAM and RAM methods provided considerable improvements.
This was especially the case in the star dataset, where the ﬂexibility of LAM made it possible to learn the widely diﬀerent conditional probabilities present in the data.
Similarly, we observe that the best performing models in real-world datasets are those that incorporate a ﬂexible transformation and conditional model.
In fact, the best model (according to validation dataset) always has LAM or RAM autoregressive components.
Hence, validation across models would always select one of these methods.
In fact, 95% of top10 models (aggregated across all datasets) have a LAM and RAM conditional model (see
Tables 8-14).
It is interesting to see that many of these top models also contain a linear transformation.
Of course, linear transformations of variables are common to most parametric models, however they have been under-explored in the context of autoregressive density estimation.
Our methodology for eﬃciently learning linear transformations coupled with their strong empirical performance encourages their inclusion in autoregressive models.
Finally, we digest results over the real-world datasets by computing the percentage of the top likelihood achieved by each transformation t , and conditional model m , in dataset D : s ( t,m,D ) =
exp ( l t,m,D )/ max a,b exp ( l a,b,D ) , where l t,m,D is the test log-likelihood for t,m on D .
We then average S over the datasets: S ( t,m ) =
1 T ∑ D S ( t,m,D ) , where T is the total number of datasets.
We show this score in the Appendix, Table 7.
This table gives a summary of which models performed better (closer to the best performing model per dataset) over multiple datasets.
We see that RAM conditional with L RNN transformation, and LAM conditional with L RNN+4xAdd+Re were the two best performers.
Conclusion:
In conclusion, this work jointly leverages transformations of variables and autoregressive models, and proposes novel methods for both.
We show a considerable improvement with our methods, through a comprehensive study over both real world and synthetic data.
Also, we illustrate the utility of our models in outlier detection and digit modeling tasks.
Appendix
[Table]
[Table] [Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]