N EURAL A PPROXIMATION OF AN A UTO -R EGRESSIVE P ROCESS THROUGH C ONFIDENCE G UIDED S AMPLING
Anonymous authors
Paper under double-blind review
A BSTRACT
We propose a generic conﬁdence-based approximation that can be plugged in and simplify the auto-regressive generation process with a proved convergence.
We ﬁrst assume that the priors of future samples can be generated in an independently and identically distributed (i.i.d.)
manner using an efﬁcient predictor.
Given the past samples and future priors, the mother AR model can post-process the priors while the accompanied conﬁdence predictor decides whether the current sample needs a resampling or not.
Thanks to the i.i.d.
assumption, the post-processing can update each sample in a parallel way, which remarkably accelerates the mother model.
Our experiments on different data domains including sequences and images show that the proposed method can successfully capture the complex structures of the data and generate the meaningful future samples with lower computational cost while preserving the sequential relationship of the data.
I NTRODUCTION
The auto-regressive (AR) model, which infers and predicts the causal relationship between the previous and future samples in a sequential data, has been widely studied since the beginning of machine learning research.
The recent advances of the auto-regressive model brought by the neural network have achieved impressive success in handling complex data including texts (Sutskever et al., 2011), audio signals (Vinyals et al., 2012; Tamamori et al., 2017; van den Oord et al., 2016a), and images (van den Oord et al., 2016b; Salimans et al., 2017).
It is well known that AR model can learn a tractable data distribution p ( x ) and can be easily extended for both discrete and continuous data.
Due to their nature, AR models have especially shown a good ﬁt with a sequential data, such as voice generation (van den Oord et al., 2016a) and provide a stable training while they are free from the mode collapsing problem (van den Oord et al., 2016b).
However, these models must infer each element x i of the data x = [ x 1 , x 2 , · · · , x i , · · · , x N ] in a serial manner, requiring O ( N ) times more than the other non-sequential estimators, which outputs x at once (Garnelo et al., 2018; Kim et al., 2019; Kingma & Welling, 2014; Goodfellow et al., 2014).
Moreover, it is difﬁcult to employ recent parallel computation because AR models always require a previous time step by deﬁnition.
This mostly limits the use of the AR models in practice despite their advantages.
To resolve the problem, we introduce a new and generic approximation method, Neural AutoRegressive model Approximator (NARA) , which can be easily plugged into any AR model.
We show that NARA can reduce the generation complexity of AR models by relaxing an inevitable AR nature and enables AR models to employ the powerful parallelization techniques in the sequential data generation, which was difﬁcult previously.
NARA consists of three modules; (1) a prior-sample predictor, (2) a conﬁdence predictor, and (3) the original AR model.
To relax the AR nature, given a set of past samples, we ﬁrst assume that each sample of the future sequence can be generated in an independent and identical manner.
Thanks to the i.i.d.
assumption, using the ﬁrst module of NARA, we can sample a series of future priors and these future priors are post-processed by the original AR model, generating a set of raw predictions.
The conﬁdence predictor evaluates the credibility of these raw samples and decide whether the model needs re-sampling or not.
The conﬁdence predictor plays an important role in that the approximation errors can be accumulated during the sequential AR generation process if the AR model Parallel Inference
ConfidencePredictor g V
Parallel
Inference p(x i+1 | x ≤i ) ν i+1 (≥ ε) Accept p(x i+2 | x ≤i ,m i+1 ) ν i+2 (≥ ε) Accept p(x i+j | x ≤i , m (i,i+ j-1] ) ν i+j (< ε) Re-sample [ x 1 , … , x i , m i+1 , … , m i+M ] p(x i+M | x ≤i , m (i,i+ M-1] )
Input: x ≤i =
[ x 1 , x 2 , … , x i ] Prior values: m (i,i+ M] = [ m i+1 , … , m i+M ] Prior-sample Predictor f W ν i+M (< ε) Re-sample erroneous samples with low conﬁdence are left unchanged.
Therefore, in our model, the sample can be drawn either by the mixture of the AR model or the proposed approximation method, and ﬁnally the selection of the generated samples are guided by the predicted conﬁdence.
We evaluate NARA with various baseline AR models and data domains including simple curves, image sequences (Yoo et al., 2017), CelebA (Liu et al., 2015a), and ImageNet (Deng et al., 2009).
For the sequential data (simple curves and golf), we employed the Long Short-Term Memory models (LSTM)
(Hochreiter & Schmidhuber, 1997) as a baseline AR model while PixelCNN++ (Salimans et al., 2017) is used for the image generation (CelebA and ImageNet).
Our experiments show that
NARA can largely reduce the sample inference complexity even with a heavy and complex model on a difﬁcult data domain such as image pixels.
The main contributions of our work can be summarized as follows: (1) we introduce a new and generic approximation method that can accelerate any AR generation procedure.
Compared to a full AR generation, the quality of approximated samples remains reliable by the accompanied conﬁdence prediction model that measures the sample credibility.
(3) Finally, we show that this is possible because, under a mild condition, the approximated samples from our method can eventually converge toward the true future sample.
Thus, our method can effectively reduce the generation complexity of the AR model by partially substituting it with the simple i.i.d. model.
P RELIMINARY : A UTO - REGRESSIVE M ODELS
Auto-regressive generation model is a probabilistic model to assign a probability p ( x ) of the data x including n samples.
This method considers the data x as a sequence { x i | i
= 1 , · · · n } , and the probability p ( x ) is deﬁned by an AR manner as follows:
[Equation]
From the formulation, the AR model provides a tractable data distribution p ( x ) .
Recently, in training the model parameters using the training samples ˆ x T , the computation parallelization are actively employed for calculating the distance between the real sample ˆ x t ∈ ˆ x T and the generated sample x t from equation (1).
Still, for generating the future samples, it requires O ( N ) by deﬁnition.
P ROPOSED M ETHOD 3. O VERVIEW
Figure 9 shows the concept of the proposed approximator NARA .
NARA consists of a prior-sample predictor f W and conﬁdence predictor g V .
Given samples x ≤ i = { x 1 , · · · , x i } , the prior-sample predictor predicts a chunk of M number of the prior values m ( i,i + M ] .
Afterward, using the prior samples, we draw the future samples x
( i,i + M ] in parallel.
We note that this is possible because the prior m ( i,i + M ] is i.i.d.
variable from our assumption.
Subsequently, for the predicted x ( i,i + M ] , the conﬁdence predictor predicts conﬁdence scores ν ( i,i + M ] .
Then, using the predicted conﬁdence, our model decides whether the samples of interest should be redrawn by the AR model (re-sample x i +1 ) or they are just accepted .
The detailed explanation will be described in the following sections.
3. A PPROXIMATING S AMPLE D ISTRIBUTION OF AR M ODEL
Given the samples x ≤ i = { x 1 , · · · , x i } , a AR model deﬁnes the distribution of future samples x ( i,j ] = { x i [Equation] 1 , · · · , x j } as follows:
[Equation]
Here, θ denotes the parameters of the AR model.
The indices i, j are assumed to satisfy the condition j > i, ∀ i, j ∈ [1 , N ] .
To approximate the distribution p θ ( x ( i,j ] | x ≤ i ) , we introduce a set of prior samples m ( i,j − 1] =
f W ( x ≤ i ; W ) , where we assume that they are i.i.d.
given the observation x ≤ i .
Here, W is the model parameter of the prior-sample predictor f W ( · ) .
Based on this, we deﬁne an approximated distribution q θ,W ( x ( i,j − 1] | x ≤ i , m ( i,j − 1] ) characterized by the original AR model p θ and the prior-sample predictor f W ( · ) as follows:
[Equation]
Here, approximation (A) is true when m ( i,j − 1] approaches to x ( i,j − 1] .
Note that it becomes possible to compute q θ,W in a constant time because we assume the prior variable m i to be i.i.d.
while p θ requires a linear time complexity.
Then, we optimize the network parameters θ and W by minimizing the negative log-likelihood (NLL) of q θ,W ( x ( i,j ]
= ˆx
( i,j ] | x ≤ i , f W ( x ≤ i )) where ˆx ( i,j ] is a set of samples that are sampled from the baseline AR model.
We guide the prior-sample predictor f W ( · ) to generate the prior samples that is likely to come from the distribution of original AR model m by minimizing the original AR model θ and prior-sample predictor W jointly as follows:
[Equation]
where x ( g ) denotes the ground truth sample value in the generated region of the training samples.
Note that both p θ ( x ) and its approximated distribution q θ,W ( x ) approaches to the true data distribution when (1) our prior-sample predictor generates the prior samples m close to the true samples x and (2) the NLL of the AR distribution approaches to the data distribution.
Based on our analysis and experiments, we later show that our model can satisfy these conditions theoretically and empirically in the following sections.
3. C ONFIDENCE P REDICTION
Using the prior-sample predictor f W ( · ) , our model generates future samples based on the previous samples.
However, accumulation of approximation errors in the AR generation may lead to an unsuccessful sample generation.
To mitigate the problem, we introduce an auxiliary module that determines whether to accept or reject the approximated samples generated as described in the previous subsection, referred to as conﬁdence predictor .
First, we deﬁne the conﬁdence of the generated samples as follows:
[Equation]
where ˆ x k ∼ p θ ( x k | x ≤ k − 1 ) and k ∈ { 1 , · · · , j } .
The conﬁdence value ν k provides a measure of how likely the generated samples from q θ,W ( · ) is drawn from p θ ( x k | x ≤ k − 1 ) .
Based on the conﬁdence value ν k , our model decides whether it can accept the sample x k or not.
More speciﬁcally, we choose a threshold (cid:15) ∈ [0 , 1] and accept samples which have the conﬁdence score larger than the threshold (cid:15) .
When the case (cid:15) = 1 , our model always redraws the sample using the AR model no matter how our conﬁdence is high.
Note that our model becomes equivalent to the target AR model when (cid:15)
= 1 .
When (cid:15) = 0 , our model always accepts the approximated samples.
In practice, we accept ˆ k ( (cid:15) )
− 1 samples among approximated M samples where ˆ k ( (cid:15) ) =
arg min k ν k > (cid:15) .
Subsequently, we re-sample x ˆ k
( (cid:15) ))
from the original AR model and repeat approximation scheme until reach the maximum length.
However, it is impractical to calculate equation (5) directly because we need the samples ˆ x k from the original AR model.
We ﬁrst need to go forward using the AR model to see the next sample and come backward to calculate the conﬁdence to decide whether we use the sample or not, which is nonsense.
To detour this problem, we introduce a network g V ( · ) that approximates the binary decision variable h (cid:15)k = I ( ν k ≥ (cid:15) ) as follows: [Equation]
[Equation]
where h (cid:15) ( i,j ] = { h (cid:15)i +1 , · · · , h (cid:15)j } .
The network g V ( · ) is implemented by a auto-encoder architecture with a sigmoid activation output that makes the equation (6) equivalent to the logistic regression.
3. T RAINING DETAILS
To train the proposed model, we randomly select the sample s ( i ) ∈ [1 , N ] for the sequence x ( i ) in a training batch.
Then, we predict l ( i ) =
min( B, N − s ( i ) ) sample values after s ( i ) , where B denotes the number of samples the prediction considers.
To calculate equation (4), we minimize the loss of the training sample x ( k ) , k = 1 · · · K, and the locations s ( i ) ∈ [1 , N ] as,
[Equation]
Here, ˆx ( i )( j ) ( s ( i ) ,s ( i ) + l ( i ) ] for j ∈ { 1 · · · M } denotes M number of the sequences from the AR distribution p θ ( x ( s ( i ) ,s ( i ) + l ( i ) ]
| x ( i ) ≤ s ( i ) ) for i -th training data.
From the experiment, we found that M = 1 sample is enough to train the model.
This training scheme guides the distribution drawn by NARA to ﬁt the original AR distribution as well as to generate future samples, simultaneously.
To train g V ( · ) , binary cross-entropy loss is used with h (cid:15) in equation (6), with freezing the other parameters.
3. T HEORETICAL E XPLANATION
Here, we show that the proposed NARA is a regularized version of the original AR model.
At the extremum, the approximated sample distribution from NARA is equivalent to that of the original AR model.
In NARA, our approximate distribution q ( x ( i,j +1] | x ≤ i , m ( i,j ] ) is reformulated as follows:
[Equation]
where the parameter φ denotes the network parameters of the approximated distribution q ( · ) .
Therefore [Equation] our proposed cost function can be represented as the negative log-likelihood of the AR model with a regularizer − log R ( p θ , q φ , m ( i,j ] ) :
[Equation]
[Figure]
Note that the proposed cost function is equivalent to that of the original AR model when log R ( p θ , q φ , m ( i,j ] ) = 0 , which is true under the condition of m ( i,j ] = x
( i,j ] and q φ ( ·|· ) =
p θ ( ·|· ) .
Here, m ( i,j ] = f W ( x ≤ i ) .
By minimizing the equation (9), R ( p θ , q φ , m ( i,j ] ) enforces the direction of the optimization to estimate the probability ratio of q φ and p θ while it minimize the gap between q
[Equation]
so that m ( i,j ] = f W ( x ≤ i ) approaches to x ( i,j ] .
R ELATED W ORK
Deep AR and regression models: After employing the deep neural network, the AR models handling sequential data has achieved signiﬁcant improvements in handling the various sequential data including text (Sutskever et al., 2011), sound (Vinyals et al., 2012; Tamamori et al., 2017), and images (van den Oord et al., 2016b; Salimans et al., 2017).
The idea has been employed to “Flow based model” which uses auto-regressive sample ﬂows (Kingma & Dhariwal, 2018; Germain et al., 2015; Papamakarios et al., 2017; Kingma et al., 2016) to infer complex distribution, and reported meaningful progresses.
Also, the attempts (Yoo et al., 2017; Garnelo et al., 2018; Kim et al., 2019) to replace the kernel function of the stochastic regression and prediction processes to neural network has been proposed to deal with semi-supervised data not imposing an explicit sequential relationship.
Approximated AR methods: Reducing the complexity of the deep AR model has been explored by a number of studies, either targeting multiple domain (Seo et al., 2018; Stern et al., 2018) or speciﬁc target such as machine translation (Wang et al., 2018;
Ghazvininejad et al., 2019; Welleck et al., 2019; Wang et al., 2018; 2019) and image generation (Ramachandran et al., 2017).
Adding one step further to the previous studies, we propose a new general approximation method for
AR methods by assuming the i.i.d.
condition for the “easy to predict” samples.
This differentiates our approach to (Seo et al., 2018) in that we do not sequentially approximate the future samples by using a smaller AR model but use a chunk-wise predictor to approximate the samples at once.
In addition, our conﬁdence prediction module can be seen as a stochastic version of the veriﬁcation step in (Stern et al., 2018), which helps our model to converge toward the original solution.
This conﬁdence guided approximation can be easily augmented to the other domain speciﬁc AR approximation methods because our method is not limited to a domain speciﬁc selection queues such as quotation (Welleck et al., 2019;
Ghazvininejad et al., 2019) or nearby convolutional features (Ramachandran et al., 2017).
E XPERIMENTS
In this section, we demonstrate the data generation results from the proposed NARA.
To check the feasibility, we ﬁrst test our method into time-series data generation problem , and second, into image generation .
The detailed model structures and additional results are attached in the
Supplementary material.
The implementation of the methods will be available soon.
[Table] [Table]
E XPERIMENTAL S ETTING
Time-series data generation problem: In this problem, we used LSTM as the base model.
First, we tested our method with a simple one-dimensional sinusoidal function.
Second, we tested the video sequence data (golf swing) for demonstrating the more complicated case.
In this case, we repeated the swing sequences 20 times to make the periodic image sequences and resize each image to 64 ×resolution.
Also, beside the LSTM, we used autoencoder structures to embed the images into latent space.
The projected points for the image sequences are linked by LSTM, similar to (Yoo et al., 2017).
For both cases, we used ADAM optimizer (Kingma & Ba, 2015) with a default setting and a learning rate 0 . 001 .
Image generation: For the image generation task, we used PixelCNN++ (Salimans et al., 2017) as the base model.
The number of channels of the network was set to 160 and the number of logistic mixtures was set to 10 .
See (Salimans et al., 2017) for the detailed explanation of the parameters.
In this task, the baseline AR model (PixelCNN++) is much heavier than those used in the previous tasks.
Here, we show that the proposed approximated model can signiﬁcantly reduce the computational burden of the original AR model.
The prior-sample predictor f W ( · ) and the conﬁdence estimator g V ( · ) were both implemented by U-net structured autoencoder (Ronneberger et al., 2015).
We optimized the models using ADAM with learning rate 0 . 0001 .
Every module was trained from scratch.
We mainly used CelebA (Liu et al., 2015b) resizing the samples to 64 × 64 resolution.
In the experiments, we randomly pick 36 , 000 images for training and 3 , 000 images for validation.
Training and evaluation: For the ﬁrst problem, we use single GPU (NVIDIA Titan XP), and for the second problem, four to eight GPUs (NVIDIA Tesla P40) were used 1 .
The training and inference code used in this section are implemented by PyTorch library.
For the quantitative evaluation, we measure the error between the true future samples and the generated one, and also employ Fr´echet
Inception Distance score (FID) (Heusel et al., 2017) as a measure of the model performance and visual quality of the generated images for the second image generation problem.
5. A NALYSIS 5.2. T IME - SERIES D ATA G ENERATION
Figure 2a shows the generation results of the one-dimensional time-series from our approximation model with different acceptance ratios (red, green, and blue) and the baseline LSTM models (black).
The graph in Figure 2b shows the acceptance ratio and the (cid:96) 1 -error over the conﬁdence threshold (cid:15) ∈ (0 , 1] .
The error denotes the distance between the ground truth samples x and the generated ones.
As expected, our model accepted more samples as the threshold (cid:15) decreases.
However, contrary to our initial expectations, the error-threshold graph shows that the less acceptance of samples does not always bring the more accurate generation results.
From the graph, the generation with an intermediate acceptance ratio achieved the best result.
Interestingly, we report that this tendency between the acceptance ratio and the generation quality was repeatedly observed in the other datasets as well.
Figure 3 shows the image sequence generation results from NARA.
From the result, we can see that the proposed approximation method is still effective when the input data dimension becomes much larger and the AR model becomes more complicated.
In the golf swing dataset, the proposed approximation model also succeeded to capture the periodic change of the image sequence.
The table 3 shows that the proper amount of approximation can obtain better accuracy than the none, similar to the other previous experiments.
One notable observation regarding the phenomenon is that the period of image sequence was slightly changed among different ratio of approximated sample acceptance (Figure 3).
One possible explanation would be that the approximation module suppress the rapid change of the samples, and this affects the interval of a single cycle.
5.2. I MAGE G ENERATION
Figure 4 shows that our method can be integrated into PixelCNN++ and generates images with the signiﬁcant amount of the predicted sample acceptance (white region).
We observed that the conﬁdence was mostly low (blue) in the eyes, mouth, and boundary regions of the face, and the
PixelCNN is used to generate those regions.
This shows that compared to the other homogeneous regions of the image, the model ﬁnds it relatively hard to describe the details, which matches with our intuition.
The graphs in Figure 5 present the quantitative analysis regarding the inference time and the NLL in generating images.
In Figure 5a, the relation between inference time and the skimming ratio is reported.
The results show that the inference speed is signiﬁcantly improved as more pixels are accepted.
Table 2 further supports this that our approximation method generates a fair quality of images while it speeds up the generation procedure 5 ∼ 10 times faster than the base model.
In the image generation example also, we found that the fair amount of acceptance can improve the perceptual visual quality of the generated images compared to the vanilla
PixelCNN++ (Table 2).
Our method beneﬁts from increasing the acceptance ratio to some extent in terms of FID showing a U-shaped trend over the (cid:15) variation, similar to those in Figure 2b.
Note that a lower FID score identiﬁes a better model.
Consistent with previous results, we can conjecture that the proposed approximation scheme learns the mean-prior of the images and guides the AR model to prevent generating erroneous images.
The conﬁdence maps and the graph illustrated in Figure 4, 5a, and 5c p i x e l s / s ec
Confidenc e threshold ( ε )
Generation speed
Acceptance ratio 59.4x 11.0x
7.4x 5.9x 5.2x 4.1x 3.1x 2.4x 1.7x 1.2x 1.0x (a) Generation speed (pixels/sec) over (cid:15) ∈ [1 . 0 , 0 . 0] .
0 10 20 30 40b it s / d
i m epochs
No skim
Full skim (b) NLL Curve.
0 10 20 30 40ℓ 1 l o ss epochs ⨯ 10Test ℓ 1 / dim (c) (cid:96) 1 error of the prior m i .
(cid:15) 1.
SU 1.0x 1.2x
[Table]
support this conjecture.
Complex details such as eyes, mouths and contours have largely harder than the backgrounds and remaining faces.
In Figure 5b and Figure 5c, the graphs show the results supporting the convergence of the proposed method.
The graph in Figure 5b shows the NLL of the base PixelCNN++ and that of our proposed method under the full-accept case, i.e. we fully believe the approximation results.
Note that the NLL of both cases converged and the PixelCNN achieved noticeably lower NLL compared to the fully accepting the pixels at every epoch.
This is already expected in Section 3.2 that the baseline AR model approaches more closely to the data distribution than our module.
This supports the necessity of re-generating procedure by using the PixelCNN++, especially when the approximation module ﬁnds the pixel has a low conﬁdence.
The graph in Figure 5c presents the (cid:96) 1 distance between the generated prior pixel m and the corresponding ground-truth pixel x in the test data reconstruction.
Again, similar to the previous time-series experiments, the model successfully converged to the original value ( m approaches to x ).
Combined with the result in Figure 5b, this result supports the convergence conditions claimed in section 3.2.
Regarding the convergence, we compared the NLL of the converged PixelCNN++ distribution from the proposed scheme and that of PixelCNN++ with CelebA dataset from the original paper (Salimans et al., 2017).
C ONCLUSION
In this paper, we proposed the efﬁcient neural auto-regressive model approximation method, NARA, which can be used in various auto-regressive (AR) models.
By introducing the prior-sampling and conﬁdence prediction modules, we showed that NARA can theoretically and empirically approximate the future samples under a relaxed causal relationships.
This approximation simpliﬁes the generation process and enables our model to use powerful parallelization techniques for the sample generation procedure.
In the experiments, we showed that NARA can be successfully applied with different
AR models in the various tasks from simple to complex time-series data and image pixel generation.
These results support that the proposed method can introduce a way to use AR models in a more efﬁcient manner.
A I MPLEMENTATION D ETAILS
Time serious sample generation: For the 1-dimensional samples case, the LSTM consists of two hidden layers, and the dimension of each layer was set to 51 .
We observed o = 200 steps and predicted p = 20 future samples.
The chunk-wise predictor f W ( · ) and the conﬁdence predictor g V ( · ) were deﬁned by single fully-connected layer with input size 200 and the output size 20 .
In this task, our model predicted every 20 samples by seeing the 200 previous samples.
For the visual sequence case, we used autoencoder structures to embed the images into latent space.
The encoder consists of four “Conv-activation” plus one ﬁnal Conv ﬁlters.
Here, the term “Conv” denotes the convolutional ﬁlter.
Similarly, the decoder consists of four “Conv transpose-activation” with one ﬁnal Conv transpose ﬁlters.
The channel sizes of the encoder and decoder were set to { 32 , 32 , 32 , 64 } and { 64 , 32 , 32 , 32 } .
All the Conv and Conv Transpose ﬁlters were deﬁned by 3 × 3 .
The activation function was deﬁned as Leaky ReLU function.
The embedding space was deﬁned to have 10-dimensional space, and the model predicts everyfuture samples given 50 previous samples.
The prior sample predictor and the conﬁdence predictor are each deﬁned by a fully-connected layers, and predict future samples in conjunction to the decoder.
We note that the encoder and decoder were pre-trained by following (Kingma & Welling, 2014).
Image generation: The prior sample and conﬁdence predictors consist of an identical backbone network except the last block.
The network consists of four number of “Conv-BN-activation” blocks, and the four number of “Conv transpose-BN-Activation” block.
All the Conv and Conv Transpose ﬁlters were deﬁned by 4 × 4 .
The last activation of the prior sample predictor is tangent hyperbolic function and that of conﬁdence predictor is deﬁned as sigmoid function.
The other activation functions are deﬁned as Leaky-ReLU.
Also, the output channel number of the last block are set to 3 and 1 , respectively.
The channel size of each convolution ﬁlter and convolution-transpose ﬁlter was set to be { 64 , 128 , 256 , 512 } and { 256 , 128 , 64 , 3 } .
We used batch normalization (Ioffe & Szegedy, 2015) for both networks and stride is set to two for all the ﬁlters.
The decision threshold (cid:15) is set to the running mean of the κ log( q θ,W ( · )) .
We set κ = 2 .for
every test and conﬁrmed that it properly works for all the cases.
B S UPPLEMENTARY E XPLAIN ON P ROPOSED S AMPLE G ENERATION
Figure 6 shows the detailed process of sampling when the proposed NARA module is attached to the mother AR model.
The diagram describes the Sinusoidal function generation example, which is the simplest example in our paper.
In each auto-regressive step, we decide if we use the approximated value predicted by our sample predictor (predict the value in chunkwise manner assuming i.i.d condition) or sample the value with the mother auto-regressive model.
This decision is conducted based on the conﬁdence estimation by the Conﬁdence predictor of the paper.
The chunk-wise estimation step can be boosted by recent parallel computing methods different from the mother autoregressive model, and this can possibly relax the computation burden of the original auto-regressive model.
More importantly, this scheme can be applied to various auto-regressive models.
C S UPPLEMENTARY E XPERIMENTS C. S UPPLEMENTARY G ENERATION R ESULTS
In addition to the results presented in the paper, we show supplement generation examples in below ﬁgures.
Figure 3 and Table 3 present the image sequence generation result from the other golf swing sequence.
In this case also, we can observe the similar swing cycle period changes and acceptance ratio-error tendencies reported in the paper.
Our approximation slightly affects the cycle of the time-serious data, and the result also shows that the approximation can achieve even better prediction results than “none-acceptance” case.
Figure 11 and 12 shows the additional facial image generation results among (cid:15) ∈ [0 . 0 , 1 . 0] .
We can see that pixels from boundary region were more frequently re-sampled compared to other relatively simple face parts such as cheek or forehead.
Also, we tested our model with ImageNet classes, and the results were presented in Figure 10. x ≤ i =
[ x 1 , x 2 , …, x i ] m ( i,i+ M ] =
[ m i+1 , …, m i+M ]
Prior-sample Predictor f W (parallel inference)
Prior values Input [ x 1 , …, x i , m i+1 , …, m i+M ] Original AR model (parallel inference) [ x 1 , …, x i , x′ i+1 , …, x′ i+M ] Input + prior values
Input + sampled values
Confidence Predictor g V (parallel inference) [ σ i+1 , …, σ i+M ] k = argmin j σ i+j < ε
Accept: [ x 1 , …, x i , x′ i+1 , …, x′ i+k ]
Re-sample: [ x′ i+k+1 , …, x′ i+M ]
Confidence scores
[Figure]
ε
Approximated samples
C. A NALYSIS ON WHEN C ONFIDENCE M ODULE F AILS .
If the prior-sample predictor f W performs worse than our expectation, the conﬁdence module g V will reject all prior samples; hence, in this case, the model repeats to draw samples using the original AR model in a sample-by-sample manner.
However, in an even worse situation, the conﬁdence module could always accept prior-samples, including low-quality prior samples.
To simulate the failure or both f W and g V , we manually ﬁx the “accept region” regardless of the predicted conﬁdence score.
We select ten samples with the lowest conﬁdence scores and report the results in Figure 8.
In the ﬁgure, we can observe that the drastic error can occur when both the prior-sample predictor f W and conﬁdence module g V fail.
However, in practice, our conﬁdence module g V can detect the drastic error during the sampling stage as shown in Figure 8b; hence, such extreme error will occur rarely.
C. A PPLYING NARA WITH F ASTER P IXEL CNN++
To show that our method can be added to diverse AR mode, we also combined our skimming method to the fast version of PixelCNN++ 2 .
Figure 9 and Table 4 show the generated samples and the generation time using the incorporated model ( skim +Fast PixelCNN++).
Due to the lack of time to fully investigate all datasets with the new implementation, we conducted the experiment on CIFAR-10, which is the most widely used dataset in PixelCNN works and is more diverse than the CelebA dataset.
From the results, we show that our method, augmented with Fast PixelCNN++, can make the baseline algorithm much faster as we suggested in the paper
[Table]
[Table]
[Figure]
[Table]
[Figure]