Faster Discovery of Neural Architectures by Searching for Paths in a Large Model
Anonymous authors
Paper under double-blind review
Abstract
We propose an approach for automatic model designing, which is signiﬁcantly faster and less expensive than previous methods.
In our method, which we name Eﬃcient Neural Architecture Search (ENAS), a controller learns to discover neural architectures by searching for an optimal path within a larger predetermined model.
The parameters of the predetermined model are trained to minimize a canonical loss function, such as the cross entropy, on the training dataset.
The controller learns the path with policy gradient to maximize the expected reward on the validation set.
In our experiments, ENAS achieves comparable test accuracy while being 10x faster and requiring 100x less resources than NAS.
On the CIFARdataset, ENAS can design novel architectures that achieve the test error of 3 .
86%, compared to 3 .
41% by standard NAS (Zoph et al., 2017).
On the
Penn Treebank dataset
, ENAS also discovers a novel architecture, which achieves the test perplexity of 64 . 6 compared to 62 .
4 by standard NAS.
Introduction
Neural architecture search (NAS) has recently been used successfully to design model architectures for a number of tasks, especially in the areas of image classiﬁcation and language modeling (Zoph & Le, 2017;
Zoph et al., 2017).
Despite its success, NAS is computationally expensive and time consuming.
For instance, Zoph et al.
(2017) use 450 GPUs and train for 3-4 days.
On the other hand, using less computing resources often leads to less compelling results (Negrinho & Gordon, 2017; Baker et al., 2017).
In this work, we propose a method to accelerate NAS.
First, we observe that it is possible to share parameters among all child models in a particular search space.
This observation eﬀectively avoids having to train from scratch each new architecture found during the search process, leading to a faster search time.
Though it is appealing, sharing parameters among diﬀerent architectures can lead to very noisy updates to the parameters, as an update to one architecture may carry negative eﬀects to other architectures.
To address this problem, we determine a large model ahead of time, with a careful weight sharing scheme that allows our method to generate a discrete mask within this model, which we describe in Section 4.
We then interleave two processes: (1) training the shared parameters to minimize a canonical loss function, such as the cross entropy, on the training data; and (2) searching for good architectures, which are represented by discrete sequences of tokens, using policy gradient on the validation set.
This method can quickly discover novel architectures whilst using far less computing resources.
Because of this eﬃciency, we name our method Eﬃcient Neural
Architecture Search (ENAS).
We give a theoretical justiﬁcation of the convergence guarantee of ENAS, and then demonstrate empirically its eﬀectiveness on two tasks: image classiﬁcation on CIFAR-10 and language modeling on Penn Treebank.
For both tasks, ENAS is able to discover novel neural architectures that achieve comparable accuracies to best reported results by NAS methods.
Speciﬁcally, ENAS achieves 3 .
86% error rate on CIFAR-10 and 64 .
6 perplexity on Penn
Treebank.
In comparison, the best results reported by previous NAS approaches are 3 .
41% error rate on CIFAR-10 and 62.4 perplexity on Penn Treebank.
More importantly, ENAS is signiﬁcantly faster and less computationally intensive than previous methods.
In our ex- periments, ENAS takes less than 15 hours to train, running on a single Nvidia GTX 1080Ti
GPU, meaning that ENAS uses approximately 100x less computational resources and is approximately 10x faster than NAS to reach 3 .
86% test error on CIFAR-10.
Neural Architecture Search with Reinforcement Learning
The goal of NAS is to ﬁnd a model architecture m that achieves a high reward R ( m ) on a particular task.
This setting is general: R ( m ) can be the accuracy for an image classiﬁer, or the perplexity of a language model.
Recent approaches such as Zoph & Le (2017); Zoph et al.
; Baker et al.
; Bello et al.
; Zhong et al.
(2017) used reinforcement learning to search the combinatorial space of architectures, attempting to model a parameterized policy π ( m ; θ ) of m to maximize the expected reward
[Equation]
The key diﬀerences between these NAS approaches are in the design choices for the search space, the parametrization of π ( m , θ ), and the learning algorithm for optimizing J ( θ ).
There are two main approaches for designing the search space.
The ﬁrst approach designs the whole architecture all at once.
For instance, if we were to design a convolutional network, we would choose how many ﬁlters to use at each layer, the ﬁlter sizes, how to set up the skip connections, etc.
We call this approach macro search space design .
Macro design is used by Zoph & Le (2017);
Baker et al.
(2017); Negrinho & Gordon (2017) ; Brock et al.
(2017); Saxena & Verbeek (2016).
The second approach, which we call micro search space design , aims to search for smaller modules which are then replicated and assembled to form the ﬁnal model.
Micro design is used by Zoph et al.
; Zhong et al.
We note that Zoph & Le (2017) also use micro design when designing the RNN cell.
An architecture m is often represented as a sequence, with a grammar and meaning speciﬁed by the NAS algorithm.
Because modeling variable length sequences is necessary, the reinforcement learning policy π ( m ; θ ) is often parametrized with a recurrent neural network.
J ( θ ) is then learned with various reinforcement learning algorithms.
For instance, Zoph &
Le (2017) use REINFORCE (Williams, 1992) with a moving average baseline and Zoph et al.
(2017) use Proximal Policy Optimization (Schulman et al., 2017) to learn J ( θ ).
Deep
Q-learning approaches have also been explored (Baker et al., 2017; Zhong et al., 2017).
The Source of the Computational Expense
NAS methods are typically time consuming and computationally expensive.
Most significantly, Zoph & Le (2017) and Zoph et al.
(2017) respectively train in 3-4 weeks and 3days, using 800 and 450 GPUs concurrently at any time during their training.
Zhong et al.
(2017) use 32 GPUs, training in 3 days.
This amount of computing resource, albeit reduced compared to the ﬁrst two studies, is still not widely available.
Baker et al.
(2017) use
GPUs, training in 5 days, but the architectures found on the task of CIFAR-10 do not yield as compelling results, likely because the method did not have an opportunity to explore enough architectures.
The computational expense of NAS methods is due to the fact that they train each child model m from scratch.
To see this, note that in Eqn. 1, the reward R ( m ) cannot be computed without the optimal parameters
ω ∗ m of the model m .
To ﬁnd ω ∗ m , we need to train m to convergence.
Due to the high sample complexity of reinforcement learning, many models m have to be sampled, trained, and evaluated.
We posit that this is the main weakness of NAS approaches: after training each model m , only the reward R ( m ; ω ∗ m ) is kept, while the learned parameters in ω ∗ m , which could be reused, are thrown away.
Several methods have been proposed to avoid training each architecture m from scratch, such as convolutional neural fabrics (ConvFabrics) (Saxena & Verbeek, 2016) and SMASH (Brock et al., 2017).
These methods are often more computationally eﬃcient.
However, the tradeoﬀ in ConvFabrics is that their search space is not ﬂexible enough to include novel architectures, e.g., architectures with arbitrary skip connection patterns as in Zoph & Le (2017).
In contrast, SMASH can design many interesting architectures, but requires a hypernetwork (Ha et al., 2017) to generate the weights ω m , conditional on the architecture m .
While a hypernetwork can eﬃciently rank diﬀerent architectures, as shown in the paper, the real performance of each network, i.e., R ( m , ω ∗ m ), is diﬀerent from its performance with ω m generated by a hypernetwork.
Such discrepancy can cause misleading signals, which may make SMASH unsuitable for reinforcement learning.
In this work, we propose a novel approach that combines the strengths of both ConvFabrics and SMASH.
Unlike SMASH, we do not use a hypernetwork to generate ω m for each m , but instead keep a shared set of parameters ω among all architectures m .
Unlike ConvFabrics, we design a diﬀerent mechanism to specify computing paths in our model, leading to a very ﬂexible search space.
In fact, our search space is larger than the search space of SMASH as well as the search space described in Zoph & Le (2017).
As a result, our method can eﬃciently discover novel architectures that achieve better accuracy than SMASH.
Efficient Neural Architecture Search 4. Parameters Sharing in Convolutional Models
We propose a scheme to share parameters among diﬀerent convolutional architectures, which is illustrated in Figure 1.
Our method requires a pre-speciﬁed number of layers, denoted by L , in all models in our search space.
1 Suppose that at each layer, we have to make a decision of which of B operations to perform.
In this paper, we use B = 6 operations at each layer: four convolutions with square ﬁlters of sizes 1, 3, 5, 7, an average pooling, and a max pooling of ﬁlter size 3 × 3.
All these operations have the spatial strides 1 × 1 and their outputs are centrally padded to preserve the spatial dimensions.
Each of these operations has a maximum number of output channels C , set to 256.
Parameters are independent to each branch in each layer.
We denote by ω the set of all shared parameters in the model.
During each data pass, each of the L × B × C channels in the network is controlled by a binary mask, which, if turned oﬀ, will remove the corresponding channel from the network and if turned on, will allow information to ﬂow through that channel.
These binary masks are speciﬁed by a controller network, which we will describe in Section 4.3.
Since the number of channels can be large, we group them into blocks of S channels, where S evenly divides
C .
Consequentially, for each channel, we only predict C/S binary masks.
In order to allow architectures with arbitrary patterns of skip connections (He et al., 2016;
Zoph & Le, 2017), we insert skip connections between every pair of layers in the network, which the controller can also turn on or turn oﬀ.
Therefore, each conﬁguration of the channels and the skip connections realizes an architecture in our search space.
The resulting search space has 2 LBC/S + L
( L − 1) / 2 conﬁgurations.
Structure of Convolutional Layers.
Each convolutional operation in our method is followed by a batch normalization (Ioﬀe & Szegedy, 2015) and then a ReLU layer.
We ﬁnd the alternate setting of batch norm-conv-ReLU (Zoph et al., 2017) to have worse results.
Stabilizing Stochastic Skip Connections.
If a layer receives skip connections from multiple layers before it, then these layers’ outputs are concatenated in their depth dimension, and then a convolution of ﬁlter size 1 × 1 (followed by a batch normalization layer and a ReLU layer) is performed to ensure the number of output channels is still equal to C .
This is necessary because if we were to follow Zoph & Le (2017) in concatenating the skip connected layers, that would result in ‘ ( ‘ − 1)channels at layer ‘ th , leading to a prohibitively large number of parameters.
Using 1 × 1 convolutions to keep the number of channels at C is more economical.
Global Average Pooling.
After the ﬁnal convolutional layer, we average all the activations of each channel and then pass them to the Softmax layer.
This trick was introduced by Lin et al.
(2013), with the purpose of reducing the number of parameters in the dense connection to the Softmax layer to avoid overﬁtting.
This trick is important to ENAS, because in ENAS, each conﬁguration of the lower layers can lead to radically diﬀerent outputs in the ﬁnal layer, especially in the early training steps.
If the Softmax head overﬁts to any of these conﬁgurations, subsequent updates made to model parameters ω become unstable, e.
g. parameters in the Softmax layer have much larger gradients than parameters in other layers
As we shall see in Section 4.4, our training algorithm can lead to noisy updates on ω .
These three aforementioned tricks are thus crucial to our scheme of sharing parameters among diﬀerent convolutional models.
We carry out ablation studies which show that removing any of the tricks signiﬁcantly de-stablizes our ENAS training process and substantially reduces the accuracy of the models found.
4. Parameters Sharing in Recurrent Models
For RNNs, NAS is used to design the architecture for each cell; this cell is then replicated at each time step and at each layer (Zoph & Le, 2017).
Rather than searching for a completely novel cell architecture like Zoph & Le (2017), we start with the recurrent highway network (RHN), a recurrent cell architecture that not only allows gradients to propagate asympotically without loss but also allows the cell to model more complex transformations (Zilly et al., 2017).
Each RHN cell resembles a dense layer in a feed forward network, augmented with a linear carousel.
In He et al.
, the authors found that skip connections can improve the gradient ﬂows in deep networks, leading to better performance.
We hypothesize whether RHN can also beneﬁt from having skip connections among their highway layers.
We thus use ENAS to learn the skip connections pattern in the highway layers.
Speciﬁcally, if layer ‘ th receives the skip connections from layers i 1 < i 2 < ... <
i k , then the output s ‘ is computed as follows
[Equation]
where ⊗ denotes the elementwise product and W ( h ) ‘ and W ( t ) ‘,j are weight matrices.
When k
=
and i 1 = ‘ − 1 for all layers ‘ , we have the original RHN (Zilly et al., 2017).
It is worth noting that the update rule in Eqn. 3 can be rewritten as
[Equation]
If E data ∼D [ δ ‘ ] = 0, this update rule is consistent with the feature identity and the unrolled iterative estimation interpretations of highway networks (Greﬀ et al., 2017).
Eqn. 2 has a pitfall: if a layer s ‘ does not receive any connection from all previous layers, the model fails to compile.
This phenomenon was noted in Zoph & Le (2017) as compilation failure.
We resolve the issue by forcing each layer s ‘ to be connected to s ‘ − 1 , eﬀectively absorbing the highway connection into the skip connections.
4. The Policy Network
Following Zoph & Le (2017), we parameterize the policy π ( m ; θ ) with an RNN.
In all experiments, we use a two-layer stacked Long Short-Term Memory (LSTM) (Hochreiter & Schmidhuber, 1997; Sutskever et al., 2014) with 64 hidden units per layer to model a sequence of tokens.
The meaning of the sequence of tokens depends on the class of architectures we want to design, as we discuss below.
General Form of the Controller.
Figure 2 illustrates the RNN that models our general controller.
In order to design a convolutional architecture with L layers, each with B branches, we let the controller RNN run for L blocks of B + 1 steps.
In each block, the ﬁrst B steps sample B masks for the corresponding B channels.
Each mask is an integer in (0 , 2 C/S − 1], where C and S are as in Section 4.1.
The last step in each block is the anchor step, whose output is used in a content-based attention function (Bahdanau et al., 2015) to sample the binary decisions whether to establish a skip connections between layers.
Speciﬁcally, for layers i and j if i < j then
[Equation]
) , (5) where W 1 , W 2 , and v are trainable parameters.
If at layer j the controller samples k skip connections to the previous layers i 1 , i 2 , ..., i k then the embedding to the next step in the next block of the RNN is the average of the corresponding k anchor steps.
Practical Restricted Controller.
Since the described controller results in a very large search space, we explore two of its restrictions, corresponding to two sub-search spaces.
In the ﬁrst restriction, we remove the anchor step in each block, equivalently removing the controller’s ability to design skip connections.
Instead, we enforce the network to use the skip connections pattern of DenseNet (Huang et al., 2016).
In the second restriction, we keep only the anchor step in each blocks, remove all the mask predictions, and force the network to use convolutions of ﬁlter size 3 × 3 and C channels, equivalently focusing the controller to ﬁnd a good skip connection pattern.
Our best convolutional architecture is obtained by this variation, described in more details in
Section 5.
This is consistent with the observation in (Zoph & Le, 2017) that (1) if the search space is too large, the controller fails to ﬁnd the good conﬁgurations, and (2) ﬁxing a particular dimension in the search space and then using some manual designs lead to the best models.
For recurrent models, as we only want to design the skip connection patterns in RHN cells (cf.
Section 4.2), we simply employ this second restriction.
4. Optimization Algorithm and Convergence Guarantee
With the described speciﬁcations on the models and parameters, the training algorithm of
ENAS consists of two alternating phases, which we describe below.
The ﬁrst phase trains ω on a whole pass through the training data set, then the second phase trains θ for a ﬁxed number of steps, which we set to 2000.
Training the Shared Parameters ω .
In this step, we ﬁx the policy π ( m ; θ ) and perform stochastic gradient descent (SGD) updates on ω to minimize the expected loss function
L ( m ; ω ).
The gradient is computed via the unbiased Monte Carlo estimate
[Equation]
We ﬁnd that M = 1 works just ﬁne, i.e. we can update ω using the gradient from any single model m sampled from π ( m ; θ ).
We train ω for a whole pass through the training data set.
Training the Policy π ( m ; θ ) .
In this step, we ﬁx ω and update the policy parameters θ , aiming to maximize the expected reward E m ∼ π ( m ; θ )
[ R ( m , ω )].
We employ the Adam optimizer (Kingma & Ba, 2015), for which the gradient is computed via the REINFORCE equation (Williams, 1992), with variance reduction using a moving average baseline ˆ b :
[Equation]
[ R ( m ,
[Equation]
The reward R ( m , ω ) is computed on a minibatch of examples sampled from the validation set, rather than the training set, as we want to encourage ENAS to select the model that generalizes well, rather than the model that overﬁts the training set well.
We try two alternative implementations of this step, where we evaluate R ( m , ω ) only on the training set, and on minibatches either from the training set or the validation set.
The resulting training algorithms converge faster, but the found architectures generalize poorly.
The reward function for image classiﬁcation is the rate of correct classiﬁcation on a minibatch of images sampled from the validation set.
The reward function for language model is c/ valid ppl, where the perplexity is computed on a minibatch, also sampled from the validation set.
This reward function is diﬀerent from the reward function proposed by Zoph & Le (2017), which is c/ valid ppl 2 .
In Appendix A, we derive these choices with theoretical justiﬁcations that they stabilize the interleaving updates in Eqn. 6 and 7.
4. Inference
We propose two approaches to inference.
In the ﬁrst approach, we sample several models from π ( m , θ ).
We compute each model’s reward on a single minibatch sampled from the validation set.
We then take the model with the highest reward to re-train from scratch.
In the second approach, we also sample models from π ( m ; θ ), but unlike the ﬁrst approach, we keep ω and evaluate R ( ˆ m , ω ) as the ensemble reward of these models.
Speciﬁcally, with the image classiﬁcation task, we average the pre-softmax logits of the sampled models, and with the language model task, we average the post-softmax logits of the sampled models.
In our experiments, the ﬁrst approach yields better results.
To match the performance of an architecture retrained using the ﬁrst approach, e.g. in terms of accuracy on CIFAR-10, the second approach needs as many as 171 samples, making its inference more expensive than the inference of a single retrained model.
Our interpretation is that ENAS does achieve the same eﬀect of NAS methods, i.e. discovering good architectures, but does not learn a good set of parameters to be shared by all of them.
Experiments 5. Image Classification on CIFAR-
Dataset.
The CIFAR-10 dataset (Krizhevsky, 2009) consists of 50,000 training images and 10,000 test images.
We use the standard data pre-processing and augmentation techniques, i.e. subtracting the mean and dividing the standard deviation from each channel computed on the training images, centrally padding the training images to 40 × 40 and randomly cropping them back to 32 × 32, and randomly ﬂipping them horizontally.
Training details.
The shared parameters ω is trained with Nesterov momentum (Nesterov, 1983), where the learning rate follows the cosine schedule with l max = 0 .
05, l min = 0 .
001, T 0 = 10 and T mul = {+
+}
(Loshchilov & Hutter, 2017).
Each architecture search is run for 10 + 20 + 40 + 80 + 160 = 310 epochs.
Each weight component in ω is initialized from a scaled Gaussian as described in He et al.
We also apply an ‘ 2 weight decay of 10 − 4 .
The same settings are employed to train the architecture recommended by the controller.
The policy parameters θ are initialized uniformly in [ − 0 . 1 , 0 . 1], and trained with the Adam optimizer at a learning rate of 10 − 3 .
We additionally utilize three techniques to prevent the premature convergence of REINFORCE.
First, a temperature τ = 5 . 0 and a tanh constant c = 2 . 5 are applied to the controller’s logits, i.e. every sample s from the controller’s logits u comes from the distribution s ∼ softmax( c · tanh ( u /τ )).
Second, we add to the reward the entropy term of the controller’s samples weighted by λ ent = 0 .
1, which discourages convergence (Williams & Peng, 1991).
Lastly, we enforce the sparsity in the skip connections by adding to the reward the Kullback-Leibler divergence between the skip connection probability in Eqn.
5 and a chosen probability ρ = 0 . 4, which represents the prior belief of a skip connection being formed.
The KL divergence term is weighted by λ kl = 0 .
5 in our reward.
Results.
Table 1 summarizes the test errors of ENAS and other approaches.
The time taken to discover our models are reported in Table 2.
More details are as follows.
In the ﬁrst experiment, we search for the masks at each branch and each layer in a 12-layer network.
Layers 4 th and 8 th are max pooling layers with a kernel size of 2 × 2 and a stride of 2, which reduce each spatial dimension of the layers’ outputs by a factor of 2.
Within each group of 3 layers where the spatial dimensions of the layers remain constant, we connect each layer to all layers before it (Huang et al., 2016).
We use the block size of S = 32, resulting in C/S = 256 / 32 = 8 blocks per branch per layer.
The resulting model, depicted in Figure 3, almost always has 64 or 96 channels at each branch and each layer, indicating that the controller does not choose to activate all blocks.
This is the desired behavior, as doing so would over-parametrize the model and result in overﬁtting.
The search for this architecture takes only 11 .
6 hours
and yet the resulting model achieves the test error of 4 .
35%, which is better than all but one model reported by Zoph & Le (2017).
In the second experiment, we search for a good pattern of skip connections.
We use 3 ×convolutions with 48 output channels at all layers.
Our controller is allowed to form skip connections between arbitrary layers, but forming such connections between layers with diﬀerent spatial dimensions would result in compilation failures.
To circumvent, after each max pooling in the network, we centrally pad the output so that its spatial dimensions remain unchanged.
In 12 . 4 hours, this process discovers the pattern of skip connections depicted in Figure 4.
This pattern has the property that skip connections are formed much more densely at higher layers than at lower layers, where most connections are only between took the authors’ published code, ran it for a few epochs and interpolated the running time.
[Table] [Table]
consecutive layers.
The architecture found in this experiment has the test error of 5 . 04%, which is slightly worse than the one found in the ﬁrst experiment.
We manually augment the pattern of skip connections using two approaches.
First, similar to Zoph & Le (2017), we replace the 3 × 3 convolutions at each layer with 3 consecutive layers that are densely connected, each has 256 channels.
The resulting model achieves a better test error of 4 .
Second, following Xie et al.
(2017) and Ahmed & Torresani (2017), we replace blocks of 3 densely connected layers with 4 parallel branches, where each branch performs a 3 × 3 convolution on the input layer and their outputs are combined using a 1 × 1 convolution.
The resulting architecture achieves the test error of 3 . 86%.
Sanity Check with Ablation Study.
To assert the role of ENAS, we carry out two sanity check experiments.
In the ﬁrst study, we uniformly at random pick a conﬁguration of channels and skip connections and just train a model.
As a result, about half of the channels and skip connections are selected, resulting in a model with 47 .
1M parameters, which has the error rate of 5 . 86%.
This error rate is worse than the models designed by
ENAS, which have fewer parameters.
In the second study, we only train ω and do not update the controller.
The eﬀect is similar to dropout with a rate of 0 .
5 on both the channels and the skip connections.
At convergence, the model has the error rate of 11 .
On the validation set, the ensemble of 250 Monte Carlo conﬁgurations of the trained model could only reach 8 . 99% test error rate.
We therefore conclude that the appropriate training of the
ENAS controller is crucial for good performance.
5. Language Model with Penn Treebank
Training details.
Our controller is trained with the same settings as described in Section 5.1 for CIFAR-10, except that now we anneal the skip connection probability to ρ = 0 .and
set learning rate for Adam to 0 .
The shared parameters ω are trained using stochastic gradient descent with a learning rate of 0 . 2, decayed by a factor of 0 .
9 after every 3 epochs starting at epoch 15, for a total of 150 epochs.
During the architecture search process, following Melis et al.
(2017), we randomly reset the starting state with probability of 0 . 001.
We also employ weight tying (Inan et al., 2017).
No additional regularization technique is applied.
When retraining the architecture recommended by the controller, however, we use variational dropout and an ‘ 2 regularization with weight decay of 10 − 7 .
Results.
Table 3 presents our results in comparison with other methods.
In 8 . 2 hours, our controller ﬁnds the architecture as depicted in Figure 5 in the Appendix.
This architecture has only 8 million parameters and achieves the test perplexity of 71 . 3, which is better than
LSTM trained using the same setting.
Finally, if we increase the number of hidden units in our model so that the total number of parameters is 21M, then it achieves the test perplexity of 64 . 6, which is very close to the test perplexity found by full-scaled NAS (Zoph & Le, 2017), which uses orders of magnitude more computing resources and time.
[Table]
Vanilla Dropout
[Table]
Zaremba
[Table]
[Table]
Recurrent Highway Network (Zilly et
[Table]
[Table]
[Table]
Melis
[Table]
[Table]
[Table]
[Table]
Dynamic Eval
[Table]
Krause
[Table]
[Table] [Table]
[Table]
[Table] [Table]
[Table] [Table]
[Table]
Conclusion
Neural Architecture Search (NAS) is an important advance that allows faster architecture design for neural networks.
However, the computational expense of NAS prevents it from being widely adopted.
In this paper, we presented ENAS, an alternative method to NAS, that requires less resources and time.
The key insight of our method is to share parameters across child models during architecture search.
This insight is implemented by having NAS search for a path within a larger model.
We demonstrate empirically that the method works well on both CIFAR-10 and Penn Treebank datasets.
Appendices A Derivation of Reward Functions
For model m with shared parameters ω , the cross-entropy loss L ( m , ω ) can be computed by integrating over the data space x ∼ P ( x ) [Equation]
[Equation]
Below, we show that the reinforcement learning signals R ( m , ω ) that we speciﬁed in Section 4.4 are the unbiased Monte Carlo approximations of the posterior of the data, i.e. R dP ( x ) p ( x | m , ω ).
Before we present the derivation, let us put forth a remark.
In Chapter 3.3.2 of Bottou (1991), it was shown that if the estimations of gradients ˆ ∇ ω and ˆ ∇ θ are unbiased, then under reasonable assumptions of the function to optimize, the SGD updates of ω converge almost surely.
In other words, if we ﬁx π ( m , θ ) and just perform SGD to update ω , it will converge to a local minimum.
However, the role of π ( m ; θ ) is to calibrate the expectation E m [ L ( m , ω )].
Speciﬁcally, by Jensen’s inequality
[Equation]
[Equation]
Thus, maximizing E m ∼ π ( m ,θ ) [ R ( m , ω )] pushes down a surrogate function of E m [ L ( m , ω )], similar to the M-step in the family of Expectation-Maximization algorithms.
Now we present the derivation.
On an image classiﬁcation task, the data x is actually a pair ( x , y ) of an image and its label.
If instead of predicting the label by taking argmax of the model’s probabilities, we sample the label ˆ y ∼ p ( y | x , m , ω ), as done in Bayesian classiﬁers, then the expected accuracy is
[Equation]
The accuracy on a minibatch is thus a Monte Carlo approximation of of E m [ p ( x | m , ω )].
On a language model task, the data x is a sequence of words.
Letting | x | be the number of words in x , the expectation of the proposed reward in Section 4.4 is
[Equation]
Since we use R ( m , ω ) as the signal for reinforcement learning, we can omit the constant c R dP ( x ) exp | x | since it will integrate to 0 in the REINFORCE equation, just like the constant baseline (Williams, 1992).
This justiﬁes the use of the signal c valid ppl( x ) for learning.
B Models found for CIFAR-
[Figure]
C Models found for Penn Treebank