A F LEXIBLE A PPROACH TO A UTOMATED RNN A RCHITECTURE G ENERATION
Anonymous authors
Paper under double-blind review
A BSTRACT
The process of designing neural architectures requires expert knowledge and extensive trial and error.
While automated architecture search may simplify these requirements, the recurrent neural network (RNN) architectures generated by existing methods are limited in both ﬂexibility and components.
We propose a domain-speciﬁc language (DSL) for use in automated architecture search which can produce novel RNNs of arbitrary depth and width.
The DSL is ﬂexible enough to deﬁne standard architectures such as the Gated Recurrent Unit and
Long Short Term Memory and allows the introduction of non-standard RNN components such as trigonometric curves and layer normalization.
Using two different candidate generation techniques, random search with a ranking function and reinforcement learning, we explore the novel architectures produced by the
RNN DSL for language modeling and machine translation domains.
The resulting architectures do not follow human intuition yet perform well on their targeted tasks, suggesting the space of usable RNN architectures is far larger than previously assumed.
I NTRODUCTION
Developing novel neural network architectures is at the core of many recent AI advances (Szegedy et al., 2015;
He et al., 2016; Zilly et al., 2016).
The process of architecture search and engineering is slow, costly, and laborious.
Human experts, guided by intuition, explore an extensive space of potential architectures where even minor modiﬁcations can produce unexpected results.
Ideally, an automated architecture search algorithm would ﬁnd the optimal model architecture for a given task.
Many explorations into the automation of machine learning have been made, including the optimization of hyperparameters (Bergstra et al., 2011;
Snoek et al., 2012) and various methods of producing novel model architectures (Stanley et al., 2009; Baker et al., 2016; Zoph and Le, 2016).
For architecture search, ensuring these automated methods are able to produce results similar to humans usually requires traversing an impractically large search space, assuming high quality architectures exist in the search space at all.
The choice of underlying operators composing an architecture is further typically constrained to a standard set across architectures even though recent work has found promising results in the use of non-standard operators (Vaswani et al., 2017).
We propose a meta-learning strategy for ﬂexible automated architecture search of recurrent neural networks (RNNs) which explicitly includes novel operators in the search.
It consists of three stages, outlined in Figure 1, for which we instantiate two versions.
A candidate architecture generation function produces potential RNN architectures using a highly ﬂexible DSL.
The DSL enforces no constraints on the size or complexity of the generated tree and can be incrementally constructed using either a random policy or with an RL agent.
A ranking function processes each candidate architecture’s DSL via a recursive neural network, predicting the architecture’s performance.
By unrolling the RNN representation, the ranking function can also model the interactions of a candidate architecture’s hidden state over time.
An evaluator, which takes the most promising candidate architectures, compiles their DSLs to executable code and trains each model on a speciﬁed task.
The results of these evaluations form architecture-performance pairs that are then used to train the ranking function and RL generator.
[Figure]
Incremental RL Generator
Full Training
Evaluation 107.
Full Training
Evaluation 107.
Ranking Function Candidate Architecture Generation
Evaluator
[Figure]
A D OMAIN S PECIFIC L ANGUAGE FOR D EFINING RNN S
In this section, we describe a domain speciﬁc language (DSL) used to deﬁne recurrent neural network architectures.
This DSL sets out the search space that our candidate generator can traverse during architecture search.
In comparison to Zoph and Le (2016), which only produced a binary tree with matrix multiplications at the leaves, our DSL allows a broader modeling search space to be explored.
When deﬁning the search space, we want to allow for standard RNN architectures such as the Gated
Recurrent Unit (GRU) (Cho et al., 2014) or Long Short Term Memory (LSTM) (Hochreiter and
Schmidhuber, 1997) to be deﬁned in both a human and machine readable manner.
The core operators for the DSL are 4 unary operators, 2 binary operators, and a single ternary operator: [ MM , Sigmoid , Tanh , ReLU ]
[ Add , Mult ] [ Gate3 ] .
MM represents a single linear layer with bias, i.e. MM ( x ) :
=
Wx + b .
Similarly, we deﬁne:
Sigmoid ( x ) = σ ( x ) .
The operator Mult represents element-wise multiplication:
Mult ( x, y ) =
x ◦ y .
The Gate3 operator performs a weighted summation between two inputs, deﬁned by
Gate3 ( x, y, f ) =
σ ( f ) ◦ x +(1 − σ ( f )) ◦
y.
These operators are applied to source nodes from the set [ x t , x t − 1 , h t − 1 , c t − 1 ] , where x t and x t − 1 are the input vectors for the current and previous timestep, h t − 1 is the output of the RNN for the previous timestep, and c t − 1 is optional long term memory.
The Gate3 operator is required as some architectures, such as the GRU, re-use the output of a single
Sigmoid for the purposes of gating.
While allowing all possible node re-use is out of scope for this
DSL, the Gate3 ternary operator allows for this frequent use case.
Using this DSL, standard RNN cell architectures such as the tanh RNN can be deﬁned: tanh( Add ( MM ( x t ) , MM ( h t − 1 ))) .
To illustrate a more complex example that includes Gate3 , the GRU is deﬁned in full in Appendix A.
2. A DDING SUPPORT FOR ARCHITECTURES WITH LONG TERM MEMORY
With the operators deﬁned above it is not possible to refer to and re-use an arbitrary node.
The best performing RNN architectures however generally use not only a hidden state h t but also an additional hidden state c t for long term memory.
The value of c t is extracted from an internal node computed while producing h t .
The DSL above can be extended to support the use of c t by numbering the nodes and then specifying which node to extract c t from (i.e. c t
= Node 5 ).
We append the node number to the end of the DSL deﬁnition after a delimiter.
2. E XPRESSIBILITY OF THE DOMAIN SPECIFIC LANGUAGE
While the domain speciﬁc language is not entirely generic, it is ﬂexible enough to capture most standard RNN architectures.
This includes but is not limited to the GRU, LSTM, Minimal Gate
Unit (MGU)
(Zhou et al., 2016), Quasi-Recurrent Neural Network (QRNN)
(Bradbury et al., 2016),
Neural Architecture Search Cell (NASCell) (Zoph and Le, 2016) and simple RNNs.
2. E XTENDING THE D OMAIN S PECIFIC L ANGUAGE
While many standard and non-standard RNN architectures can be deﬁned using the core DSL, the promise of automated architecture search is in designing radically novel architectures.
Such architectures should be formed not just by removing human bias from the search process but by including operators that have not been sufﬁciently explored.
For our expanded DSL, we include:
[Equation]
SeLU ] .
These extensions add inverses of currently used operators ( Sub ( a, b ) = a − b instead of addition,
Div ( a, b ) = ab instead of multiplication), trigonometric curves ( Sin and Cos are sine and cosine activations respectively, PosEnc introduces a variable that is the result of applying positional encoding (Vaswani et al., 2017) according to the current timestep), and optimizations ( LayerNorm applies layer normalization (Ba et al., 2016) to the input while SeLU is the activation function deﬁned in Klambauer et al.
2. C OMPILING A DSL DEFINITION TO EXECUTABLE CODE
For a given architecture deﬁnition, we can compile the DSL to code by traversing the tree from the source nodes towards the ﬁnal node h t .
We produce two sets of source code – one for initialization required by a node, such as deﬁning a set of weights for matrix multiplication, and one for the forward call during runtime.
For details regarding speed optimizations, refer to Appendix A2.
C ANDIDATE A RCHITECTURE GENERATION
The candidate architecture generator is responsible for producing candidate architectures that are then later ﬁltered and evaluated.
Architectures are grown beginning at the output h t and ordered to prevent multiple representations for equivalent architectures:
Growing architectures from h t up Beginning from the output node h t , operators are selected to be added to the computation graph, depicted in Figure 2.
Whenever an operator has one or more children to be ﬁlled, the children are ﬁlled in order from left to right.
If we wish to place a limit on the height (distance from h t ) of the tree, we can force the next child to be one of the source nodes when it would otherwise exceed the maximum height.
Preventing duplicates through canonical architecture ordering Due to the ﬂexibility allowed by the DSL, there exist many DSL speciﬁcations that result in the same RNN cell.
To solve the issue of commutative operators (i.e. Add ( a, b ) =
Add ( b, a ) ), we deﬁne a canonical ordering of an architecture by sorting the arguments of any commutative nodes.
Special consideration is required for non-commutative operators such as Gate3 , Sub , or Div .
For full details, refer to Appendix A3.
h t h t
Tanh
Add ;; ;; h t h t
Tanh
Add ;; MM ;; h t h t
Tanh
Add ;; MM x t x t h t h t
Tanh
Add
MM MM x t x t ;; h t h t
Tanh ;; h t h t ;; h t h t
Tanh
Add
MM MM x t x t h t  1 h t
3. I NCREMENTAL A RCHITECTURE C ONSTRUCTION USING R EINFORCEMENT L EARNING
Architectures in the DSL are constructed incrementally a node at a time starting from the output h t .
The simplest agent is a random one which selects the next node from the set of operators without internalizing any knowledge about the architecture or optima in the search space.
Allowing an intelligent agent to construct architectures would be preferable as the agent can learn to focus on promising directions in the space of possible architectures.
For an agent to make intelligent decisions regarding which node to select next, it must have a representation of the current state of the architecture and a working memory to direct its actions.
We propose achieving this with two components:
a tree encoder that represents the current state of the (partial) architecture.
an RNN which is fed the current tree state and samples the next node.
The tree encoder is an LSTM applied recursively to a node token and all its children with weights shared, but the state reset between nodes.
The RNN is applied on top of the encoded partial architecture and predicts action scores for each operation.
We sample with a multinomial and encourage exploration with an epsilon-greedy strategy.
Both components of the model are trained jointly using the REINFORCE algorithm (Williams, 1992).
As a partial architecture may contain two or more empty nodes, such as h t =
Gate 3( ∅ , ∅ , σ ( ∅ )) , we introduce a target token, T , which indicates which node is to next be selected.
Thus, in h t =
Gate ( T, ∅ , σ ( ∅ )) , the tree encoder understands that the ﬁrst argument is the slot to be ﬁlled.
3. F ILTERING C ANDIDATE A RCHITECTURES USING A R ANKING F UNCTION
Even with an intelligent generator, understanding the likely performance of an architecture is difﬁcult, especially the interaction of hidden states such as h t − 1 and c t − 1 between timesteps.
We propose to approximate the full training of a candidate architecture by training a ranking network through regression on architecture-performance pairs.
This ranking function can be speciﬁcally constructed to allow a richer representation of the transitions between c t − 1 and c t .
As the ranking function uses architecture-performance samples as training data, human experts can also inject previous best known architectures into the training dataset.
This is not possible for on-policy reinforcement learning and when done using off-policy reinforcement learning additional care and complexity are required for it to be effective (Harutyunyan et al., 2016; Munos et al., 2016).
Given an architecture-performance pair, the ranking function constructs a recursive neural network that reﬂects the nodes in a candidate RNN architecture one-to-one.
Sources nodes are represented by a learned vector and operators are represented by a learned function.
The ﬁnal vector output then passes through a linear activation and attempts to minimize the difference between the predicted and real performance.
The source nodes ( x t , x t − 1 , h t − 1 , and c t − 1 ) are represented by learned vector representations.
For the operators in the tree, we use TreeLSTM nodes (Tai et al., 2015).
All operators other than [ Gate3 , Sub , Div ] are commutative and hence can be represented by Child-Sum
TreeLSTM nodes.
The [ Gate3 , Sub , Div ] operators are represented using an N -ary TreeLSTM which allows for ordered children.
[Figure]
L o g p e r p l e x i t y o n
W i
k
i T e x t -
Unrolling the graph for accurately representing h t − 1 and c t − 1 : A strong assumption made above is that the vector representation of the source nodes can accurately represent the contents of the source nodes across a variety of architectures.
This may hold true for x t and x t − 1 but is not true for h t − 1 or c t − 1 .
The value of h t and c t are deﬁned by the operations within the given architecture itself.
To remedy this assumption, we can unroll the architecture for a single timestep, replacing h t −and c
t − 1 with their relevant graph and subgraph.
This would allow the representation of h t − 1 to understand which source nodes it had access to and which operations were applied to produce h t − 1 .
While unrolling is useful for improving the representation of h t − 1 , it is essential for allowing an accurate representation of c t − 1 .
This is as many small variations of c t − 1 are possible – such as selecting a subgraph before or after an activation – that may result in substantially different architecture performance.
E XPERIMENTS
We evaluated our architecture generation on two experiments: language modeling (LM) and machine translation (MT).
Due to the computational requirements of the experiments, we limited each experiment to one combination of generator components .
For language modeling, we explore the core DSL using randomly constructed architectures (random search) directed by a learned ranking function.
For machine translation, we use the extended DSL and construct candidate architectures incrementally using the RL generator without a ranking function
. 4. L ANGUAGE M ODELING USING R ANDOM S EARCH WITH A R ANKING F UNCTION
For evaluating architectures found during architecture search, we use the WikiText-2 dataset (Merity et al., 2017b).
When evaluating a proposed novel RNN cell c , we construct a two layer c -RNN with a 200 unit hidden size.
Aggressive gradient clipping is performed to ensure that architectures such as the ReLU RNN would be able to train without exploding gradients.
The weights of the ranking network were trained by regression on architecture-perplexity pairs using the Adam optimizer and mean squared error (MSE).
Further hyperparameters and training details are listed in Appendix B1.
Explicit restrictions on generated architectures During the candidate generation phase, we ﬁlter the generated architectures based upon speciﬁc restrictions.
These include structural restrictions and restrictions aimed at effectively reducing the search space by removing likely invalid architectures.
For Gate3 operations, we force the input to the forget gate to be the result of a sigmoid activation.
We also require the cell to use the current timestep x t and the previous timestep’s output h t − 1 to satisfy the requirements of an RNN.
Candidate architectures were limited to 21 nodes, the same number of nodes as used in a GRU, and the maximum allowed distance (height) from h t was 8 steps.
We also prevent the stacking of two identical operations.
While this may be an aggressive ﬁlter it successfully removes many problematic architectures.
These problematic architectures include when two sigmoid activations, two ReLU activations, or two matrix multiplications are used in succession – the ﬁrst of which is unlikely to be useful, the second of which is a null operator on the second activation, and the third of which can be mathematically rewritten as a single matrix multiplication.
If a given candidate architecture deﬁnition contained c t − 1 , the architecture was queried for valid subgraphs from which c t could be generated.
The subgraphs must contain c t − 1 such that c t is recurrent and must contain three or more nodes to prevent trivial recurrent connections.
A new candidate architecture is then generated for each valid c t subgraph.
Random architecture search directed by a learned ranking function Up to 50,000 candidate architecture DSL deﬁnitions are produced by a random architecture generator at the beginning of each search step.
This full set of candidate architectures are then simulated by the ranking network and an estimated perplexity assigned to each.
Given the relative simplicity and small training dataset, the ranking function was retrained on the previous full training results before being used to estimate the next batch of candidate architectures.
Up to 32 architectures were then selected for full training.
28 of these were selected from the candidate architectures with the best perplexity while the last 4 were selected via weighted sampling without replacement, prioritizing architectures with better estimated perplexities.
c t architectures were introduced part way through the architecture search after 750 valid h t architectures had been evaluated with h t architectures being used to bootstrap the c t architecture vector representations.
Figure 3 provides a visualization of the architecture search over time, showing valid h t and c t architectures.
Analyzing the BC3 cell After evaluating the top 10 cells using a larger model on WikiText-2, the top performing cell BC3 (named after the identifying hash, bc3dc7a . . .
) was an unexpected layering of two Gate3 operators,
[Equation]
(5) where ◦ is an element-wise multiplication and all weight matrices W, U, V, X
∈ R H
× H .
Equations 1 to 3 produce the ﬁrst Gate 3 while equations 4 and 5 produce the second Gate3 .
The output of the ﬁrst Gate3 becomes the value for c t after passing through a tanh activation.
While only the core DSL was used, BC3 still breaks with many human intuitions regarding RNN architectures.
While the formulation of the gates f and o are standard in many RNN architectures, the rest of the architecture is less intuitive.
The Gate3 that produces c t (equation 3) is mixing between a matrix multiplication of the current input x t and a complex interaction between c t − 1 and x t (equation 2).
In BC3, c t − 1 passes through multiple matrix multiplications, a gate, and a tanh activation before becoming c t .
This is non-conventional as most RNN architectures allow c t − 1 to become c t directly, usually through a gating operation.
The architecture also does not feature a masking output gate like the LSTM, with outputs more similar to that of the GRU that does poorly on language modeling.
That this architecture would be able to learn without severe instability or succumbing to exploding gradients is not intuitively obvious.
4.1. E VALUATING THE BC3 CELL
For the ﬁnal results on BC3, we use the experimental setup from Merity et al.
(2017a)
and report results for the Penn Treebank (Table 1) and WikiText-2 (Table 2) datasets.
To show that not any standard RNN can achieve similar perplexity on the given setup, we also implemented and tuned a
GRU based model which we found to strongly underperform compared to the LSTM, BC3, NASCell, or Recurrent Highway Network (RHN).
Full hyperparameters for the GRU and BC3 are in Appendix
B4.
Our model uses equal or fewer parameters compared to the models it is compared against.
While
BC3 did not outperform the highly tuned AWD-LSTM (Merity et al., 2017a) or skip connection
LSTM (Melis et al., 2017), it did outperform the Recurrent Highway Network (Zilly et al., 2016) and NASCell (Zoph and Le, 2016) on the Penn Treebank, where NASCell is an RNN found using reinforcement learning architecture search speciﬁcally optimized over the Penn Treebank.
[Table] [Table]
4. I NCREMENTAL A RCHITECTURE C ONSTRUCTION USING RL FOR M ACHINE T RANSLATION
For our experiments involving the extended DSL and our RL based generator, we use machine translation as our domain.
The candidate architectures produced by the RL agent were directly used without the assistance of a ranking function.
This leads to a different kind of generator: whereas the ranking function learns global knowledge about the whole architecture, the RL agent is trimmed towards local knowledge about which operator is ideal to be next.
Training details Before evaluating the constructed architectures, we pre-train our generator to internalize intuitive priors.
These priors include enforcing well formed RNNs (i.e. ensuring x t , h t − 1 , and one or more matrix multiplications and activations are used) and moderate depth restrictions (between 3 and 11 nodes deep).
The full list of priors and model details are in Appendix C1.
For the model evaluation, we ran up to 28 architectures in parallel, optimizing one batch after receiving results from at least four architectures.
As failing architectures (such as those with exploding gradients) return early, we needed to ensure the batch contained a mix of both positive and negative results.
To ensure the generator yielded mostly functioning architectures whilst understanding the negative impact of invalid architectures, we chose to require at least three good architectures with a maximum of one failing architecture per batch.
For candidate architectures with multiple placement options for the memory gate c t , we evaluated all possible locations and waited until we had received the results for all variations.
The best c t architecture result was then used as the reward for the architecture.
Baseline Machine Translation Experiment Details To ensure our baseline experiment was fast enough to evaluate many candidate architectures, we used the Multi30k English to German (Elliott et al., 2016) machine translation dataset.
The training set consists of 30,000 sentence pairs that brieﬂy describe Flickr captions.
Our experiments are based on OpenNMT codebase with an attentional unidirectional encoder-decoder LSTM architecture, where we speciﬁcally replace the LSTM encoder with architectures designed using the extend DSL.
For the hyper-parameters in our baseline experiment, we use a hidden and word encoding size of 300, 2 layers for the encoder and decoder RNNs, batch size of 64, back-propagation through time oftimesteps, dropout of 0 . 2 , input feeding, and stochastic gradient descent.
The learning rate starts at
Analysis of the Machine Translation Architecture Search Figure 4 shows the relative frequency of each operator in the architectures that were used to optimize the generator each batch.
For all the architectures in a batch, we sum up the absolute number that each operator occurs and divide by the total number of operators in all architectures of the batch.
By doing this for all batches (x-axis), we can see which operators the generator prefers over time.
Intriguingly, the generator seems to rely almost exclusively on the core
DSL ( MM , Gate3 , Tanh , Sigmoid , x t , h t − 1 ) when generating early batches.
The low usage of the extended DSL operators may also be due to these operators frequently resulting in unstable architectures, thus being ignored in early batches.
Part way through training however the generator begins successfully using a wide variety of the extended DSL ( Sub , Div , Sin , Cos , . . . ).
We hypothesize that the generator ﬁrst learns to build robust architectures and is only then capable of inserting more varied operators without compromising the RNN’s overall stability.
Since the reward function it is ﬁtting is complex and unknown to the generator, it requires substantial training time before the generator can understand how robust architectures are structured.
However, the generator seems to view the extended DSL as beneﬁcial given it continues using these operators.
Overall, the generator found 806 architectures that out-performed the LSTM based on raw test BLEU score, out of a total of 3450 evaluated architectures (23%).
The best architecture (determined by the validation BLEU score) achieved a test BLEU score of 36 . 53 respectively, compared to the standard LSTM’s 34 .
Multiple cells also rediscovered a variant of residual networks ( Add ( Transformation ( x t ) , x t ) )
(He et al., 2016) or highway networks ( Gate3 ( Transformation ( x t ) , x t , Sigmoid ( . . . )) )
(Srivastava et al., 2015).
Every operation in the core and extended DSL made their way into an architecture that outperformed the LSTM and many of the architectures found by the generator would likely not be considered valid by the standards of current architectures.
These results suggest that the space of successful RNN architectures might hold many unexplored combinations with human bias possibly preventing their discovery.
In Table 3 we take the top ﬁve architectures found during automated architecture search on the
Multi30k dataset and test them over the IWSLT 2016 (English to German) dataset (Cettolo et al., 2016).
The training set consists of 209,772 sentence pairs from transcribed TED presentations that cover a wide variety of topics with more conversational language than in the Multi30k dataset.
This dataset is larger, both in number of sentences and vocabulary, and was not seen during the architecture search.
While all six architectures achieved higher validation and test BLEU on Multi30k than the LSTM baseline, it appears the architectures did not transfer cleanly to the larger IWSLT dataset.
This suggests that architecture search should be either run on larger datasets to begin with (a computationally expensive proposition) or evaluated over multiple datasets if the aim is to produce general architectures.
We also found that the correlation between loss and BLEU is far from optimal:
[Table]
We did not perform any hyperparameter optimizations on either dataset to avoid unfair comparisons, though the initial OpenNMT hyperparameters likely favored the architectures performing exceptionally well on the loss sometimes scored poorly on BLEU.
It is also unclear how these metrics generalize to perceived human quality of the model (Tan et al., 2015) and thus using a qualitatively and quantitatively more accurate metric is likely to beneﬁt the generator.
For hyper parameters of the IWSLT model, refer to Appendix C3.
R ELATED W ORK
Architecture engineering has a long history, with many traditional explorations involving a large amount of computing resources and an extensive exploration of hyperparamters (Jozefowicz et al., 2015; Greff et al., 2016;
Britz et al., 2017).
The approach most similar to our work is Zoph and
Le (2016) which introduces a policy gradient approach to search for convolutional and recurrent neural architectures.
Their approach to generating recurrent neural networks was slot ﬁlling, where element-wise operations were selected for the nodes of a binary tree of speciﬁc size.
The node to produce c t was selected once all slots had been ﬁlled.
This slot ﬁlling approach is not highly ﬂexible in regards to the architectures it allows.
As opposed to our DSL, it is not possible to have matrix multiplications on internal nodes, inputs can only be used at the bottom of the tree, and there is no complex representation of the hidden states h t − 1 or c t − 1 as our unrolling ranking function provides.
Many other similar techniques utilizing reinforcement learning approaches have emerged such as designing CNN architectures with Q-learning (Baker et al., 2016).
Neuroevolution techniques such as NeuroEvolution of Augmenting Topologies (NEAT) (Stanley and Miikkulainen, 2001) and HyperNEAT (Stanley et al., 2009) evolve the weight parameters and structures of neural networks.
These techniques have been extended to producing the non-shared weights for an LSTM from a small neural network (Ha et al., 2016) and evolving the structure of a network (Fernando et al., 2016; Bayer et al., 2009)
C ONCLUSION
We introduced a ﬂexible domain speciﬁc language for deﬁning recurrent neural network architectures that can represent most human designed architectures.
It is this ﬂexibility that allowed our generators to come up with novel combinations in two tasks.
These architectures used both core operators that are already used in current architectures as well as operators that are largely unstudied such as division or sine curves.
The resulting architectures do not follow human intuition yet perform well on their targeted tasks, suggesting the space of usable RNN architectures is far larger than previously assumed.
We also introduce a component-based concept for architecture search from which we instantiated two approaches: a ranking function driven search which allows for richer representations of complex
RNN architectures that involve long term memory ( c t ) nodes, and a Reinforcement Learning agent that internalizes knowledge about the search space to propose increasingly better architectures.
As computing resources continue to grow, we see automated architecture generation as a promising avenue for future research.
1. A RCHITECTURE OPTIMIZATIONS DURING DSL COMPILATION
To improve the running speed of the RNN cell architectures
[Table]
we can collect all matrix multiplications performed on a single source node ( x t , x t − 1 , h t −
[Table]
or c t − 1 ) and batch them into a single matrix multiplication.
As an example
[Table]
this optimization would simplify the LSTM’s 8 small matrix multiplications (4 for x t , 4 for h t − 1 ) into 2 large matrix multiplications.
This allows for higher GPU utilization and lower CUDA kernel launch overhead.
2. PREVENTING DUPLICATES THROUGH CANONICAL ARCHITECTURE ORDERING
There exist many possible DSL speciﬁcations that result in an equivalent RNN cell.
When two matrix multiplica- tions are applied to the same source node, such as Add ( MM ( x t ) , MM ( x t )) , a matrix multiplication reaching an equivalent result can be achieved by constructing a speciﬁc matrix and calculating MM ( x t ) .
Additional equivalences can be found when an operator is commutative, such as Add ( x t , h t − 1 ) being equivalent to the reordered Add ( h t − 1 , x t ) .
We can deﬁne
A PPENDIX B: L ANGUAGE M ODELING 1. B ASELINE EXPERIMENTAL SETUP AND HYPERPARAMETERS
The models are trained using stochastic gradient descent (SGD) with an initial learning rate of 20 .
Training continues for 40 epochs with the learning rate being divided by 4 if the validation perplexity has not improved since the last epoch.
Dropout is applied to the word embeddings and outputs of layers as in Zaremba et al.
at a rate of 0 . 2 .
Weights for the word vectors and the softmax were also tied (Inan et al., 2016; Press and
Wolf, 2016).
Aggressive gradient clipping ( 0 . 075 ) was performed to ensure that architectures such as the ReLU RNN would be able to train without exploding gradients.
The embeddings were initialized randomly between [ − 0 .
During training, any candidate architectures that experienced exploding gradients or had perplexity over 500 after ﬁve epochs were regarded as failed architectures.
Failed architectures were immediately terminated.
While not desirable, failed architectures still serve as useful training examples for the ranking function.
2. R ANKING FUNCTION HYPERPARAMETERS
For the ranking function, we use a hidden size of 128 for the TreeLSTM nodes and a batch size of 16 .
We use L 2 regularization of 1 × 10 − 4 and dropout on the ﬁnal dense layer output of 0 . 2 .
As we are more interested in reducing the perplexity error for better architectures, we sample architectures more frequently if their perplexity is lower.
For unrolling of the architectures, a proper unroll would replace x t with x t − 1 and x t − 1 with x t − 2 .
We found the ranking network performed better without these substitutions however and thus only substituted h t − 1 to h t − 2 and c t − 1 to c t − 2 .
3. H OW BASELINE EXPERIMENTAL SETTINGS MAY IMPAIR ARCHITECTURE SEARCH
The baseline experiments that are used during the architecture search are important in dictating what models are eventually generated.
As an example, BC3 may not have been discovered if we had used all the standard regularization techniques in the baseline language modeling experiment.
Analyzing how variational dropout (Gal and Ghahramani, 2016) would work when applied to BC3 frames the importance of hyperparameter selection for the baseline experiment.
On LSTM cells, variational dropout (Gal and Ghahramani, 2016) is only performed upon h t − 1 , not c t − 1 , as otherwise the long term hidden state c t would be destroyed.
For BC3, equation 6 shows that the ﬁnal gating operation mixes c t and h t − 1 .
If variational dropout is applied to h t − 1 in this equation, BC3’s hidden state h t will have permanently lost information.
Applying variational dropout only to the h t − 1 values in the two gates f
and o ensures no information is lost.
This observation provides good justiﬁcation for not performing variational dropout in the baseline experiment given that this architecture (and any architecture which uses h t − 1 in a direct manner like this) would be disadvantaged otherwise.
4. H YPERPARAMETERS FOR PTB AND W IKI T EXT -2 BC3 EXPERIMENTS
For the Penn Treebank BC3 language modeling results, the majority of hyper parameters were left equal to that of the baseline AWD-LSTM.
The model was trained for 200 epochs using NT-ASGD with a learning rate of 15, a batch size of 20 and BPTT of 70 .
The variational dropout for the input, RNN hidden layers, and output were set to 0 .
25 , and 0 .
4 respectively.
Embedding dropout of 0 .
1 was used.
The word vectors had dimensionality of 400 and the hidden layers had dimensionality of 1080 .
The BC3 used 3 layers with weight drop of 0 .
5 applied to the recurrent weight matrices.
Activation regularization and temporal activation regularization of 2 and 2 were used.
Gradient clipping was set to 0 .
Finetuning was run for an additional 13 epochs.
For the WikiTextBC3 language modeling results, the parameters were kept equal to that of the Penn Treebank experiment.
The model was run for a total of 100 epochs with 7 epochs of ﬁnetuning.
For the Penn Treebank GRU language modeling results, the hyper parameters were equal to that of the BC3 PTB experiment but with a hidden size of 1350 , weight drop of 0 .
3 , learning rate of
A PPENDIX C: M ACHINE T RANSLATION 1. B ASELINE EXPERIMENTAL SETUP AND HYPERPARAMETERS FOR RL
To represent an architecture with the encoder, we traverse through the architecture recursively, starting from the root node.
For each node, the operation is tokenized and embedded into a vector.
An LSTM is then applied to this vector as well as the result vectors of all of the current node’s children.
Note that we use the same LSTM for every node but reset its hidden states between nodes so that it always starts from scratch for every child node.
Based on the encoder’s vector representation of an architecture, the action scores are determined as follows: action scores = softmax ( linear ( LSTM ( ReLU ( linear ( architecture encoding )))))
We then choose the speciﬁc action with a multinomial applied to the action scores.
We encourage exploration by randomly choosing the next action according to an epsilon-greedy strategy with (cid:15)
= 0 .
The reward that expresses how well an architecture performed is computed based on the validation loss.
We re-scale it according to a soft exponential so that the last few increases (distinguishing a good architecture from a great one) are rewarded more.
The speciﬁc reward function we use is R ( loss ) = 0 .
2 × (140 − loss ) + 4 0 .
× (140 − loss ) − 50) which follows earlier efforts to keep the reward between zero and 140.
For pre-training the generator, our list of priors are:
1. to maintain a depth between 3 and 11 2. use the current input x t , the hidden state h t − 1 , at least one matrix multiplication ( MM ) and at least one activation function 3.
do not use the same operation as a child 4.
do not use an activation as a child of another activation 5.
do not use the same inputs to a gate 6.
that the matrix multiplication operator ( MM ) should be applied to a source node
2. F ULL DSL FOR DEEPLY NESTED Gate3 AND RESIDUAL x t WITH POSITIONAL ENCODING
For obvious reasons these DSL deﬁnitions would not ﬁt within the result table.
They do give an indication of the ﬂexibility of the DSL however, ranging
[Table] [Table]
Residual x t with positional encoding
[Table]
3. H YPER PARAMETERS FOR THE IWSLT’16 MODELS
For the models trained on the IWSLT’16 English to German dataset, the hyper parameters were kept largely equivalent to that of the default OpenMT LSTM baseline.
All models were unidirectional and the dimensionality of both the word vectors and hidden states were 600 , required as many of the generated architectures were residual in nature.
The models were 2 layers deep, utilized a batch size of 64 , and standard dropout of 0 .between layers.
The learning rate began at 1 and decayed by 0 .
5 whenever validation perplexity failed to improve.
When the learning rate fell below 0 .
03 training was ﬁnished.
Models were evaluated using a batch size of 1 to ensure RNN padding did not impact the results.
4. F INDING AN ARCHITECTURE FOR THE DECODER AND ENCODER / DECODER
We also brieﬂy explored automated architecture generation for the decoder as well as for the encoder and decoder jointly which yielded good results with interesting architectures but performances fell short of the more promising approach of focusing on the encoder.
With additional evaluated models, we believe both of these would yield comparable or greater results.
5. V ARIATION IN ACTIVATION PATTERNS BY GENERATED ARCHITECTURES
Given the differences in generated architectures, and the usage of components likely to impact the long term hidden state of the RNN models, we began to explore the progression of the hidden state over time.
Each of the activations differs substantially from those of the other architectures even though they are parsing the same input.
As the input features are likely to not only be captured in different ways but also stored and processed differently, this suggests that ensembles of these highly heterogeneous architectures may be effective.
6. E XPLORATION VERSUS E XPLOITATION