L EARNING I NVARIANTS THROUGH S OFT U NIFICATION
Anonymous authors
Paper under double-blind review
A BSTRACT
Human reasoning involves recognising common underlying principles across many examples by utilising variables.
The by-products of such reasoning are invariants that capture patterns across examples such as “if someone went somewhere then they are there” without mentioning speciﬁc people or places.
Humans learn what variables are and how to use them at a young age, and the question this paper addresses is whether machines can also learn and use variables solely from examples without requiring human pre-engineering.
We propose Uniﬁcation Networks that incorporate soft uniﬁcation into neural networks to learn variables and by doing so lift examples into invariants that can then be used to solve a given task.
We evaluate our approach on four datasets to demonstrate that learning invariants captures patterns in the data and can improve performance over baselines.
I NTRODUCTION
Humans have the ability to process symbolic knowledge and maintain symbolic thought (Unger & Deacon, 1998).
When reasoning, humans do not require combinatorial enumeration of examples but instead utilise invariant patterns with placeholders replacing speciﬁc entities.
Symbolic cognitive models (Lewis, 1999) embrace this perspective with the human mind seen as an information processing system operating on formal symbols such as reading a stream of tokens in natural language.
The language of thought hypothesis (Morton & Fodor, 1978) frames human thought as a structural construct with varying sub-components such as “X went to Y”.
By recognising what varies across examples, humans are capable of lifting examples into invariant principles that account for other instances.
This symbolic thought with variables is learned at a young age through symbolic play (Piaget, 2001).
For instance a child learns that a sword can be substituted with a stick (Frost et al., 2004) and engage in pretend play.
[Equation]
Z
[Equation]
Z :lily is A :green what colour is X :bernhard
A :
[Equation]
where X :bernhard denotes a variable with default symbol bernhard.
This single invariant accounts for all the training and test examples.
Although variables are inherent in models of computation and symbolic formalisms, as in ﬁrst-order logic (Russell & Norvig, 2016), they are pre-engineered and used to solve speciﬁc tasks by means of uniﬁcation or assignments that bound variables to given values.
However, when learning from data only, being able to recognise when and which symbols should take on different values, i.e. symbols that can act as variables, is crucial for lifting examples into general principles that are invariant across multiple instances.
Figure 1 shows the invariant learned by our approach: if someone is the same thing as someone else then they have the same colour.
With this invariant, our approach can solve all of the training and test examples in task 16 of the bAbI dataset (Weston et al., 2016).
In this paper we address the question of whether a machine can learn and use the notion of a variable , i.e. a symbol that can take on different values.
For instance, given an example of the form “bernhard is a frog” the machine would learn that the token “bernhard” could be someone else and the token “frog” could be something else.
If we consider uniﬁcation a selection of the most appropriate value for a variable given a choice of values, we can reframe it as a form of attention.
Attention models (Bahdanau et al., 2015; Luong et al., 2015; Chaudhari et al., 2019) allow neural networks to focus, attend to certain parts of the input often for the purpose of selecting a relevant portion.
Since attention mechanisms are also differentiable they are often jointly learned within a task.
This perspective motivates our idea of a uniﬁcation mechanism that utilises attention and is therefore fully differentiable which we refer to as soft uniﬁcation .
Hence, we propose an end-to-end differentiable neural network approach for learning and utilising the notion of a variable that in return can lift examples into invariants used by the network to perform reasoning tasks.
Speciﬁcally, we (i) propose a novel architecture capable of learning and using variables by lifting a given example through soft uniﬁcation, (ii) present the empirical results of our approach on four datasets and (iii) analyse the learned invariants that capture the underlying patterns present in the tasks.
Our implementation using Chainer (Tokui et al., 2015) is publicly available at [link removed](anonymous link provided with submission) .
S OFT U NIFICATION
Reasoning with variables involves identifying what variables are, the setting in which they are used as well as the process by which they are assigned values.
When the varying components, i.e. variables, of an example are identiﬁed, the remaining structure can be lifted into an invariant which then accounts for multiple other instances.
Deﬁnition 1 (Variable) .
Given a set of symbols S , a variable X is deﬁned as a pair X (cid:44) ( x , s d ) where s d ∈ S is the default symbol of the variable and x is a discrete random variable of which the support is S .
The representation of a variable φ V ( X : s d ) is equal to the expected value of the corresponding random variable x given the default symbol s d :
[Equation]
where φ : S → R d is a d -dimensional real-valued feature of a symbol s .
For example, φ could be an embedding and φ V ( X : s d ) would become a weighted sum of symbol embeddings as in conventional attention models.
The default symbol of a variable is intended to capture the variable’s bound meaning following the idea of referants by Frege (1948).
We denote variables using X , Y , A etc.
such as X :bernhard where X is the name of the variable and bernhard the default symbol as shown in Figure 1.
Deﬁnition 2 (Invariant) .
Given a structure (e.g. list, grid) G over S , an invariant is a pair I (cid:44) ( G, ψ ) where G ∈ G is the invariant example such as a tokenised story with tokens as symbols and ψ : S → [0 , 1] is a function representing the degree to which the symbol is considered a variable.
Thus, the ﬁnal representation of a symbol s included in G , φ I ( s ) is:
[Equation]
the linear interpolation between its representation φ ( s ) and its variable bound value with itself as the default symbol φ V ( X : s ) .
We adhere to the term invariant and refrain from mentioning rules, unground rules, etc. used in logicbased formalisms, e.g. Muggleton & de Raedt (1994), since neither the invariant structure needs to be rule-like nor the variables carry logical semantics.
This distinction is clariﬁed in Section 6.
Deﬁnition 3 (Uniﬁcation) .
Given an invariant I and an example K ∈ G , uniﬁcation binds the variables in I to symbols in K .
Deﬁned as a function g : I × G → G , uniﬁcation binds variables by computing the probability mass functions, P in equation 1, and returns the uniﬁed representation using equation 2.
The probability mass function of a variable X : s d is:
[Equation]
where φ U : S → R d is the unifying feature of a symbol and φ U ( K ) ∈ R | K |× d is applied element wise to symbols in K .
If g is differentiable, it is referred to as soft uniﬁcation.
We distinguish φ from φ U to emphasise that the unifying properties of the symbols might be different from their representations.
For example, φ ( bernhard ) could represent a speciﬁc person whereas φ U ( bernhard ) the notion of someone .
Overall soft uniﬁcation incorporates 3 learnable components: φ, ψ, φ U which denote the base features, variableness and unifying features of a symbol respectively.
Given an upstream, potentially task speciﬁc, network f : G → S , an invariant I ∈ I and an input example K ∈ G with a corresponding desired output a ∈ S , the following holds:
[Equation]
where f now predicts based on the uniﬁed representation produced by g .
In this work, we focus on g , the invariants it produces together with the interaction of f ◦ g .
U NIFICATION N ETWORKS
Since soft uniﬁcation is end-to-end differentiable, it can be incorporated into existing task-speciﬁc upstream architectures.
We present 3 architectures that model f ◦ g using multi-layer perceptrons (MLP), convolutional neural networks (CNN) and memory networks (Weston et al., 2015) to demonstrate the ﬂexibility of our approach.
In all cases, the d dimensional representation of symbols are learnable embeddings φ ( s ) =
O [ s ] T E with E ∈ R | S |×
d randomly initialised by N (0 , 1) and O [ s ] the one-hot encoding of the symbol.
The variableness of symbols is a learnable weight ψ w ( s ) =
σ ( w s ) where w ∈ R | S | and σ is the sigmoid function.
We consider every symbol independently a variable irrespective of its surrounding context and leave further contextualised formulations as future work.
The underlying intuition of this conﬁguration is that a useful symbol for a correct prediction might need to take on other values for different inputs.
This usefulness can be viewed as the inbound gradient to the corresponding w s parameter and ψ w ( s ) acting as a gate.
For further model details including the size of the embeddings, please refer to Appendix A.
Uniﬁcation MLP (UMLP) ( f : MLP, g : RNN)
We combine soft uniﬁcation into a multi-layer perceptron to process ﬁxed length inputs.
In this case, the structure G is a sequence of symbols with a ﬁxed length l , e.g. a sequence of digits 4234 .
Given an embedded input φ ( k ) ∈ R l × d , the upstream MLP computes the output symbol based on the ﬂattened representations f ( φ ( k ))
=
softmax( hE T ) where h ∈ R d is the output of the last layer.
However, to compute the unifying features φ U , deﬁnition 3, g uses a bi-directional GRU (Cho et al., 2014) running over φ ( k ) such that φ U ( k )
=
W U Φ( k ) where Φ( k ) ∈
R
d is the hidden state of the GRU at symbol k and W
U ∈ R
d × d is a learnable parameter.
This model emphasises the ﬂexibility around the boundary of f ◦ g and that the unifying features can be computed in any differentiable manner.
[Figure]
Uniﬁcation CNN (UCNN) ( f : CNN, g : CNN)
Given a grid of embedded symbols φ ( K ) ∈
R w × h ×
d
where w is the width and h the height, we use a convolutional neural network such that the ﬁnal prediction is f ( φ ( K ))
=
softmax(( W h + b ) E T ) where h this time is the result of global max pooling and W , b are learnable parameters.
We also model g using a separate convolutional network with the same architecture as f and set φ U ( k ) =
c 2 (relu( c 1 ( k ))
where c 1 , c 2 are the convolutional layers.
The grid is padded with 0s to obtain w × h × d after each convolution such that every symbol has a unifying feature.
This model conveys how soft uniﬁcation can be adapted to the speciﬁcs of the domain for example by using a convolution in a spatially structured input.
Uniﬁcation Memory Networks (UMN) ( f : MemNN, g : RNN)
Soft uniﬁcation does not need to happen prior to f in a f ◦ g fashion but can also be incorporated at any intermediate stage multiple times.
To demonstrate this ability, we unify the symbols at different memory locations at each iteration of a Memory Network (Weston et al., 2015).
Memory networks can handle a list of lists
[Table]
structure such as a tokenised story as shown in Figure 2.
The memory network f uses the ﬁnal hidden state of a bi-directional GRU (outer squares in Figure 2) as the sentence representations to compute a context attention.
At each iteration, we unify the words between the attended sentences using the same approach in UMLP with another bi-directional GRU (inner diamonds in Figure 2) for unifying features φ U ( bernhard ) =
W U Φ( bernhard ) .
Following equation 2, the new uniﬁed representation of the memory slot is computed and f uses it to perform the next iteration.
Concretely, g produces an uniﬁcation tensor U ∈ R M × m × N × d where M and m is the number of sentences and words in the invariant respectively, and N is the number of sentences in the example such that after the context attentions are applied over M and N , we obtain φ ( k ) ∈
R m × d as the uniﬁed sentence at that iteration.
Note that unlike in the UMLP case, the sentences can be of varying length.
The prediction is then softmax( W h JI + b ) where h JI is the hidden state of the invariant after J iterations.
This setup, however, requires pre-training f such that the context attentions match the correct sentences.
A task might contain different questions such as “Where is X?” and “Why did X go to Y?”.
To let the models differentiate between questions and potentially learn different invariants, we extend them with a repository of invariants I ∈
I and aggregate the predictions from each invariant.
One simple approach is to sum the predictions of the invariants (cid:80) I ∈
I f ◦ g
( I, K ) used in UMLP and UCNN.
Another approach could be to use features from the invariants such as memory representations in the case of UMN.
For UMN, we weigh the predictions using a bilinear attention η
based on the hidden states at the ﬁrst iteration h
I and h 0 K such that η = softmax
( h 0 I W h
0 T K ) .
To initially form the repository of invariants, we use the bag-of-words representation of the questions and ﬁnd the most dissimilar ones based on their cosine similarity as a heuristic to obtain varied examples.
D ATASETS
We use 4 datasets consisting of context, query and an answer ( C , q , a ) : ﬁxed length sequences of symbols, shapes of symbols in a grid, story based natural language reasoning with the bAbI (Weston et al., 2016) dataset and logical reasoning represented as logic programs, examples shown in Tablewith further samples in Appendix B.
In each case we use an appropriate model: UMLP for ﬁxed length sequences, UCNN for grid and UMN for iterative reasoning.
We use synthetic datasets of which the data generating distributions are known to evaluate not only the quantitative performance but also the quality of the invariants learned by our approach.
Fixed Length Sequences We generate sequences of length l = 4 with 8 unique symbols represented as digits to predict (i) a constant, (ii) the head of the sequence, (iii) the tail and (iv) the duplicate symbol.
We randomly generate 1000 triples and then only take the unique ones to ensure the test split contains unseen examples.
The training is then performed over a 5-fold cross-validation.
Grid To spatially organise symbols, we generate a grid of size 3 × 3 with 8 unique symbols organised into 2 × 2 box of identical symbol, a vertical, diagonal or horizontal sequence of length 3, a cross or a plus shape and a triangle.
In each task we predict (i) the identical symbol, (ii) the head of the sequence, (iii) the centre of the cross or plus and (iv) the corner of the triangle respectively.
We follow the same procedure from sequences and randomly generate 1000 discarding duplicate triples.
bAbI The bAbI dataset has become a standard benchmark for evaluating memory based networks.
It consists of 20 synthetically generated natural language reasoning tasks (refer to Weston et al.
(2016) for task details).
We take the 1k English set and use 0.1 of the training set as validation.
Each token is lower cased and considered a unique symbol.
Following previous works (Seo et al., 2017;
Sukhbaatar et al., 2015), we take multiple word answers also to be a unique symbol in S .
Logical Reasoning To demonstrate the ﬂexibility of our approach and distinguish our notion of a variable from that used in logic based formalisms, we generate logical reasoning tasks in the form of logic programs using the procedure by Cingillioglu & Russo (2019).
The tasks involve learning f ( C, Q ) =
True ↔ C (cid:96) Q over 12 classes of logic programs exhibiting varying paradigms of logical reasoning including negation by failure (Clark, 1978).
We generate 1k and 10k logic programs per task for training with 0.1 as validation and another 1k for testing.
We set the arity of literals to 1 or 2 using one random character from the English alphabet for predicates and constants, e.g. p ( p ) and an upper case character for logical variables, e.g. p ( X ) .
E XPERIMENTS
We probe three aspects of soft uniﬁcation: the impact of uniﬁcation on performance over unseen data, the effect of multiple invariants and data efﬁciency.
To that end, we train UMLP and UCNN with and without uniﬁcation, UMN with pre-training using 1 or 3 invariants over either the entire training set or only 50 examples.
Every model is trained 3 times via back-propagation using
Adam (Kingma & Ba, 2015) on an Intel Core i7-6700 CPU using the following objective function:
[Equation]
where L nll is the negative log-likelihood with sparsity regularisation over ψ at τ = 0 .
1 to discourage the models from utilising spurious number of variables.
For UMLP and UCNN, we set λ K = 0 , λ U = 1 for training just the uniﬁed output and the converse for the non-unifying versions.
To pre-train the UMN, we start with λ K = 1 , λ U = 0 for 40 epochs then set λ U
= 1 to jointly train the uniﬁed output.
For iterative tasks, the mean squared error between hidden states ( h jI − h jK )at each iteration j and, in the strongly supervised cases, the negative log-likelihood for the context attentions using provided supporting facts are also added to the objective function.
Further details such as batch size and total number of epochs are available in Appendix C.
[Figure]
Figure 3 portrays how soft uniﬁcation generalises better to unseen examples in test sets - the same sequence or grid never appears in both the training and test sets as outlined in Section 4 - over plain models.
Despite f ◦ g having more trainable parameters than f alone, the models with uniﬁcation not only maintain higher accuracy in each iteration and solve the tasks in as few as 250 iterations with ≤ 1000 training examples but also improve accuracy by ≈ 0 .
3 when trained with only ≤per task.
We believe soft uniﬁcation architecturally biases the models towards learning structural patterns which in return achieves better results on recognising common patterns of symbols across examples.
Results with multiple invariants are identical and the models seem to ignore the extra invariants due to the fact that the tasks can be solved with a single invariant and the regularisation applied on ψ zeroing out unnecessary invariants; further results in Appendix D. The ﬂuctuations in accuracy around iterations 750 to 1000 in UCNN are also caused by penalising ψ which forces the model to relearn the task with less variables half way through training.
[Table]
Following Tables 2 and 3, we observe a trend of better performance through strong supervision, more data per task and using only 1 invariant.
We believe strong supervision aids with selecting the correct sentences to unify and in a weak setting the model attempts to unify arbitrary context sentences often failing to follow the iterative reasoning chain.
The increase in performance with more data and strong supervision is consistent with previous work reﬂecting how f ◦ g can be bounded by the efﬁcacy of f modelled as a memory network.
As a result, only in the supervised case do we observe a minor improvement over MemNN by 0.7 in Table 2 and no improvement in the weak case over comparable memory based networks in Table 3.
This dependency on f also limits the ability of f ◦ g to learn from 50 examples per task failing 17/20 and 12/12 of bAbI and logical reasoning tasks respectively.
The increase in error rate with 3 invariants, we speculate, stems from having more parameters and more pathways in the model rendering training more difﬁcult.
[Table]
A NALYSIS
After training, we can extract the learned invariants by applying a threshold on ∀ s ∈ S : ψ ( s ) > t indicating whether a symbol is used as a variable or not.
We set t = 0 .
for all datasets except for bAbI, we use t = 0 .
The magnitude of this threshold seems to depend on the amount of regularisation τ , equation
[Table]
and the number of training steps along with batch size all controlling how much ψ is pushed towards 0.
Sample invariants shown in Figure 4 describe the common patterns present in the tasks with parts that contribute towards the ﬁnal answer becoming variables.
Extra symbols such as is or travelled do not emerge as variables, as shown in Figure 4a; we attribute this behaviour to the fact that changing the token travelled to went does not inﬂuence the prediction but changing the action, the value of Z :left to ‘picked’ does.
However, based on random initialisation, our approach can convert an arbitrary symbol into a variable and let f compensate for the uniﬁcations it produces.
For example, the invariant “ X :8 5 2 2” could predict the tail of another example by unifying the head with the tail using φ U , equation 3, of those symbols.
Further examples are shown in Appendix D. Pre-training f as done in UMN seems to produce more robust and consistent invariants compared to immediately training f ◦ g since, we speculate, by equationf might encourage g ( I, K ) ≈ K .
Interpretability versus Ability A desired property of interpretable models is transparency (Lipton, 2018).
A novel outcome of the learned invariants in our approach is that they provide an approximation of the underlying general principle present in the data such as the structure of multi-hop reasoning shown in Figure 4e.
However, certain aspects regarding the ability of the model such as how it performs temporal reasoning, are still hidden inside f .
In Figure 4b, although we observe
Z :morning as a variable, the overall learned invariant captures nothing about how changing the value of Z :morning alters the behaviour of f .
The model might look before or after a certain time point
Y :john Z :left the X :football
Y :john travelled to the A :ofﬁce where is the X :football
A :ofﬁce
[Figure]
) bAbI task 2, two supporting facts.
The also learns since can also drop or pick up ob- potentially affecting answer.
this Z :morning X :bill went to the Y :school yesterday X :bill journeyed to the A :park where was X :bill before the Y :school
A :park
[Figure] [Figure]
[Figure] [Figure]
[Figure]
centre
[Figure]
[Figure]
X :i ( T ) ← Z :l ( T
[Figure]
Z
[Figure]
l
[Figure]
← R :x ( U )
[Figure]
R :
[Figure] [Figure]
←
[Figure]
:n
[Figure] [Figure]
[Figure] [Figure]
occasionally producing uniﬁcations such bathroom to the bathroom” and still f predicting, perhaps unsurprisingly, the correct answer as bathroom.
Hence, regularising ψ with the correct amount τ in equation 5 to reduce the capacity of uniﬁcation seems critical in extracting not just any invariant but one that represents the common structure.
Soft uniﬁcation from equation 3 reveals three main patterns: one-to-one, one-to-many or many-toone bindings as shown in Figure 5; further examples are in Appendix D. Figure 5a captures what one might expect uniﬁcation to look like where variables unify with their corresponding counterparts, e.g. X :bernhard with brian and Y :frog with lion.
However, occasionally the model can optimise to use less variables and squeeze the required information into a single variable, for example by binding
Y :bathroom to john and kitchen as shown in Figure 5b.
We believe this occurs due to the sparsity constraint on ψ ( s ) encouraging the model to be as conservative as possible.
Finally, if there are more variables than needed as in Figure 5c, we observe a many-to-one binding with Y :w and Z :e mapping to the same constant q .
This behaviour begs the question how does the model differentiate between p ( q ) and p ( q, q ) .
We speculate the model uses the magnitude of ψ ( w )
= 0 .
037 and ψ ( e ) = 0 .to
encode the difference despite both variables unifying with the same constant.
[Figure]
R ELATED W ORK
Learning an underlying general principle in the form of an invariant is often the means for arguing generalisation in neural networks.
For example, Neural Turing Machines (Graves et al., 2014) are tested on previously unseen sequences to support the view that the model might have captured the underlying pattern or algorithm.
In fact, Weston et al.
(2015) claim “MemNNs can discover simple linguistic patterns based on verbal forms such as (X, dropped, Y), (X, took, Y) or (X, journeyed to, Y) and can successfully generalise the meaning of their instantiations.”
However, this claim is based on the output of f and unfortunately it is unknown whether the model has truly learned such a representation or indeed is utilising it.
Our approach sheds light to this ambiguity and presents these linguistic patterns explicitly as invariants ensuring their utility through g without solely analysing the output of f on previously unseen symbols.
Although we associate these invariants with our existing understanding of the task to mistakenly anthropomorphise the machine, for example by thinking it has learned X :mary as someone , it is important to acknowledge that these are just symbolic patterns.
In these cases, our interpretations do not necessarily correspond to any understanding of the machine, relating to the Chinese room argument made by Searle (1980).
Learning invariants by lifting ground examples is related to least common generalisation (Reynolds, 1970) by which inductive inference is performed on facts (Shapiro, 1981) such as generalising went(mary,kitchen) and went(john,garden) to went(X,Y) .
Unlike in a predicate logic setting, our approach allows for soft alignment and therefore generalisation between varying length sequences.
Existing neuro-symbolic systems (Broda et al., 2002) focus on inducing rules that adhere to given logical semantics of what variables and rules are.
For example, δILP (Evans & Grefenstette, 2018) constructs a network by rigidly following the given semantics of ﬁrst-order logic.
Similarly, Lifted
Relational Neural Networks (Sourek et al., 2015) ground ﬁrst-order logic rules into a neural network while Neural Theorem Provers (Rocktschel & Riedel, 2017) build neural networks using backwardchaining (Russell & Norvig, 2016) on a given background knowledge base with templates.
However, the notion of a variable is pre-deﬁned rather than learned with a focus on presenting a practical approach to solving certain problems, whereas our motivation stems from a cognitive perspective.
At ﬁrst it may seem the learned invariants, Section 6, make the model more interpretable; however, this transparency is not of the model f
but of the data.
The invariant captures patterns that potentially approximates the data generating distribution but we still do not know how the model f uses them upstream.
Thus, from the perspective of explainable artiﬁcial intelligence (XAI) (Adadi &
Berrada, 2018), learning invariants or interpreting them do not constitute an explanation of the reasoning model f even though “if someone goes somewhere then they are there” might look like one.
Instead, it can be perceived as causal attribution (Miller, 2019) in which someone being somewhere is attributed to them going there.
This perspective also relates to gradient based model explanation methods such as Layer-Wise Relevance Propagation (Bach et al., 2015) and Grad-CAM (Selvaraju et al., 2017; Chattopadhay et al., 2018).
Consequently, a possible view on ψ , equation 2, is a gradient based usefulness measure such that a symbol utilised upstream by f to determine the answer becomes a variable similar to how a group of pixels in an image contribute more to its classiﬁcation.
Finally, one can argue that our model maintains a form of counterfactual thinking (Roese, 1997) in which soft uniﬁcation g creates counterfactuals on the invariant example to alter the output of f towards the desired answer, equation 4.
The question where Mary would have been if Mary had gone to the garden instead of the kitchen is the process by which an invariant is learned through multiple examples during training.
This view relates to methods of causal inference (Pearl, 2019; Holland, 1986) in which counterfactuals are vital as demonstrated in structured models by Pearl (1999).
C ONCLUSION
We presented a new approach for learning variables and lifting examples into invariants through the usage of soft uniﬁcation.
Evaluating on four datasets, we analysed how Uniﬁcation Networks perform comparatively to existing similar architectures while having the beneﬁt of lifting examples into invariants that capture underlying patterns present in the tasks.
Since our approach is end-toend differentiable, we plan to apply this technique to multi-modal tasks in order to yield multi-modal invariants for example in visual question answering.
A M ODEL D ETAILS A. U NIFICATION MLP & CNN
Uniﬁcation MLP (UMLP)
To model f as a multi-layer perceptron, we take symbol embeddings of size d = 16 and ﬂatten sequences of length l = 4 into an input vector of size φ ( k ) ∈ R 64 .
The
MLP consists of 2 hidden layers with tanh non-linearity of sizes 2 d and d respectively.
To process the query, we concatenate the one-hot encoding of the task id to φ ( k ) yielding a ﬁnal input of size 64 + 4 = 68 .
For uniﬁcation features φ U , we use a bi-directional GRU with hidden size d and an initial state of 0 .
The hidden state at each symbol is taken with a linear transformation to give φ U ( s ) =
W U Φ( s ) where Φ( s ) is the hidden state of the biGRU.
The variable assignment is then computed as an attention over the according to equation 3.
Uniﬁcation CNN (UCNN)
We take symbols embeddings of size d = 32 to obtain an input grid φ ( K ) ∈ R 3 × 3 × 32 .
Similar to UMLP, for each symbol we append the task id as a one-hot vector to get an input of shape 3 × 3 × (32 + 4) .
Then f consists of 2 convolutional layers with d ﬁlters each, kernel size of 3 and stride 1.
We use relu non-linearity in between the layers.
We pad the grid with 2 columns and 2 rows to a 5 × 5 such that the output of the convolutions yield again a hidden output
H ∈ R 3 × 3 × d of the same shape.
As the ﬁnal hidden output h , we take a global max pool to over H to obtain h ∈ R d .
Uniﬁcation function g is modelled identical to f without the max pooling such that φ U ( K ij )
=
H (cid:48) ij where H (cid:48) is the hidden output of the convolutional layers.
A. U NIFICATION M EMORY N ETWORKS
Unlike previous architectures, with UMN we interleave g into f .
We use embedding sizes of d =and model f with an iterative memory network.
We take the ﬁnal hidden state of a bi-directional
GRU, with initial state 0 , Φ M to represent the sentences of the context C and query q in a ddimensional vector M i =
Φ M ( C i ) and the query m q =
Φ M ( q ) .
The initial state of the memory network is h
= m q .
At each iteration j :
[Equation]
where Φ A is another d -dimensional bi-directional GRU and ρ ( x , y ) =
[ x ; y ; x (cid:12) y ; ( x − y ) 2 ] with (cid:12) the element-wise multiplication and [; ] the concatenation of vectors.
Taking β j as the context attention, we obtain the next state of the memory network:
[Equation]
and iterate J many times in advance.
The ﬁnal prediction becomes f ( C , q ) =
softmax( W h J + b ) .
All weight matrices W and bias vectors b are independent but are tied across iterations.
B G ENERATED D ATASET S AMPLES
[Table]
[Table]
C T RAINING D ETAILS C. U NIFICATION MLP & CNN
Both uniﬁcation models are trained on a 5-fold cross-validation over the generated datasets foriterations with a batch size of 64.
We don’t use any weight decay and save the training and test accuracies every 10 iterations, as presented in Figure 3.
C. U NIFICATION M EMORY N ETWORKS
We again use a batch size of 64 and pre-train f for 40 epochs then f together with g for 260 epochs.
We use epochs for UMN since the dataset sizes are ﬁxed.
To learn g alongside f , we combine error signals from the uniﬁcation of the invariant and the example.
Following equation 4, the objective function not only incorporates the negative log-likelihood L nll of the answer but also the mean squared error between intermediate states h jI and h jK at each iteration as an auxiliary loss:
[Equation]
We pre-train by setting λ U = 0 for 40 epochs and then set λ U
= 1 .
For strong supervision we also compute the negative log-likelihood L nll for the context attention β j , described in Appendix A, at each iteration using the supporting facts of the tasks.
We apply a dropout of 0.1 for all recurrent neural networks used and only for the bAbI dataset weight decay with 0.001 as the coefﬁcient.
D F URTHER R ESULTS
[Figure]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
AND 31.
[Table]
[Table] [Table]
[Table] [Table]
OR 25.
3.0 Transitivity 50.
[Table] [Table]
4.0 AND NBF 51.
[Table]
8.0 OR NBF
[Table]
[Table] [Table]
[Table] [Table]
Mean
[Table] [Table]
[Table] [Table]
[Figure]
X :m ( Y :e ) (cid:96) X :m ( Y :e )
X :a ( Y :w , Z :e ) (cid:96) X :a ( Y :w , Z :e )
X :m ( T ) (cid:96) X :m ( c )
[Figure]
) (cid:96) X
[Figure]
Last invariant on
11 lifts the around negation by denoted
[Figure]
X 00 10 0 Y 0 Y1 X0 26 4 Y
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]