A S IMPLE R ECURRENT U NIT WITH R EDUCED T ENSOR P RODUCT R EPRESENTATIONS
Anonymous authors
Paper under double-blind review
A BSTRACT
Widely used recurrent units, including Long-short Term Memory (LSTM) and the
Gated Recurrent Unit (GRU), perform well on natural language tasks, but their ability to learn structured representations is still questionable.
Exploiting reduced
Tensor Product Representations (TPRs) — distributed representations of symbolic structure in which vector-embedded symbols are bound to vector-embedded structural positions — we propose the TPRU, a simple recurrent unit that, at each time step, explicitly executes structural-role binding and unbinding operations to incorporate structural information into learning.
A gradient analysis of our proposed
TPRU is conducted to support our model design, and its performance on multiple datasets shows the effectiveness of our design choices.
Furthermore, observations on a linguistically grounded study demonstrate the interpretability of our TPRU.
I NTRODUCTION
Recurrent units are widely used in sequence modelling tasks, including text, speech, DNA/RNA sequencing data, etc.
Due to their ability to produce distributed vector representations for sequences with various lengths, and their property of being a universal function approximators (Sch ¨ afer &
Zimmermann, 2006), they quickly succeeded in numerous research areas.
The training algorithm, backpropagation through time (BPTT, Werbos et al., 1990), has remained the same across the years, with known gradient vanishing and exploding issues.
Many theoretically justiﬁable or empirically useful regularisations (Pascanu et al., 2013; Kanai et al., 2017) have been proposed to ease the learning process and to smooth the gradient ﬂow for long sequences, and informative analysis methods have been conducted to visualise the information encoded in the learnt vector representations (Zeiler & Fergus, 2013; Karpathy et al., 2015).
The main difﬁculty in understanding the learning process of a particular RNN unit comes from the extreme nonlinear activation functions, which cause the gradient ﬂow to be chaotic.
Here, we aim to propose a recurrent unit that has fewer nonlinear operations by incorporating Tensor Product Representations (Smolensky, 1990).
Tensor product representation (TPR) is an instantiation of general neural-symbolic computing in which symbol structures are embedded in a vector space via a ﬁller-role decomposition : the structure is captured by a set of N roles
r i (e.g., left-child-of-root), each of which is bound to a ﬁller f i (a symbol or substructure).
A TPR embedding of a symbol structure S derives from vector embeddings of the roles { r i } and their ﬁllers { f i } via the outer or tensor product: S =
(cid:80) Ni =1 r i ⊗ f i =
(cid:80)
Ni =1 r i f (cid:62) i = RF (cid:62) , where R and F respectively denote matrices having the role or ﬁller vectors of S as columns.
Each r i ⊗ f
i is the embedding of a role-ﬁller binding: a constituent; ⊗ is the binding operation.
The unbinding operation returns the ﬁller of a particular role in S ; it is performed by the inner product: f i = u (cid:62) i S , where u i is the dual of r i , satisfying u (cid:62) i r j = δ ij ≡ 1 if i = j else 0 .
Letting U be the matrix with columns { u i } , we have U (cid:62) R =
I .
Our goal is to propose a recurrent unit that uses a reduced version of a tensor product representation, a vector rather than an order-2 tensor: this arises by reducing the dimension of the ﬁller embeddings to 1; with role embeddings of dimension d , the representation becomes b = (cid:80) Ni =1 f i r i ∈ R d . {+We let this sum range over all N possible roles, letting f i = 
 when role r i is unﬁlled.+}
Then at time t , the hidden state of our RNN — which we call the binding complex — will be b t = (cid:80)
{+Ni =
 ( f t ) i+} r i where f t is the vector of ﬁller values ( f t ) i ∈ R at time t . Now binding is performed by a simple matrix-vector product, and unbinding by a simple inner product:
[Equation]
In this paper:
• In the context of related work (Sec. 2) and based on the reduced tensor product representations introduced above, we propose a gated recurrent unit, the TPRU (Sec. 3), along with a set of analyses of the gradient ﬂow to justify our decisions in architecture design (Sec. 4).
Even though the number of nonlinear operations is reduced, the success in experiments attests to the ﬂexibility of our TPRU in learning and our analysis on derivatives further supports it.
• Our proposed TPRU demonstrates strong performance on the Logical Entailment task (Evans et al., 2018), Natural Language Inference datasets (Williams et al., 2017), and language modelling, while having comparably many parameters as the GRU, and signiﬁcantly fewer than the widely-used LSTM (Sec. 5).
• The instantiation of reduced TPRs allows our model to learn decomposable representations, and the ﬁller f t provides an interface for us to inspect the information encoded in b t (Sec. 6).
We conducted a linguistically grounded study, which shows that our TPRU is capable of distinguishing multiple senses of a polysemous word and discovering structured information in sentences.
R ELATED W ORK
Recent efforts on understanding RNNs includes these three classes: (i) theoretical analysis through advancements in dynamic systems on the functions learnt by RNN units (Chang et al., 2019); (ii) linguistically grounded analysis by developing sets of probing tasks to illustrate the properties of learnt vector representations of RNNs (Conneau et al., 2018; Chrupała & Alishahi, 2019); and (iii) introducing inductive biases into the architecture of networks and comparing the behaviour of learnt models against those of widely-used counterparts.
Our work falls in this last category: learning structured representations by incorporating the inductive biases inherent in TPRs.
Some prior work has incorporated TPRs into RNN representations.
Question-answering on SQuAD (Rajpurkar et al., 2016) was addressed (Palangi et al., 2017) with an LSTM with hidden state given by a single TPR binding; the present work deploys complexes containing multiple bindings (albeit reduced ones).
TPR-style unbinding was applied in caption generation (Huang et al., 2018), but the representations were deep-learnt and not explicitly designed to be TPRs as in the present work.
A contracted version of TPRs, Holographic Reduced Representations, was utilised to decompose input space and output space with ﬁller-role decomposition (Luo et al., 2019).
Our work differs from these prior papers in that TPR ﬁller-role binding and unbinding operations are explicitly carried out in our proposed recurrent unit, and the two operations directly determine the update of the hidden states.
Closest to the present work is Schlag & Schmidhuber (2018), which performed questionanswering by using TPRs to embed simple knowledge graphs as tensors; our model does not deploy graph-based inference, which is less natural for the inference tasks we address than it is for the question-answering task addressed in Schlag & Schmidhuber (2018).
Our proposed TPRU inherits the explicit binding and unbinding operations in TPRs in the sense that a set of role vectors are shared across time steps and sequences to serve as a basis for composing distributed vector representations.
We provide a set of analyses from both theoretical and empirical perspectives: gradient analysis of our model helps us make sense of the architecture design, while empirical results from three different categories of tasks show the effectiveness of the proposed model. Additional study on the linguistic and learning aspects provide evidence that our model indeed inherits the merits of TPRs, which enhance interpretability and the capability of learning structured representations.
P ROPOSED R ECURRENT U NIT : T HE TPRU
As shown in both LSTM (Hochreiter & Schmidhuber, 1997) and GRU (Chung et al., 2014) design, a gating mechanism helps the hidden state at the current time step to directly copy information from the previous time step, and alleviate vanishing and exploding gradient issues.
Our proposed recurrent unit deploys a single gate, adopting the input gate of the GRU, omitting the reset gate.
We expect that further augmenting our TPRU to 2 gates as in the GRU, or 3 gates as in the LSTM, will lead to yet better performance.
At each time step, the TPRU receives two input vectors: b t − 1 ∈ R d , the binding complex from the
TPRU at the previous time-step, and x t ∈ R d (cid:48) from the external input; it produces b t ∈ R d .
An input gate g t ∈ R d is computed to produce a weighted sum of the information generated at the current time step, (cid:101) b t ∈ R d , and the previous binding complex b t − 1 , b t = g t ◦ (cid:101) b t + (1 − g t ) ◦
b t − 1 , where g t = σ ( W b b t − 1 + W x x t ) , (2) σ ( · ) is the logistic sigmoid function (applied element-wise), ◦ is the Hadamard (element-wise) product, and W b ∈ R d × d and W x ∈ R d × d (cid:48) are matrices of learnable parameters.
As we now explain, the calculation of (cid:101) b t is carried out by the unbinding and binding operations of TPRs (Eq. 1).
3. U NBINDING O PERATION
Consider a set of hypothesised unbinding vectors U
= [ u 1 , u 2 , ..., u N ] ∈ R d × N .
At time step t , these can be used to unbind ﬁllers f b,t from the previous binding complex b t − 1 using Eq.
We posit two matrices V x ∈ R d × d (cid:48) and V b ∈ R d × d that transform the current input x t ∈ R d (cid:48) and the binding complex b t − 1 into the binding space R d yielding ﬁllers f x,t and f b,t :
[Equation]
Normalisation is applied for stable learning by gradient descent, and the speciﬁc technique (Yang et al., 2018)
we adopted here also enforces sparsity on both f b,t and f x,t :
[Equation]
4) where ( (cid:101) f t ) n = max(0 , ( f b,t ) n + b b ) + max(0 , ( f x,t ) n + b x ) (5) in which b b and b x are two scalar parameters for stable learning, and ( · ) n refers to the n -th entry of the vector in the parentheses.
3. B INDING O PERATION
Given a hypothesised set of binding role vectors R =
[ r 1 , r 2 , ..., r N ] ∈ R d × N , applying the binding operation in Eq.
1 to the ﬁllers f t at time t gives the candidate update (cid:101) b t for the binding complex,
[Equation]
The gating mechanism controls the weighted sum of the candidate vector (cid:101) b t and the previous binding complex b t − 1 to produce a binding complex b t at the current time step, as given by Eq. 2.
3. U NBINDING AND B INDING R OLE V ECTORS
In the TPRU, there is a matrix of role vectors R used for the binding operation and a matrix of unbinding vectors U used for the unbinding operation.
To control the number of parameters in our TPRU, rendering it independent of N , instead of directly learning two sets of vectors, a set of N ddimensional random vectors E is sampled from a normal distribution, and two linear projections
W r , W u ∈ R
d × d are learnt to transform E to U
= W u E and R = W r E .
Therefore, in total, our proposed TPRU has six learnable matrices: W u , W r , V b ,
W b ∈ R
d × d and
V x , W x ∈ R d × d (cid:48) .
The number of parameter matrices the same as that of a GRU and less than that of an LSTM, and, when d (cid:48) <
d , the number of learnable parameters is less than that of a GRU.
G RADIENT A NALYSIS AND C OMPLEXITY
This section analyses the derivatives in our proposed TPRU, and aims to justify the architecture design.
For simplicity, we omit the derivatives w.r.t.
bias terms in our derivations as biases can be absorbed into the weight matrices by adding an extra component 1 to the input.
As known, the gradient of an RNN cell is accumulated in time ∂ b T [Equation]
b 1 = (cid:81) T − [Equation] =1 ∂ b t +1 ∂ b t , therefore, studying the properties of the derivative of the output at the current time step b t (Eq. 2) w.r.t.
the output of the previous time step b t − 1 is important for understanding the behaviour of a recurrent unit. For our TPRU, the derivative is shown below:
[Equation]
where Λ 1 , Λ 2 and Λ 3 are diagonal matrices in R d × d , in which ( Λ 1 ) ii =
( (cid:101) b t ) i ( g t ) i (1 − ( g t ) i ) , ( Λ 2 ) ii = ( g t ) i and ( Λ 3 ) ii
= (1 − ( g t ) i ) .
It is clear that the gating mechanism smooths the gradient ﬂow as it jitters the Jacobian matrix by adding small values to the diagonal terms, and from a dynamic system perspective, it helps stabilise the learning system (Moya-Cessa & Soto-Eguibar, 2011). Now we focus on the following derivative which carries the main information of our proposed TPRU.
[Equation]
where N t , S b,t ∈ R N × N are diagonal matrices, in which ( N t ) nn = 2 / ( (cid:101) f t ) n and ( S b,t ) nn = 1 ( f b,t ) n > 0 , and ( J t )
mn =
( f b,t ) m
( δ mn − ( f b,t ) n ) .
Appendix A shows that J t is a positive semi-deﬁnite matrix, and its eigenvalues are all smaller than max n ( f b,t ) n ≤ 1 .
There are three crucial design choices, including (1) learnable parameter matrices W r and W u to transform the basis matrix E separately, (2) learnable linear transformation V b on the binding complex at each time step b t , and (3) the relu-squared normalisation in Eq. 4.
The following analysis shows that certain modiﬁcations could lead to issues during learning, and helps understand the necessity of each component.
Modiﬁcation I. [Equation]
and softmax normalisation in Eq.
The derivative of the softmax normalisation of outputs w.r.t. the inputs is simply J t as S b,t = N t =
I N .
Given the conditions, the derivative can be simpliﬁed to:
[Equation]
In this case, as the derivative gets multipled several times by BPTT, the gradient w.r.t. each parameter matrix either explodes or vanishes depending on the spectrum of W r E .
Also, a careful initialisation is required otherwise the gradient could overﬂow or underﬂow after the ﬁrst iteration.
Modiﬁcation II.
W r = W u , and softmax normalisation.
Here we allow learnable parameter matrices V b to transform the binding complex from the previous time step b t − 1 , thus, the derivative of (cid:101) b t w.r.t.
b t − 1 and that to i -th row of V b are shown below:
[Equation]
Since W r EJ t E (cid:62) W (cid:62) r (cid:23) 0 , ∂ (cid:101) b t /∂
( V b ) i · always has a non-negative inner product with b t − 1 , which means that the update of V b always contains a positive portion of b t − 1 .
The same situation also applies when updating V x as it keeps a positive portion of x t as well.
It limits the learning system from eliminating accumulated useless information.
Although it is still possible that, after backpropagation through several time steps, the positive portion of b t − 1 could be suppressed, we set W r and
W u to be independent matrices so that the above constraint isn’t imposed.
Modiﬁcation III. Softmax normalisation.
The only modiﬁcation here is to replace the normalisation step in Eq.
4 in our TPRU to a softmax function that also normalises (cid:101) f t to form a distribution f t .
At each time step, the derivative of (cid:101) b t w.r.t.
any parameter matrix before the normalisation step contains ∂ f t /∂ (cid:101) f t , and in the softmax normalisation, only the relative difference, ( (cid:101) f t ) n − max m ( (cid:101) f t ) m , ∀ n ∈ { 1 , 2 , ..., N } , matters.
Therefore, there is no explicit regularisation in the derivative of ∂ (cid:101) b t /∂
b t − 1 = W r EJ t E (cid:62) W (cid:62) u V b to penalise extreme values in (cid:101) f t , which could lead to gradients with large norms and make the learning process unstable.
This may also explain why, in the transformer module (Vaswani et al., 2017), layer normalisation (Ba et al., 2016) is necessary, as it scales the gradient by the variance of the input.
We don’t rule out the possibility that there exists a suitable initialisation which gives stable learning and decent performance, but the normalisation step we adopted from prior work (Yang et al., 2018) avoids heavy tuning of initialisations for stable learning
Ours.
Here, we recall that the derivative of ∂ (cid:101) b t /∂ b t − 1 is in Eq. 8.
The term N t scales the derivative by the magnitude of the input (cid:101)
f t to Eq.
4, which regularises the norm of the gradient to each parameter matrix.
Speciﬁcally, the following derivative indicates the update of i -th row in W u :
[Equation]
b t [Equation] where N t normalises the contribution from f b,t and f x,t , and S b,t and S x,t select positive contributions only, thus the normalisation step in Eq.
4 helps the gradients w.r.t.
parameter matrices to stay stable.
Eq.
11 also shows that the update of each row in W u depends on a sparse linear transformation of W r , which could potentially avoid overﬁtting as not all dimensions in the representation space are utilised and our TPRU learns to discover the representation space.
Given the derivative above, it is easy to see that the presampled matrix E acts similarly as the preconditioning matrix in solving large linear systems, and with samples from a normal distribution or zero-centred uniform distribution, it smooths the spectrum of the Jacobian matrix and stabilizes the learning system.
[Table]
Time Complexity in Forward Computation
Table 1 presents the time complexity of calculating each of our proposed TPRU, LSTM and GRU for a single time step.
In the calculation, we only kept the terms that involve second order of variables.
As shown, the time complexity of TPRU is in between LSTM and GRU when the number of role vectors N is less than the dimension of the recurrent unit d .
However, the number of parameters in our TPRU is fewer than in an LSTM, and comparable to a GRU.
In addition, as concerns nonlinearities, our TPRU contains max( x, 0) 2 only N times, and sigmoid functions only d times; this is fewer than sigmoid/tanh squashing functions appearing 5 d times in an
LSTM and 3 d times in a GRU.
The nonlinear operations are indeed reduced in our TPRU, and the following sections show that our TPRU still provides strong performance on evaluation tasks.
T ASKS AND T RAINING D ETAILS
Three entailment tasks, including an abstract logical entailment task (Evans et al., 2018) and two relatively realistic natural language entailment tasks, including Multi-genre Natural Language Inference (MNLI, Williams et al.
and Question Natural Language Inference (QNLI, Wang et al.
(2018)), as well as a language modelling task on a large corpus and a small one, (WikiText-103 and WikiText-2 (Merity et al., 2018)), are considered to demonstrate the learning ability of our proposed TPRU.
Each of the three entailment tasks provides pairs of samples, and for each pair, the model needs to tell whether the ﬁrst (the premise) entails the second (the hypothesis).
Language modelling is used to demonstrate the ability of our proposed model on sequential modelling.
Experiments are conducted in PyTorch (Paszke et al., 2017) with the Adam optimiser (Kingma &
Ba, 2014) and gradient clipping (Pascanu et al., 2013).
Reported results are averaged from the results of three random initialisations.
Our proposed TPRU has only one input gate and also has similar number of parameters, so the GRU is the most relevant comparison partner.
LSTM is also included as it is widely-used as well and it generally yields better performance than GRU does.
For each model, hyperparameters are searched based on the small selected validation set on each task, including dropout rates, learning rate and batch size.
For entailment tasks, we tested both plain and BiDAF architectures (Seo et al., 2017).
On both NLI tasks, ELMo (Peters et al., 2018) encoding is applied for embedding words into contextualised vectors.
More details about the tasks and the training details can be found in Sections B and C in the Appendix.
D ISCUSSION
On the Logical Entailment task in Table 2, when it comes to larger dimensionality ( N = 512 ), our TPRU appears to be more stable than GRU during learning as we observed that the GRU with the [Table] BiDAF architecture failed to converge in two out of three trials.
On certain test sets, our (singlygated) TPRU slightly underperforms the LSTM, possibly due to the three gates (and more parameters) of the LSTM: future research could incorporate more sophisticated gating mechanisms into the TPRU.
On the MNLI tasks in Table
[Table]
our proposed TPRU consistently outperforms the LSTM when a large number of role vectors is deployed.
We omit the results from GRUs as they are worse than LSTMs and often unstable in learning.
Unexpectedly, all models perform better on the mismatched dev set than on the matched one, perhaps because it happens to be easier.
[Table]
On the language modelling task on {+WikiText-
+}
and WikiText-
[Table]
our TPRU provides similar perplexity on the test set to the LSTM, as shown in Table 4.
The learning plot in Figure 4 shows that our TPRU learns faster on the training set than the GRU and the LSTM.
The learning plot and the performance table both serve as evidential support for the gradient analysis of our model.
6. P OLYSEMY
In TPR, multiple senses of a single polysemous word can be encoded by binding the same symbol that represents the polysemous word to different role vectors { r i } . TPR is composing a distributed representation that keeps multiple senses of a polysemous word and recovering each sense with no information loss if a large number of dimensions is available.
In our case, the magnitude of each entry in f t determines the contribution of each role vector to the update candidate (cid:101) b t (Eq. 6).
[Table] We train our TPRU with a BiDAF architecture on the MNLI dataset [Table] and word vectors are obtained from fastText (Bojanowski et al., 2017) as the input to our model so that the vector representation of a polysemous word stays the same even when it is presented in different contexts.
After learning, our model provides an interface for us to check whether a speciﬁc role vector correlates to a speciﬁc sense of a polysemous word in different phrases in the dev set by computing β t = arg max n ( f t ) n .
For example, a common polysemous word bank has two senses when it is used as a noun.
In the dev set of MNLI, the only case when bank is used in the river-context is in the phrase “ bank ﬁshing ”, and the rest belong to the other sense in the context of economy.
We observed that β is the same when bank occurs in “ bank ﬁshing ” and we denote it as β river , thus, P ( β = β river | bank in the river-context )
= 1 .
It is also important to see if the speciﬁc β river is dedicated to the word bank in the river context.
After examining the dev set, we have P ( bank in the river-context | β = β river ) =
P ( bank in “ the west bank ” | β = β river ) = 0 . 5 ; clearly, the sense of “ bank ” relevant to the phrase “ the west bank ” is the one related to rivers, not ﬁnances.
6. POS T AGGING
We calculated the pointwise mutual information (PMI) between the following two terms: 1) The maximally selected role vector at each time step β t = arg max n ( f t ) n and 2)
The part-of-speech tag (POS Tag) provided by a pretrained tagger.
For a given POS tag, the PMIs of role vectors are sorted in a descending order, and Table 5 presents the top two PMI values for those POS tags for which these two values differ by at least 0 . 5 : this includes 7 out of 17 ﬁne-grained POS tags.
The idea is that if the top PMI value is sufﬁciently greater than the next one, and hence all the rest, then it is justiﬁed to say that each POS tag in the table shown below has a role ’dedicated’ to it.
[Table]
As the model is only trained on MNLI to predict the entailment relationship between a given pair of sentences without access to POS tags, it is interesting that some role vectors become associated with speciﬁc POS tags.
Since POS tags carry structural information of a given sentence, and since the roles learnt by the TPRU are informative about the POS tags, we have evidence that the roles that explicitly deﬁne the structure of the model’s representations are capturing some of the distinctions between words that are believed to be fundamental to linguistic structure.
As pointed out in prior work, models trained on MNLI focus on words in lexical (non-functional) categories in determining the entailment relationship between a sentence pair; here we see that POS tags of lexical words, including NOUN, PROPN, VERB, etc., have a larger gap between the tophighest PMI values as they are apparently emphasised by the learning signal.
This ﬁnding suggests that our learnt model might help to identify the biases in datasets — which could be helpful in improving fairness and inclusiveness.
The above two empirical observations point out that our proposed TPRU inherits the interpretability of the original TPRs, and provides a certain level of human-readability.
It provides a simple interface for us to analyse the learnt model through the explicit binding and unbinding operations.
6. E FFECT OF I NCREASING THE N UMBER OF R OLE V ECTORS
In the original TPRs, roles are typically ﬁlled by at most a single symbol, so the total number of distinct roles indicates the number of symbols that can be used in the representation of a single data item.
Since each symbol is capable of representing a speciﬁc substructure of the input data, increasing the number of role vectors eventually leads to more highly structured representations if there is no limit on the dimensionality of role vectors.
Experiments are conducted to show the effect of increasing the number of role vectors on the performance on both Logical Entailment and MNLI.
As shown in Table 3, adding more role vectors into our proposed TPRU gradually improves the performance on the two entailment tasks.
Figure 1 presents the learning curves, including training loss and accuracy, of our proposed TPRU with varying number of role vectors on the two entailment tasks.
As shown in the graphs, incorporating more role vectors leads to not only better performance, but also faster convergence during training.
The observation is consistent on both the Logical Entailment and MNLI tasks.
[Figure]
T r a i n i n g
A cc u r a c y
[Figure]
8 10 12 Epochs T r a i n i n g A cc u r a c y (d) MNLI Interestingly, on the Logical Entailment task, the performance of our proposed TPRU with only eight role vectors matches that of a GRU with the same dimensionality.
The number of role vectors required in our TPRU to approximately match the performance of a GRU and an LSTM on the MNLI task is much less than the dimension of the hidden states, which suggests that the distributed vector representations learnt in both the LSTM and GRU are highly redundant and can be reduced to fewer dimensions; this also implies that the LSTM and GRU are not able to extensively exploit the representation space even when the dimensionality of the model is larger than that required for solving the task.
Meanwhile, our proposed TPRU explores sparse weighted combinations of role vectors in Eq. 4, and it helps the model to specialise individual role vectors for different purposes, which eventually leads to a more effective exploitation of the representation space.
C ONCLUSION
We have proposed a simple recurrent unit that leverages the explicit binding and unbinding operations in previously proposed TPRs in a reduced fashion, therefore it is named the TPRU.
The explicit execution arms the proposed model with the ability to learn decomposable representations, and by sharing the role vectors across time steps and sequences, our TPRU provides research an intuitive interface through f t to study the information encoded in the learnt distributed vector representations.
Since recurrent neural networks are prone to suffer from the gradient vanishing and exploding issues, we conduct a thorough analysis of the derivatives and the gradient update of our proposed TPRU on each time step.
The conducted analysis supports our decisions in designing TPRU, and it shows that it encourages relatively more stable gradient ﬂow through BPTT.
The time complexity for executing one step of our TPRU is lower than that of LSTM, and it has the same or a smaller number of parameters than the GRU does, which justiﬁes the simplicity of our recurrent unit.
The empirical evaluation on ﬁve learning tasks in three different categories demonstrate the effectiveness of our TPRU as it performs on a par with or better than the LSTM.
The study on increasing the number of role vectors and plots on the loss curves during learning shows that our model converges faster than the GRU and the LSTM in general and it is a result of the stable gradient ﬂow through multiple time steps.
In addition, our proposed TPRU enables a faster convergence rate and better performance by increasing the number of role vectors.
A study on correlating the role vectors with linguistic properties of languages shows that our TPRU inherits the strong interpretability of TPRs.
In the toy example involving a polysemous word bank, our model can differentiate multiple senses of the word, and can assign a similar sense, even in a named entity, to its closely related one.
Also, the PMI values between POS tags and maximally selected role vectors imply that our TPRU is capable of discovering syntactically-structured linguistic information.
Given the current pressing need for interpretable machine learning and learning decomposable representations, our proposed TPRU can be regarded as a more light-weight and human-readable alternative to the widely-used LSTM or GRU.
Future work will focus on incorporating full TPRs into the design of an RNN unit and into the Transformer module.
A PPENDIX A P ROOF
J t is a PSD matrix.
For any non-zero vector v , we have
[Equation]
[Equation] Therefore, J t is a PSD matrix.
B D ETAILS ABOUT T ASKS
Three entailment tasks, including an abstract logical entailment task and two relatively realistic natural language entailment tasks, as well as a language modelling task on a large corpus and a small one, are considered to demonstrate that the learning ability of our proposed TPRU.
Each of the three entailment tasks provides pairs of samples, and for each pair, the model needs to tell whether the ﬁrst (the premise) entails the second (the hypothesis).
Language modelling is used to demonstrate the ability of our proposed model on sequential modelling.
As our goal is to learn structured vector representations, the proposed TPRU serves as an encoding function, which learns to process a proposition or sentence one token at a time, and produce a vector representation.
During learning, two vector representations are produced by the same recurrent unit given a pair of samples, then a simple feature engineering method (e.g. concatenation of the two representations) is applied to form an input vector for subsequent classiﬁcation.
In general, with a simple classiﬁer, e.g. a linear classiﬁer or a multi-layer perceptron with a single hidden layer, the learning process forces the encoding function to produce high-quality representations of samples, either propositions or sentences; better representations enable stronger performance.
B. L OGICAL E NTAILMENT (P ROPOSITIONAL LOGIC )
In propositional logic, for a pair of propositions, A and B , the value of A (cid:15) B is independent of the identities of the shared variables between A and B , and is dependent only on the structure of the expression and the connectives in each subexpression.
Because of α -equivalence, p ∧ q (cid:15) q holds no matter how we replace variable p or q with any other propositional variables.
3 Logical entailment is thus a good test-bed for evaluating a model’s ability to carry out abstract, highly structure-sensitive reasoning (Evans et al., 2018).
Theoretically, it is possible to construct a truth table that contains as rows/worlds all possible combinations of values of variables in both propositions A and B ; the value of A (cid:15) B can be checked by going through every entry in each row.
As the logical entailment task emphasises reasoning on connectives, it requires the learnt distributed vector representations to encode the structure of any given proposition to excel at the task.
The Logical Entailment dataset 4 has balanced positive ( A (cid:15) B ) and negative ( A (cid:50) B ) classes and a representative validation set.
Five test sets are provided to evaluate the generalisation ability at different difﬁculty levels: some test sets have signiﬁcantly more variables and operators than the training and validation sets (see Table 2).
B. N ATURAL L ANGUAGE I NFERENCE
Natural Language Inference — determining whether a premise sentence entails a hypothesis sentence — combines the structure-dependence of logical entailment with the NLP tasks of inferring word meaning in context as well as the hierarchical relations among sentence constituents.
Two datasets are considered, which are Multi-genre NLI and QNLI (Williams et al., 2017; Wang et al., 2018).
Compared to logical entailment, the inference and reasoning in NLI also relies on the identities of words in sentences in addition to their structure.
More importantly, the ambiguity and polysemy of language lead to the impossibility of creating a truth table that lists all cases.
Therefore, NLI is an intrinsically hard task.
The Multi-genre Natural Language Inference (MNLI) dataset (Williams et al., 2017) collects sentence pairs in ten genres; only ﬁve genres are available in the training set, while all ten genres are presented in the development set.
QNLI (Wang et al., 2018) is collected from Stanford Question Answering dataset (SQuAD) (Rajpurkar et al., 2016), on which a model needs to predict whether the context sentence contains the answer to the question sentence or not.
The relation between sentences in a pair must be classiﬁed as Entailment, Neutral or Contradiction.
The performance of a model on the ﬁve ‘mismatched’ genres that exist in the development but not the training set reﬂects how well the structure encoded in distributed vector representations of sentences learnt from genres seen in training generalises to sentence pairs in unseen genres.
As the nature of NLI tasks requires inferring both word meaning and structure of constituents in a given sentence, supervised training signals from labelled datasets enforce an encoding function to analyse meaning and structure at the same time during learning, which eventually forces distributed vector representations of sentences produced from the learnt encoding function to be structured.
To the extent that the inferences required in an NLI task are demanding of proper analysis of the logical structure implicit in sentence, an inductive bias that enhances the ability to learn to encode the structures of sentences should enhance success on the MNLI task.
B. L ANGUAGE M ODELLING
Language modelling is a fundamental task to test generalisation and sequential modelling.
Recent research has shown that models trained on language modelling tasks yield representations that can transfer well to other natural language tasks (Peters et al., 2018;
Radford et al., 2018;
Devlin et al., 2018).
We follow the autoregressive setting of decomposing the joint probability on tokens into a product of conditional probabilities on generating the current token given tokens in the left context. [Equation]
[Equation]
This requires the learnt representation to capture the potentially highly structured information accumulated prior to the current time step in order to make a conﬁdent prediction of the current token.
A proper inductive bias towards constructing structured representations should enhance performance on the task.
Two corpora used in our experiment are WikiText-103 and WikiText-2 (Merity et al., 2018), in which one contains 103 million tokens in the training set while the other only has 2 million.
The results will tell us how well our proposed TPRU generalises across different training scales.
C D ETAILS ABOUT A RCHITECTURE C. P LAIN A RCHITECTURE - I NFERENCE :
For both the Logical Entailment task and the MNLI task, we train two-layer recurrent networks built using our proposed TPRU, networks built using its direct comparison counterpart, the GRU, as well as LSTMs.
A global max-pooling over time is applied to the hidden states produced by each recurrent unit at all time steps to generate the vector representation for a given logical proposition or sentence.
Given a pair of generated representations u and v , a vector is constructed to represent the difference between two vectors [ u ; v ; | u − v | ; u ◦ v ] , where u ◦ v is the Hadamard product and | u − v | is the absolute difference, and the vector is fed into a multi-layer perceptron which has only one hidden layer with ReLU activation function.
The feature engineering and the choice of classiﬁer are suggested by prior work (Seo et al., 2017; Wang et al., 2018).
Symbolic Vocabulary Permutation (Evans et al., 2018) is applied as data augmentation during learning on the logical entailment task: exploiting α -equivalence, this systematically replaces the variables in a proposition pair with randomly sampled alternative variables, as only connectives matter on this task.
Table 2 presents the results.
For the MNLI task, the Stanford Natural Language Inference (SNLI) dataset (Bowman et al., 2015) is included as additional training data as recommended (Wang et al., 2018), and ELMo (Peters et al., 2018) is applied for producing word vectors.
The best model is chosen according to the averaged classiﬁcation accuracy on the ﬁve matched and ﬁve mismatched dev sets.
The results are presented in Table 3.
SOTA models come from Alibaba DAMO
NLP Group, and the results are available on the GLUE leaderboard.
C. B I DAF A RCHITECTURE - I NFERENCE :
Bi-directional Attention Flow ( BiDAF ) (Seo et al., 2017) has been successfully applied to various
NLP tasks (Seo et al., 2017; Chen et al., 2017), and provides strong performance on NLI tasks (Wang et al., 2018).
The BiDAF architecture contains a layer for encoding two input sequences, and another for encoding the concatenation of the output from the ﬁrst layer and the context vectors determined by the bi-directional attention mechanism.
In our experiments, the dimensions of both layers are set to be the same, and the same type of recurrent unit is applied across both layers.
The same settings are used for experiments on LSTM and our TPRU models.
Speciﬁcally, for TPRU, the recurrent units in both layers have the same number of role vectors.
Other learning details are as in the plain architecture.
Tables 2 and 3 respectively present results on the Logical Entailment and the MNLI task, with BiDAF results in parentheses.
C. L ANGUAGE M ODELLING :
On the language modelling task, the same two-layer architecture (Merity et al., 2018) is adopted in our experiments on LSTM and our proposed TPRU.
For WikiText-103, Bytepair encoding (BPE) vocabulary with 100k merges (10k for WikiText-2) are applied to encode the corpus into indices (Sennrich et al., 2016).
The output prediction layer is tied to the input word embedding layer as advised in prior work (Press & Wolf, 2017; Inan et al., 2016).
Each model is trained for 64 epochs on the training set; the best one is selected based on perplexity on the validation set, and then evaluated on the test set.
Table 4 shows the best performance of each model after the hyperparameter search.
D P OLYSEMY
In the dev set of MNLI, the word bank occurs in the following sentences:
according to this plan , areas that were predominantly arab the gaza strip , the central part of the country , the northwest corner , and the west bank were to remain under arab control as palestine , while the southern negev desert and the northern coastal strip would form the new state of israel . 5 https://gluebenchmark.com/leaderboard 40 Epochs 0. T r a i n i n g
L o ss 4 M416 M 1664 M (a) Plain Architecture    40 Epochs 0. T r a i n i n g L o ss 4 M416 M 1664 M (b) BiDAF Architecture    40 Epochs T r a i n i n g
{+A cc u r a c y 4 M416 M 1664 M (c) Plain Architecture    40 Epochs
+}
T r a i n i n g
A cc u r a c y 4 M416 M 1664 M (d) BiDAF Architecture
well uh normally i like to to go out ﬁshing in a boat and uh rather than like bank ﬁshing and just like you try and catch anything that ’s swimming because i ’ve had such problems of trying to catch any type of ﬁsh that uh i just really enjoy doing the boat type ﬁshing .
the other bank pays the fund interest based upon tiered account levels , more typical of a large commercial account .
The above sentences are three examples, there are other sentences in which the word bank occurs.
As we show in the main paper, after learning, our TPRU categories the ﬁrst two cases, in which one refers the West Bank of the Jordan river, and the other refers to the bank in the river context, as the same sense, as there is no clear learning signal that speciﬁes the ﬁrst one as a named entity, and categorises the third sentence into another sense.
E E FFECT OF I NCREASING THE D IMENSIONALITY OF F ILLERS /S YMBOLS
Both ﬁllers and roles in TPRs (Smolensky, 1990) are tensors with arbitrary dimensions, which empowers the representation ability of TPRs.
In our case, as we aim to produce a distributed vector representation of the given input sequence rather than a tensor representation, the ﬁllers are reduced to ﬁller numbers, in which case, the symbol represented by each ﬁller is limited to one-dimensional vector.
In our proposed TPRU, the unbinding operation is instantiated by the dot-product (inner-product) operation between the binding complex and the unbinding vectors.
[-T r a i n i n g A cc u r a c y 4 M416 M 1664 M (c) Plain Architecture    40 Epochs
 T r a i n i n g A cc u r a c y 4 M416 M 1664 M (d) BiDAF Architecture-]One plausible way to upgrade ﬁller numbers to ﬁller vectors is to split the binding complex into multiple slices, and split each unbinding vector accordingly: then the dot-product is carried out on each pair of slices rather than the entire vector.
In this case, the dimension of the resulting ﬁller vectors is determined by the number of slices predeﬁned prior to learning, and is limited by the dimension of the binding complex.
The normalisation step is operated on the same dimension on multiple ﬁller vectors independently and in parallel, which results in multiple normalised probability distributions.
Each binding role vector is split into the same number of slices, and each distribution is applied to bind the same slices from all binding role vectors into a single slice.
All slices are concatenated together to form a single vector which has the same dimensionality as the input binding complex.
A similar idea was proposed as the multi-head attention mechanism in prior work and has been shown to be effective in terms of performance gain without additional computational cost (Vaswani et al., 2017).
It was widely adopted in prior work on Transformer networks for various tasks.
In our implementation of the Logical Entailment task, the binding complex is split into eight slices, each with the same dimension.
Figure 2 shows training logs with eight slices and also with only one slice for three different numbers of role vectors in our proposed TPRU.
In general, shifting from ﬁller numbers to ﬁller vectors in our TPRU provides faster convergence and better performance on the training set.
However, as the number of role vectors increases, the improvement gained decreases, which leads to an interesting hypothesis that the effect of increasing number of role vectors might be correlated to that of increasing the dimensionality of ﬁller vectors.