Learning Distributed Representations of Symbolic Structure Using Binding and Unbinding Operations
Abstract
Widely used recurrent units, including long short term memory (LSTM) and gatedrecurrent units (GRU), perform well on natural language tasks, but their ability tolearn structured representations is still questionable.
Exploiting Tensor Product
Representations (TPRs) — distributed representations of symbolic structure inwhich vector-embedded symbols are bound to vector-embedded structural positions — we propose the TPRU, a recurrent unit that, at each time step, explicitlyexecutes structural-role binding and unbinding operations to incorporate structuralinformation into learning.
Experiments are conducted on both the Logical Entailment task and the Multi-genre Natural Language Inference (MNLI) task, and
our
TPR-derived recurrent unit provides strong performance with signiﬁcantly fewerparameters than LSTM and GRU baselines.
Furthermore, our learnt TPRU trainedon MNLI demonstrates solid generalisation ability on downstream tasks.
Introduction
Recent advances in deep learning beneﬁt largely from neural networks’ ability to learn distributedrepresentations of inputs from various domains; even samples from different modalities can be easilycompared in a common representation space.
In contrast to localist (1-hot) representations that arenot able to directly represent the possible componential structure within data [18], distributed representations are potentially capable of inducing implicit structure in the data or explicit structure that isnot presented along with the data.
When it comes to statistical inference, distributed representationsshow considerable power, attesting to strong ability to encode world knowledge and to efﬁciently userepresentation space.
However, the interpretability of learnt distributed representations is limited; it istypically quite unclear what speciﬁcally has been encoded in the representations.
Symbolic computing can take advantage of the presented structure of data and denote each substructure as a symbol; throughout computation, the representations derived by symbolic computingmaintain the structure of data explicitly, and each substructure can be retrieved by simple straightforward computation [8].
In terms of inducing implicit structure from data, symbolic computing aims todecompose each sample to an ensemble of unique symbols that carry potential substructure of thedata.
With enough symbols, the underlying structure of data can be encoded thoroughly.
The explicitusage of symbols in symbolic computing systems improves the capability of induced representations,but also, it brings in issues including inefﬁcient memory usage and computational expense.
Tensor product representation (TPR) [36] is an instantiation of general neural-symbolic computingin which symbol structures are given a ﬁller-role decomposition : the structure is captured by a setof N roles r i (e.g., left-child-of-root), each of which is bound to a ﬁller f i (e.g., a symbol).
A TPRembedding of a symbol structure S derives from vector embeddings of the roles { r i } and their ﬁllers{ f i } via the outer or tensor product: S =
(cid:80) Ni =1 r i ⊗ f i
=
(cid:80) Ni =1 r
i f (cid:62) i = RF (cid:62) , where R and
F respectively denote matrices having the role or ﬁller vectors as columns.
Each r i ⊗ f
i is the embedding of a role-ﬁller binding; ⊗ is the binding operation.
The unbinding operation returns theﬁller of a particular role in S ; it is performed by the inner product: f i = u (cid:62) i S , where u i is the dualof r i , satisfying u (cid:62) i r j = δ ij .
Letting U be the matrix with columns { u i } , we have U (cid:62) R =
I .
Let { r i } ⊂ R d × 1 .
For present purposes it turns out that it sufﬁces to consider ﬁller vectors { f i } ⊂ R 1 × 1 , in which case S ∈ R d × 1 ; we henceforth denote S as b , a binding complex .
Let f ∈ R N × 1 = F (cid:62) be the column vector comprised of the N { f i } ∈ R 1 × 1 .
Now the binding and unbindingoperations, simultaneously over all roles, become
[Equation]
and
[Equation]
With binding and unbinding operations and sufﬁcient role vectors, the binding complex b is able torepresent data from a wide range of domains, with the structure of the data preserved.
We aim to incorporate symbolic computing into learning distributed representations of data whenexplicit structure is not presented to neural networks.
Speciﬁcally, we propose a recurrent unit thatexecutes binding and unbinding operations according to Eq. 1.
The proposed recurrent unit leveragesboth the advantages of distributed representations and the ability to explore and maintain learntstructure from symbolic computing.
Our contribution is threefold:
• We propose a recurrent unit, named TPRU, which integrates symbolic computing with learning distributed representations, and has signiﬁcantly fewer parameters than widely used Long-short Term Memory (LSTM) [19] and Gated Recurrent Unit (GRU) [11].
• We present experimental results with the TPRU on both the Logical Entailment task [16] and
the Multi-genre Natural Language Inference (MNLI) dataset [41], both of which arguably require
high-quality structured representations for making good predictions.
The proposed unit providesstrong performance on both tasks.
• The TPRU trained on MNLI with plain (attentionless) architecture demonstrates solid generalisationability on downstream natural language tasks.
Related Work
Recent efforts on learning structured distributed representations can be roughly categorised into twotypes: enforcing a strong global geometrical constraint on representation space, such as hyperbolicembedding [25, 43], and introducing inductive biases into the architecture of networks, includingthe Relational Memory Core [34] and neural-symbolic computing methods [8].
The latter categorydivides into models that insert neural networks into discrete structures [7, 32, 37] and those thatinsert discrete structures into neural network representations [17].
Our work falls in this last category:learning structured representations by incorporating the inductive biases inherent in TPRs.
Some prior work has incorporated TPRs into RNN representations.
Question-answering on SQuAD[33] was addressed [26] with a gated recurrent unit in which the hidden state was a single TPR binding;the present work deploys complexes containing multiple bindings.
TPR-style unbinding was appliedin caption generation [21], but the representations were deep-learned and not explicitly designed to be
TPRs as in the present work.
A contracted version of TPRs, Holographic Reduced Representations,was utilised to decompose input space and output space with ﬁller-role decomposition [6].
However,our work differs from prior work in that TPR ﬁller-role binding and unbinding operations areexplicitly carried out in our proposed recurrent unit, and also, the two operations directly determinethe calculation of the update on the hidden states.
The logical entailment task we study is based on [16], in which a model was proposed that usedgiven parse trees of the input propositions and was designed speciﬁcally for the task.
Our modeldoes not receive parsed input and must learn simultaneously to identify, encode, and use the structurenecessary to compute logical entailment.
NLI (or Recognizing Textual Entailment) has assumed acentral role in NLP since [14].
Neural models have persistently made errors explicable as failureto encode propositional structure [9] and our work targets that particular capability.
Our proposedrecurrent unit contains essential operations in symbolic computing systems, which encourages thelearnt distributed representations to encode more structure-related information.
We can thus expectthat our TPRU will give strong performance on both tasks even with signiﬁcantly reduced parameters.
Proposed Recurrent Unit: The TPRU
As shown in both LSTM [19] and GRU [11] design, a gating mechanism helps the hidden state at thecurrent time step to directly copy information from the previous time step, and alleviate vanishingand exploding gradient issues.
Our proposed recurrent unit keeps the gating mechanism and adoptsthe design of the input gate in GRU.
At each time step, the TPRU receives two input vectors, one of which is the binding complex fromthe previous time step b t − 1 ∈ R d × 1 and the other is the vector representation of the external input tothe network at the current time step x t ∈ R d (cid:48)
× 1 .
The TPRU produces a binding complex b t ∈ R d × 1 .
An input gate g t is computed to calculate a weighted sum of the information produced at current timestep (cid:101) b t and the binding complex from the previous time step b t − 1 ,
[Equation]
where σ ( · ) is the logistic sigmoid function, ◦ is the Hadamard (element-wise) product, and W h ∈
R d × d and W x ∈ R d × d (cid:48) are matrices of learnable parameters.
As we now explain, the calculation of(cid:101) b t is carried out by the unbinding and binding operations of TPR (Eq. 1).
3.1 Unbinding Operation
Consider a set of hypothesised unbinding vectors U
= [ u 1 , u 2 , ..., u N ] ∈ R d × N .
At timestep t ,these can be used to unbind ﬁllers f b,t from the previous binding complex b t − 1 using Eq.
We posita matrix W ∈ R d × d (cid:48) that transforms the current input x t ∈ R d (cid:48) into the binding space R d where ittoo can be unbound, yielding ﬁllers f x
[Equation]
[Equation]
A strong sparsity constraint is enforced by applying a rectiﬁed linear unit (ReLU) to both f b,t andf x,t [42], and their interaction is calculated by taking the square of the sum of the two sparse vectors.
The resulting vector (cid:101) f t is then normalised to form a distribution f t .
[Equation]
where b h and b x are two scalar parameters for stable learning.
3.2 Binding Operation
Given a hypothesised set of binding role vectors R =
[ r 1 , r 2 , ..., r N ] ∈ R d × N , we apply the bindingoperation in Eq.
1 to the ﬁllers f t at time t to get the candidate update for the binding complex (cid:101) b t ,(cid:101) b t =
[Equation]
t .
The gating mechanism controls the weighted sum of the candidate vector (cid:101) b t and the previous bindingcomplex b t − 1 to produce a binding complex b t at current time step, as given by Eqs.
2 and 3.
3.3 Unbinding and Binding Role Vectors
In the TPRU, there is a matrix of role vectors R used for the binding operation and a matrix ofunbinding vectors U used for the unbinding operation.
To control the number of parameters inour proposed unit, instead of directly learning the role and unbinding vectors, a ﬁxed set of vectors
V =
[ v 1 , v 2 , ..., v N ] ∈ R d × N is sampled from a standard normal distribution, and two lineartransformations W u ,
W r ∈ R
d × d are learnt to transform V to U =
W u V and R =
W r V .
Therefore, in total, our proposed recurrent unit has ﬁve learnable matrices, including W u , W r ,
W h ∈ R d × d and W x , W ∈ R d × d (cid:48) .
Compared to the six parameter matrices of the GRU and theeight of the LSTM, the total number of parameters in the TPRU is signiﬁcantly fewer.
Tasks
Two entailment tasks, including an abstract logical entailment task and a relatively realistic naturallanguage entailment task, along with other downstream natural language tasks, are considered todemonstrate that the TPRU is capable of inducing structured representation through learning.
Each ofthe two entailment tasks provides pairs of samples, and for each pair, the model needs to tell whetherthe ﬁrst (the premise) entails the second (the hypothesis).
As our goal is to learn structured vector representations, the proposed TPRU serves as an encodingfunction, which learns to process a proposition or sentence one token at a time, and then produces avector representation.
During learning, two vector representations are produced by the same recurrentunit given a pair of samples, then a simple feature engineering method (e.g. concatenation of thetwo representations) is applied to form an input vector to a subsequent classiﬁer which makes a ﬁnalprediction.
In general, with a simple classiﬁer, e.g. a linear classiﬁer or a multi-layer perceptronwith a single hidden layer, the learning process forces the encoding function to produce high qualityrepresentations of samples, either propositions or sentences, and better vector representations lead tostronger performance.
4.1 Logical Entailment
In propositional logic, for a pair of propositions, A and B , the value of A (cid:15) B is independent ofthe identities of the shared variables between A and B , and is dependent only on the structure ofthe expression and the connectives in each subexpression, because of α -equivalence.
For example,p ∧ q (cid:15) q holds no matter how we replace variable p or q with any other variables or propositions.
Thus, logical entailment naturally is a good testbed for evaluating a model’s ability to carry outabstract, highly structure-sensitive reasoning [16].
Theoretically, it is possible to construct a truth table that contains as rows/worlds all possiblecombinations of values of variables in both propositions A and B ; the value of A (cid:15) B can be checkedby going through every entry in each row.
An example is given in the supplementary material.
As thelogical entailment task emphasises reasoning on connectives, it requires the learnt distributed vectorrepresentations to encode the structure of any given proposition to excel at the task.
The dataset used in our experiments has balanced positive and negative classes, and the task difﬁcultyof the training set is comparable to that of the validation set.
2 Five test sets are generated to evaluatethe generalisation ability at different difﬁculty levels: some test sets have signiﬁcantly more variablesand operators than both the training and validation sets (see Table 1).
4.2 Multi-genre Natural Language Inference
Natural language inference (NLI) tasks require inferring word meaning in context as well as thehierarchical relations among constituents in a given sentence (either premise or hypothesis), and thenreasoning whether the premise sentence entails the hypothesis sentence or not.
Compared to logicalentailment, the inference and reasoning in NLI also rely on the identities of words in sentences inaddition to their structure.
More importantly, the ambiguity and polysemicity of language lead to theimpossibility of creating a truth table that lists all cases.
Therefore, NLI is an intrinsically hard task.
The Multi-genre Natural Language Inference (MNLI) dataset [41] collected sentence pairs in tengenres; only ﬁve genres are available in the training set, while all ten genres are presented inthe development set.
The performance of a model on the mismatched genres, which exist in thedevelopment but not the training set, tells us how well the structure encoded in distributed vectorrepresentations of sentences learnt from seen genres in training generalises to sentence pairs in unseengenres.
As the nature of NLI tasks requires inferring both word meaning and structure of constituentsin a given sentence, supervised training signals from labelled datasets enforce an encoding function toanalyse meaning and structure at the same time during learning, which eventually forces distributedvector representations of sentences produced from the learnt encoding function to be structured.
Thus,a suitable inductive bias that enhances the ability to learn structures of sentences will enhance successon the MNLI task.
[Table]
4.3 Downstream Natural Language Tasks
Vector representations of sentences learnt from labelled NLI tasks demonstrate strong transferabilityand generalisation ability, which indicates that the learnt encoding function can be applied as ageneral-purpose sentence encoder to other downstream natural language tasks [13].
As our proposed
TPRU is also able to map any given sentence into a distributed vector representation, it is reasonableto evaluate the learnt vector representations on other natural language tasks, and the performance ofour proposed recurrent unit will tell us the generalisation ability of the learnt representations.
SentEval [12] presents a collection of natural language tasks in various domains, including sentimentanalysis (MR [28], SST [38], CR [20], SUBJ [27], MPQA [40]), paraphrase detection (MRPC [15]),semantic relatedness (SICK [24]), question-type classiﬁcation (TREC [23]) and semantic textualsimilarity (STS [1, 2, 3, 4, 5]).
Except for STS tasks, in which the cosine similarity of a pair ofsentence representations is compared with a human-annotated similarity score, each of the tasksrequires learning a linear classiﬁer on top of produced sentence representations to make predictions.
Training Details
Experiments are conducted in PyTorch [30] with the Adam optimiser [22] and gradient clipping [29].
Reported results are averaged from the results of three different random initialisations.
5.1 Plain Architecture
For the Logical Entailment task, we train our proposed TPRU as well as LSTM [19] and GRU [11] RNNs for 90 epochs.
Only the output at the last time step is regarded as the representation of a givenproposition, and two proposition representations are concatenated, as done in previous work [16], andfed into a multi-layer perceptron which has only one hidden layer with ReLU activation function.
Theinitial learning rate is 0 .
001 and divided by 10 every 30 epochs.
The best model is picked based onthe performance on the validation set, and then evaluated on all ﬁve test sets with different difﬁcultylevels.
Symbolic Vocabulary Permutation [16] is applied as data augmentation during learning which
[Table]
systematically replaces variables with other randomly sampled variables according to α -equivalenceas only connectives matter on this task.
Detailed results are presented Table 1.
For the MNLI task, our proposed TPRU as well as LSTM and GRU units are trained for 10 epochs.
Aglobal max-pooling over time is applied on top of binding complexes produced by each recurrent unitat all time steps to generate the vector representation for a given sentence.
Given a pair of generatedsentence representations u and v , a vector is constructed to represent the difference between twovectors [ u ; v ; | u − v | ; u ◦ v ] , where u ◦ v is the Hadamard (element-wise) product and | u − v | isthe absolute difference, and the vector is fed into a multi-layer perceptron with the same settings asgiven above.
The feature engineering and the choice of classiﬁer are suggested by prior work [35, 39].
The Stanford Natural Language Inference (SNLI) dataset [9] is added as additional training data asrecommended [39], and ELMo [31] is applied for producing vector representations of words.
Theinitial learning rate is 0.0001, and kept constant during learning.
The best model is chosen accordingto the averaged classiﬁcation accuracy on matched (ﬁve genres that exist in both training and dev set)and mismatched (ﬁve genres in dev set only) set.
LSTM and GRU models use the same settings.
Theperformance of each model is presented in Table 2.
For downstream natural language tasks, the parameters in the learnt recurrent unit — our proposed TPRU, LSTM or GRU — are ﬁxed and used to extract vector representations of sentences for eachtask.
Linear logistic regression or softmax regression is applied when additional learning is requiredto make predictions.
Details of hyperparameter settings of the classiﬁers can be found in the SentEvalpackage.
3 Table 3 presents macro-averaged results, which are Binary (MR, CR, SUBJ, MPQA, and
SST), STS (Su., including SICK-R and SICK-Benchmark), STS (Un., including STS12-16), TREC,
SICK-E, and MRPC.
5.2 BiDAF Architecture
Bi-directional Attention Flow ( BiDAF ) [35] has been adopted in various natural language tasks,including machine comprehension [35] and question answering [10], and provides strong performanceon NLI tasks [39].
The BiDAF architecture can generally be applied to any task that requiresmodelling relations between pairs of sequences.
As both the Logical Entailment MNLI
[Table]
The BiDAF architecture contains a layer for encoding two input sequences, and another for encodingthe concatenation of the output from the ﬁrst layer and the context vectors determined by the bidirectional attention mechanism.
In our experiments, the dimensions of both layers are set to bethe same, and same type of recurrent unit is applied across both layers.
The same settings areused for experiments on LSTM, GRU and our TPRU models.
Speciﬁcally, for TPRU, the recurrentunits in both layers have the same number of role vectors.
Other learning details are as in the plainarchitecture.
Tables 1 and 2 respectively present results on the Logical Entailment and the MNLItask, with BiDAF results in parentheses.
Discussion
As presented in Tables 1 and 2, our proposed TPRU provides solid performance.
On the Logical Entailment task, the TPRU provides similar performance with the LSTM and GRU under both theplain and BiDAF architectures, but the TPRU has signiﬁcantly fewer parameters.
When it comes tolarger dimensionality, our TPRU appears to be more stable than LSTM and GRU during learning aswe observed that the LSTM with the plain architecture overﬁtted the training set terribly in all threetrials and the GRU with the BiDAF architecture failed in two out of three trials.
On the MNLI task, our proposed TPRU consistently outperforms both the LSTM and GRU under allfour combinations of different dimensions and architectures.
In Table 3, the TPRU under the plainarchitecture generalises as well as the LSTM and GRU on 16 downstream tasks in SentEval.
6.1 Effect of Increasing the Number of Role Vectors
In TPR [36], the number of role vectors indicates the number of unique symbols that will be usedin the ﬁnal representations of the data.
Since each symbol is capable of representing a speciﬁcsubstructure of the input data, increasing the number of role vectors eventually leads to more highlystructured representations if there is no limit on the dimensionality of role vectors.
Experiments are conducted to show the effect of increasing the number of role vectors on theperformance on both tasks.
As shown in Tables 1 and 2, adding more role vectors into our proposed
TPRU gradually improves the performance on the two entailment tasks.
Interestingly, on the MNLItask, our proposed TPRU with only 16 role vectors achieves similar performance to that of the LSTM
[Figure]
and GRU, which implies that the distributed representations learnt in both the LSTM and GRU arehighly redundant and can be reduced to 16 or even fewer dimensions, which also shows that the LSTMand GRU are not able to extensively exploit the representation space.
Meanwhile, the introducedsymbolic computing executed by binding and unbinding operations in our proposed unit encouragesthe model to take advantage of distinct role vectors to learn useful structured representations.
Figure 1 presents the learning curves, including training loss and accuracy, of our proposed TPRUwith different number of role vectors on two entailment tasks.
As shown in ﬁgures, incorporatingmore role vectors leads to not only better performance, also faster convergence during training.
Theobservation is consistent on both Logical Entailment and MNLI tasks.
Conclusion
We proposed a recurrent unit (TPRU) that executes binding and unbinding operations in Tensor
Product Representations.
The explicit execution in our proposed recurrent unit helps it leverageadvantages of both distributed representations and neural-symbolic computing, which essentiallyallows it to learn structured representations.
Compared to widely used recurrent units, including
LSTM and GRU, our proposed TPRU has many fewer parameters.
Logical Entailment and Multi-genre Natural Language Inference tasks are selected for experimentsas both tasks require highly structured representations to make good predictions.
Plain and BiDAFarchitectures are applied on both tasks.
Our proposed TPRU outperforms its comparison partners,
LSTM and GRU, on MNLI tasks with different dimensions and architectures, and it performs similarlyto others on Logical Entailment task.
Analysis shows that adding more role vectors tends to providestronger results and faster convergence during learning, which matches the purpose of symbols insymbolic computing systems.
We believe that our work pushes the existing research topic on interpreting RNNs into anotherdirection by incorporating symbolic computing.
Future work should focus on the interpretabilty ofour proposed TPRU as the symbolic computing is explicitly conducted by binding and unbindingoperations.
A A Look-up Table Approach to Logical Entailment
[Table] [Table]