T OWARDS D ECOMPOSED L INGUISTIC R EPRESENTATION WITH H OLOGRAPHIC R EDUCED R EPRESENTATION
Anonymous authors
Paper under double-blind review
A BSTRACT
The vast majority of neural models in Natural Language Processing adopt a form of structureless distributed representations.
While these models are powerful at making predictions, the representational form is rather crude and does not provide insights into linguistic structures.
In this paper we introduce novel language models with representations informed by the framework of Holographic Reduced
Representation (HRR).
This allows us to inject structures directly into our wordlevel and chunk-level representations.
Our analyses show that by using HRR as a structured compositional representation, our models are able to discover crude linguistic roles, which roughly resembles a classic division between syntax and semantics.
I NTRODUCTION
Recent advances in representation learning have been unequivocally led by the long strides of progress in deep learning and its distributed representations.
In many tasks of Natural Language
Processing (NLP), researchers have convincingly shown that distributed representations are capable of encoding the complex structure of textual inputs (for example Mikolov et al.
J ´ ozefowicz et al.
; Sutskever et al.
The dominant approach for many NLP tasks is the encoder-decoder paradigm that uses neural networks to learn the transformations from many smaller comprising units to one complex embedding, and vice versa 2 .
The underlying structure, in a rather crude fashion, is assumed to be represented by this complex embedding.
In many cases, such crude way of representing the structure is unsatisfactory, due to a lack of transparency, interpretability and transferability.
Transparency and interpretability require that the operations of encoding and decoding have clear conceptual meaning, and transferability generally necessitates a separation of transferable features (e.g., domain-invariants) from the rest.
On account of shortcomings, much previous work has been devoted to inducing disentangled representations (Chen et al., 2016;
Hsu et al., 2017; Higgins et al., 2016; Janner et al., 2017).
We attempt to address these issues by utilizing a more principled framework to encode complex symbolic structures using distributed representations.
Speciﬁcally, we employ Holographic Reduced
Representation (HRR) to represent and manipulate structures.
As a member of the Vector Symbolic
Architecture (VSA) family (Gayler, 2003; Smolensky, 1990; Plate, 1995; Kanerva, 2009), HRR builds upon the notions of roles and ﬁllers (i.e., values for the roles).
For instance, with semantic roles, the sentence John loves his mom can be represented by three role-ﬁller pairs, namely ( agent ,
John ), ( predicate , loves ), and ( patient , his mom ).
Each role and ﬁller is represented by a high-dimensional vector, and HRR provides a mathematical framework to encode role-ﬁller pairs, compose complex embeddings, and retrieve ﬁllers given corresponding roles.
A disentangled representation, using HRR terminology, is synonymous with decomposing a complex embedding into many role-ﬁller pairs.
In this paper, we investigate the effectiveness of HRR at inducing disentangled representations on the task of language modeling (LM).
We applied HRR to language modeling because it requires be released the public. For attention mechanisms, there is actually a distribution over multiple embeddings, rather than one single complex embedding. minimal supervision, and has been proven hugely beneﬁcial for many other NLP tasks.
The versatility of language modeling demonstrates that some linguistic regularities much be present, and the training signal is sufﬁcient for them to arise.
We carefully design a language model with HRR that explicitly encodes the underlying structure as role-ﬁller pairs on both word-level and chunk-level, and show that HRR provides an inductive bias towards the learning of decomposed representations.
We demonstrate that on both Penn Treebank (PTB) and a subset of One-Billion-Word LM data set (1B), our model can effectively separate certain aspects of word or chunk representation, which roughly corresponds to a division between syntax and semantics.
We perform various analyses on the learned embeddings, and validate that they indeed capture distinct linguistic regularities.
Our paper is structured as follows.
Section 2 gives a background overview of VSA and HRR;
Section 3 details our proposed models; Experimental results are shown in Section 4, followed by related work in Section 5 and a conclusion in Section 6.
B ACKGROUND
Vector Symbolic Architecture (VSA) is a family of models that enable connectionist models to perform symbolic processing, while encoding complex structures in distributed representations.
A set of algebraic operations deﬁned by these approaches allow them to compose, decompose and manipulate symbolic structures.
Our paper focuses on such an approach, namely Holographic Reduced Representation (HRR) proposed by (Plate, 1995).
HRR uses three operations: circular convolution, circular correlation and element-wise addition, to perform encoding, decoding and composition, respectively.
The cicircular convolution (denoted by the operator ∗(cid:13) ) of two vectors x and y of dimension d , is deﬁned as z = x ∗(cid:13) y , in which
[Equation]
x ∗(cid:13) y is called the binding of x and y , or the encoding of the pair ( x , y ) .
Using this operation, the composition of a set of role-ﬁller pairs ( r 1 , f 1 ) , ( r 2 , f 2 ) , . .
, ( r m , f m ) is represented as H = r 1 ∗(cid:13) f 1 + r 2 ∗(cid:13) f 2 + . . .
r m ∗(cid:13) f m , where + is element-wise addition used as a composition operator.
The previous example John loves his mom can be represented as
[Equation]
By deﬁnition, HRR guarantees that the composed representation remains a vector of dimension d , regardless of how many items are bound together by the ∗(cid:13) operation.
This avoids the parameter explosion problem of other VSA approaches such as as Tensor Product Represenation (TPR) (Smolensky, 1990), and makes HRR a more practical choice for representing compositional structures.
To decode from an HRR and retrieve a role or ﬁller representation, an approximate inverse operation of the circular convolution, named circular correlation, is deﬁned as t = x # (cid:13) y in which
[Equation]
Now given a memory trace z = x ∗(cid:13) y , the correlation operation allows us to retrieve y from the cue x via y ≈ x # (cid:13) z .
We do not detail the exact conditions when this retrieval holds, but refer interested readers to the original paper (Plate (1995))
H OLOGRAPHIC R EDUCED R EPRESENTATION FOR L ANGUAGE M ODELING
We incorporate HRR into language models on two levels: word level and chunk level.
Our HRRenabled language models (HRRLM) posit an explicit decomposition of word or chunk representations, which enables our model to capture different aspects of linguistic regularities.
Before delving into the details of our model, we ﬁrst introduce notations and provide a brief account of the commonly used RNN-based LM.
3. RNNLM
RNN-based LMs estimate the probability of any given sentence s = w 1 w 2 . . .
w n using an RNN.
At each step t , RNN encodes the history w 1 w 2 . . .
w t into a vector h , and tries to predict the next token w t +1 .
Prediction is generally modeled by a linear layer followed by softmax operation.
Speciﬁcally, Pr
[Equation]
w t | [Equation] [Equation] where E ( · ) is the embedding operation to embed a symbol into a continuous space R d , and V is the entire or a sampled subset of the vocabulary.
The scoring function is deﬁned by dot product, and therefore maximizing this probability encourages related words to form clusters, and away from other words in the embedded space.
3. W ORD - LEVEL HRRLM
Encoding We ﬁrst use HRR to directly encode the underlying structures of words.
We assume there is a decomposition of representations along N directions.
Speciﬁcally, we embed a word w as
[Equation]
where r wordi ’s are basis role embeddings, shared by all words.
Each basis role embedding is bound to its distinct set of ﬁller embeddings, modeled by ˜ E i .
The motivation is that when properly trained, different bindings should capture disparate aspects of word representation.
For instance, the ﬁrst binding might be relevant for syntactic categories, and the second one for semantic relatedness.
With this particular decomposition, the word getting should be close to other gerunds such as giving and forgetting in the ﬁrst embedding space, and get , got or received in the second.
In this case, the composite (i.e., the sum) of these bindings essentially encodes getting = { semantics : GET , syntax : GERUND } .
We additionally assume that each set of ﬁller embeddings ˜ E i resides in a separate linear subspace.
This is achieved by modeling each ˜ E i ( w ) with a linear combination of its associated basis ﬁller embeddings f i,j .
Speciﬁcally,
[Equation]
where s wi ∈ R d (cid:48) is a word-speciﬁc d (cid:48) -dimensional vector.
In other words, F i projects s wi ∈ R d (cid:48) to ˜ E
i ( w ) ∈ R d .
This assumption has two advantages.
First, the total number of parameters for the embedding layer is now V Nd (cid:48) .
We can set a smaller value d (cid:48) to prevent overparameterization, while maintaining a d -dimensional vector as the input to RNN.
Second, by having separate bases for different role-ﬁller bindings, we introduce an inductive bias for the model to learn a decomposition of word representation.
This requirement makes sense intuitively – a decomposition of representation usually necessitates a separation of feature space.
Our preliminary experiments show that this separation is essential for obtaining decomposed representations.
The entire encoding operation is illustrated in the bottom half of Figure 1(b).
Decoding The composite embedding E ( w ) is fed as input to RNN.
From its output h given by the top hidden layer, we decode all the ﬁller vectors using circular correlation, and factorize the scoring function into N parts.
Speciﬁcally, we use the same word level loss as in Equation (2), but replace the scoring function in Equation (3) as a sum of dot products in every ﬁller space:
[Equation]
where α i ’s are scalar hyperparameters that we use to break the symmetry of the scoring function.
Speciﬁcally when N = 2 , α 1 is set to a constant 1 . 0 , and α 2 is linearly annealed from 0.0 at the start of training to 1.0 at a speciﬁed time step T , then remains constant afterwards.
Note that dot products are only computed between co-indexed ﬁller embeddings, namely between f i and ˜ E i ( w ) .
This ensures the model only learns relatedness in the i -th subspace, without interference from other subspaces.
The entire word-level HRRLM is illustrated is Figure 1(b).
Regularization on basis embeddings Basis embeddings are chosen so that they are not correlated with each other.
This decorrelation is promoted by adding an isometric regularization term to the basis embeddings. For instance, we deﬁne an isometric regularization penalty for the basis ﬁllers
F i ’s as: [Equation] where I is an identity matrix, and β is a hyperparameter.
Similarly, we add a regularization term for r wordi ’s. As an alternative, we also consider using ﬁxed random vectors for basis embeddings since high-dimensional random vectors are approximately orthogonal to each other.
3. C HUNK - LEVEL HRRLM
A direct extension to the chunk-level model using the proposed techniques above is not desirable, due to two major difﬁculties.
First, Equation (4) stipulates that each unique word type is assigned a vectorial parameter, which is computationally infeasible due to the vast number of unique chunks.
Second, the same chunk can carry different semantic roles.
For instance, the two sentences His mom loves John and John loves his mom have the same noun phrases, but with their roles of agent and patient switched.
In light of these difﬁculties, we take on a compositional approach to address the ﬁrst issue, and use a context-sensitive role to address the second.
Encoding Unlike the word-level HRR representation in Equation (4) where roles are ﬁxed, the chunk roles are represented by linear combinations of M basis role embeddings.
This formulation allows us to model context-dependency.
Speciﬁcally, we construct HRR representations for a chunk c = w 1 w 2 . . . w m in three steps:
[Equation]
We ﬁrst predict a context-sensitive role tuple for each word w k in the chunk:
[Equation]
a w k i ’s represent the distribution of chunk roles, which are predicted by the same RNN used to predict the next tokens.
Speciﬁcally, we feed the output vector from RNN through a linear layer and then a softmax layer. r chunki ’s are chunk-level basis role embeddings, shared by all chunks 3 . Stepcorresponds to the role prediction module in Figure 1(c).
S TEP 2 ( PROJECTION AND BINDING )
For each of the basis roles, we predict their associated ﬁller embeddings.
This is done by projecting the HRR word representation ˜ E ( w k ) (Equation (4)) into M vectors.
These ﬁllers are then bound with their corresponding roles and summed together.
Speciﬁcally,
[Equation]
Intuitively, the ﬁrst two steps aims to evaluate the representational contribution of w k to the entire chunk c .
Following the practice in constructing a word-level model (Equation (4)), we also require the separation of different role-ﬁller bindings.
We also note that the binding ˆ E c ( w k ) embeds the word w k into a space that is speciﬁc to the chunk c .
The ﬁrst two steps are illustrated in Figure 1(c).
S TEP 3 ( ENCODING )
After obtaining bindings for all words within a chunk, the chunk embedding is deﬁned as ˆ
[Equation] [Equation]
d)).
It can be easily veriﬁed that
[Equation]
Note that ˆ E ( c ) has the same form as Equation (4), and ˆ E i ( c ) can be interpreted as the chunk ﬁller embedding for the i -th chunk role.
However, chunk embeddings are different in two key aspects.
First, the ﬁller embeddings for chunks are projected from the word embeddings, instead of being a set of independently trainable parameters.
This compositional approach addresses the ﬁrst issue regarding the vast number of unique chunks.
Second, chunk embeddings rely on weights a wi ’s from role prediction, which provide a natural vehicle for carrying contextual information, therefore addressing the second issue of context-sensitivity.
Chunk prediction Comparable to language modeling that uses next word prediction as supervision, our chunk-level HRR model uses next chunk prediction (CP in Figure 1(a)).
Speciﬁcally in our experiments, we simply concatenate the chunk embeddings from the last two steps, and feed it through a linear layer followed by tanh activation: [Equation]
Decoding Similar to word-level HRR (Equation (5)), we decode the ﬁller embeddings from the predicted chunk embedding using r chunki as cue, and then use the decoded embeddings to factorize the scoring function. The same form of loss function is used.
To provide negative examples in the denominator of the softmax Equation (2), we use all the chunks in the mini-batch.
These chunks form a pseudo ”chunk vocabulary” that is constructed on the ﬂy.
Role annealing, and regularization on basis embeddings are also applied.
Chunk boundaries Chunk boundaries have to be supplied in order to construct a chunk embedding.
We reply on a third-party chunker to provide such annotations.
This is analogous to our word-level model, only that word boundaries are trivially provided by whitespace in English.
E XPERIMENTS 4. S ETUP
Data Sets We train all models on both Penn Treebank (PTB) (Marcus et al., 1994) and OneBillion-Word Benchmark dataset (1B) (Chelba et al., 2014). For 1B, we experimented with three training sizes: the full dataset, one-tenth of the dataset, and one-hundredth.
We evaluate all models on the following three aspects:
• Perplexity: We report perplexity to evaluate language models.
• Intrinsic evaluation: We use a test suite consisting of 18 word embedding benchmark datasets to intrinsically evaluate the quality of word embeddings (Jastrzebski et al., 2017).• Extrinsic evaluation: We evaluate all word embeddings on six downstream tasks following Nayak et al.
(2016): Part-of-speech tagging (POS), chunking, named entity recognition (NER), sentiment analysis (SA), question classiﬁcation (QC), and natural language Inference (NLI).
Training details For our models, We mainly experimented with two word-level and two chunklevel roles.
The basis embeddings are either trained with isometric constraint, or ﬁxed as constant after random initialization.
We also note that for PTB, we experimented with either contiguous input or noncontiguous input.
For contiguous input, we follow the common practice in the literature to feed the input sentences in the order that they appear in the document, and initialize the hidden state of LSTM with the last state from the last batch. We refer interested readers to the appendix for more training details.
4. P ERPLEXITY RESULTS
Although not a prime motivation of our approach, Table 1 shows that our HRR models can outperform the baseline in terms of perplexity, especially when we do not have enough training data. On
PTB, the best HRR model obtains a gain of 2.2 with noncontiguous input (Fixed-big 88.0 vs Baseline 90.2), and 1.9 with contiguous input (Fixed-big-cont 74.7 vs Baseline-cont 76.6).
We also note that increasing the number of basis ﬁllers has a positive impact on HRR models (Fixed-big 88.0 vs Fixed-medium 90.2 vs Fixed-small 97.0).
On 1B, we observe that HRR models beneﬁt from having trainable bases with isometric regularization (107.2 vs 112.3 on 1/100 1B, and 55.9 vs 57.9 on full 1B).
4. W ORD LEVEL ANALYSIS
We then demonstrate that our word-level HRRLM can effectively separate certain attributes of word representation.
Speciﬁcally, we look at our word-level models with two word-level roles, and ﬁnd that the ﬁrst set of ﬁller embeddings captures mostly syntax-related categories, especially tense and agreement for verbs, whereas the second set focuses more on the semantic content of words.
This decomposition is illustrated in Figure 2, where we visualize both sets of ﬁller embeddings for the [Table] most frequent 2500 words in PTB via t-SNE (Maaten & Hinton, 2008).
In the ﬁrst set (in blue), verbs with different inﬂectional markers form distinct clusters.
For instance, bare forms, gerunds, preterites, and third-person-singular verbs each form a visually distinguishable group.
In contrast, for the second set of ﬁller embeddings (in orange), semantically related words tend to be close regardless of their morphosyntactic markers.
For instance, give , gave , and giving are close in the second space.
We proceed to evaluate the quality of the learned word embeddings both intrinsically and extrinsically.
As Table 2 summarizes, our HRR models consistently outperform the baseline on the intrinsic evaluation task (numbers on the left ). The best HRR model Isometric-big obtains a gain of 9 . 0% , 11 . 6% , and 16 . 2% on 1/100 1B, 1/10 1B and full 1B, respectively.
In addition, the HRR model with trainable bases has a noticeable advantage over the model with ﬁxed bases ( 2 . 5% on average).
On the other hand, the HRR models have only a marginal advantage over the baseline (numbers on the right ), which echos ﬁndings in (Schnabel et al., 2015) which claims that intrinsic improvement for word embeddings does not necessarily correlate with improvement in downstream tasks.
Detailed results for each dataset and downstream task are available in Table 7 in Appendix B. [Table] [Table] A breakdown of the intrinsic evaluation reveals that we obtain huge gains on the word analogy task, especially in the verb-related categories such as plural verbs and past tense.
On the SemanticSyntactic analogy dataset Mikolov et al.
, the accuracy (top 1) for plural verbs is 0.897 if we consider the most frequent 200K words.
If we limit the vocabulary to the more common words (10K), the accuracy for present participles reaches 0.964 .
This is consistent with our previous observation that the ﬁrst set of ﬁller embeddings effectively captures the representation of different verb forms.
To further quantify how decomposed are our word-level ﬁller embeddings, we use them to classify whether two words are related in terms of either semantics or syntax.
Speciﬁcally, we use a threshold θ for cosine similarity to determine whether any pair of words ( w, w (cid:48) ) are related.
We then use different values of θ to plot ROC curve and compute its AUC.
For the ground truths, we use the syntactic categories from the Semantic-Syntactic dataset, and extract them from each a : b = c : d example.
Two sets of categorizations are obtained, driven by semantics and syntax respectively.
For instance, for the example make : making = give : giving , we obtain ( { making , giving } , { make , give } ) for syntactic considerations, and ( { making , make } , { giving , give } ) for semantics.
For the verb-related categories, Table 4 shows that the ﬁrst set of ﬁller embeddings does noticeably better than the second in the ﬁrst experiment, while the second set is much better at the second experiment.
Figure 3 shows two such categories which conﬁrm that the decomposition does make sense on a crude syntax-semantics level.
More details of the intrinsic evaluation results can be found in Table 8 in
Appendix B.
[Table]
[Table]
[Table] 4. C HUNK LEVEL ANALYSIS
We train a chunk-level HRR model by initializing it with a pretrained word-level model on PTB. 6 We evaluate the quality of chunk embeddings in two ways. First, we perform a human analysis focused on the phrase the company , which is the most frequent noun phrase in PTB.
We randomly select 100 occurrences
[Table]
and cluster their chunk embeddings using K -means into 5 categories based on the chunk-level ﬁller embeddings.
For each sentence cluster we manually identify the dominating role which the company played in that cluster of sentences.
Table 5 shows that there is a clear syntactic role performed by the phrase in each of the clusters.
We also performed a similar analysis using the second ﬁller embedding.
In this experiment we observed sentences in each cluster contains phrases with semantics related to the company , for example stock , market etc.
From this analysis we see that the ﬁrst role in chunk-level HRR is more sensitive to the different syntactic roles the company plays dependent on the context, while the second role is more associated with its related semantic concepts.
Secondly, to further investigate if our model consistently captures chunk-level roles, we run the model on the Wall Street Journal part of the OntoNotes dataset (Weischedel et al., 2013) in which semantic role labels for each sentence are provided, and compare chunk embedding clusters against the role labels.
Originally there are in all 11 types of role labels, and we merge them into 4 coarsergrained labels corresponding to AGENT, PATIENT, PREDICATE and MODIFIER.
We also limit the length of chunks to be analyzed to 6 words.
Let R i , i = 1 , . . , 4 be the set of chunks labeled with the i th role. For each chunk in R i , we ﬁnd its k nearest neighbors ( k = 1 , 5 , 10 , 25 ) according to their ﬁller embeddings, and report the percentage of the cases where the k nearest neighbors are all included in R i . This hit-rate metric gives us a sense of how accurately chunk clusters capture meaningful semantic roles.
To further quantify this property, we also compute the purity score for each embedding cluster.
The metric scores are reported in Table 6, from which we can see that chunk embedding indeed correctly captures semantic roles more than half of the time.
Perhaps most related to our work are recent attempts to integrate tensor product structure with neural networks (Palangi et al., 2017; Huang et al., 2018).
While these works and ours share a common goal of incorporating neural models with symbolic structures, there are several difference.
First of all, we make use of HRR instead of tensor product as basis for our representation to enable long sequence encoding without parameter explosion.
Moreover, we aim to induce linguistic structures from a task that requires as little supervision as possible like language modeling, whereas their work is focused on question answering which provides stronger guidance signal from labeled data.
On the other hand, recent work (Shen et al., 2018) also proposes a novel network architecture that is capable of learning syntactic roles and semantics jointly, but it is not based on structured representation like
HRR.
There have been many attempts of using symbolic architectures like HRR and TPR for linguistic analysis, see for example (Jones & Mewhort, 2007; De Vine & Bruza, 2010; Recchia et al., 2015;
Prince & Smolensky, 1997; Clark et al., 2008; Clark & Pulman, 2010; Grefenstette et al., 2011).
HRR itself as a variable binding and association mechanism, has also been integrated with neural networks with different motivations like associative memory modeling (Danihelka et al., 2016), relationship reasoning (Weiss et al., 2016) etc.
While most of the work mainly focuses on symbolic and formal analysis via algebraic operations and logic derivations, our work aims to enable neural language models to learn linguistic roles by taking advantage of the HRR properties.
C ONCLUSION
In this paper, we employ HRR to provide a principled decomposition of representation.
We design our HRR language models to work on both word-level and chunk-level.
Our analysis revealed that by introducing an inductive bias, our models can learn disentangled representations, which roughly corresponds to syntax and semantics. A PPENDIX A T RAINING DETAILS
We use the same architecture for all models on both LM data sets, with a two-layered LSTM (Hochreiter & Schmidhuber (1997)) and tied input and output embeddings.
Unless otherwise noted, the dimensionality of word embeddings and hidden states for LSTM are all 650 and uniformly initialized in (-0.05, 0.05).
For PTB with contiguous input, we follow the existing literature to use a batch size of 20, backproprogate for 35 steps (Zaremba et al., 2014), and use SGD with an initial learning rate of 1.0.
A decay of 0.8 is applied whenever the perplexity validation set does not improve.
We use A DAM (Kingma & Ba (2014)) with an initial learning rate 8 e − 5 to train all models for the One-Billion-Word benchmark.
We used a third-party chunker to provide chunk boundaries for the entire PTB data set (Daelemans & Van den Bosch, 2005).
A PPENDIX B I NTRINSIC AND EXTRINSIC EVALUATION FOR WORD EMBEDDINGS
We present detailed results for each of the 18 word embedding dataset in Table 7.
They can be divided into three types of experiments: categorization, similarity, and analogy.
Average scores for each type and also overall average are provided.
Results for three models (Baseline, Isometric-big, and Fixed-big) are presented on 1B with different training sizes.
For the Semantic-Syntactic analogy dataset (Google), we also provide a detailed breakdown of each test category in Table 8.
The HRR model scores very high for man y sy
[Table]
[Table]
[Table]
Google
[Table]
categorization
[Table]
similarity
[Table]
overall
[Table]
[Table]