N EURAL L ANGUAGE M ODELING BY J OINTLY L EARNING S YNTAX AND L EXICON
Anonymous authors
Paper under double-blind review
A BSTRACT
We propose a neural language model capable of unsupervised syntactic structure induction
The model leverages the structure information to form better semantic representations and better language modeling
Standard recurrent neural networks are limited by their structure and fail to efﬁciently use syntactic information.
On the other hand tree-structured recursive networks usually require additional structural supervision at the cost of human expert annotation.
In this paper
, We propose a novel neural language model, called the Parsing-Reading-Predict Networks (PRPN), that can simultaneously induce the syntactic structure from unannotated sentences and leverage the inferred structure to learn a better language model.
In our model, the gradient can be directly back-propagated from the language model loss into the neural parsing network
Experiments show that the proposed model can discover the underlying syntactic structure and achieve state-of-the-art performance on word character level language model tasks
I NTRODUCTION
Linguistic theories generally regard natural language as consisting of two part: a lexicon , the complete set of all possible words in a language; and a syntax , the set of rules, principles, and processes that govern the structure of sentences (Sandra & Taft, 1994).
To generate a proper sentence, tokens are put together with a speciﬁc syntactic structure.
Understanding a sentence also requires lexical information to provide meanings, and syntactical knowledge to correctly combine meanings.
Current neural language models can provide meaningful word represent (Bengio et al., 2003; Mikolov et al., 2013; Chen et al., 2013).
However, standard recurrent neural networks only implicitly model syntax, thus fail to efﬁciently use structure information (Tai et al., 2015).
Developing a deep neural network that can leverage syntactic knowledge to form a better semantic representation has received a great deal of attention in recent years (Socher et al., 2013;
Tai et al., 2015; Chung et al., 2016).
Integrating syntactic structure into a language model is important for different reasons: 1) to obtain a hierarchical representation with increasing levels of abstraction, which is a key feature of deep neural networks and of the human brain (Bengio et al., 2009; LeCun et al., 2015; Schmidhuber, 2015); 2) to capture complex linguistic phenomena, like long-term dependency problem (Tai et al., 2015) and the compositional effects (Socher et al., 2013); 3) to provide shortcut for gradient back-propagation (Chung et al., 2016).
A syntactic parser is the most common source for structure information.
Supervised parsers can achieve very high performance on well constructed sentences.
Hence, parsers can provide accurate information about how to compose word semantics into sentence semantics (Socher et al., 2013), or how to generate the next word given previous words (Wu et al., 2017).
However, only major languages have treebank data for training parsers, and it request expensive human expert annotation.
People also tend to break language rules in many circumstances (such as writing a tweet).
These defects limit the generalization capability of supervised parsers.
Unsupervised syntactic structure induction has been among the longstanding challenges of computational linguistic (Klein & Manning, 2002; 2004; Bod, 2006).
Researchers are interested in this problem for a variety of reasons: to be able to parse languages for which no annotated treebanks exist (Marecek, 2016); to create a dependency structure to better suit a particular NLP application (Wu et al., 2017); to empirically argue for or against the poverty of the stimulus (Clark, 2001; Chomsky, 2014); and to examine cognitive issues in language learning (Solan et al., 2003).
In this paper, we propose a novel neural language model: Parsing-Reading-Predict Networks (PRPN), which can simultaneously induce the syntactic structure from unannotated sentences and leverage the inferred structure to form a better language model.
With our model, we assume that language can be naturally represented as a tree-structured graph.
The model is composed of three parts.
A differentiable neural Parsing Network uses a convolutional neural network to compute the syntactic distance , which represents the syntactic relationships between all successive pairs of words in a sentence, and then makes soft constituent decisions based on the syntactic distance.
A Reading Network that recurrently computes an adaptive memory representation to summarize information relevant to the current time step, based on all previous memories that are syntactically and directly related to the current token.
A Predict Network that predicts the next token based on all memories that are syntactically and directly related to the next token.
We evaluate our model on three tasks: word-level language modeling, character-level language modeling, and unsupervised constituency parsing.
The proposed model achieves (or is close to)
the state-of-the-art on both word-level and character-level language modeling.
The model’s unsupervised parsing outperforms some strong baseline models, demonstrating that the structure found by our model is similar to the intrinsic structure provided by human experts.
R ELATED W ORK
The idea of introducing some structures, especially trees, into language modeling to help a downstream task has been explored in various ways.
For example, Socher et al.
; Tai et al.
(2015) learn a bottom-up encoder, taking as an input a parse tree supplied from an external parser.
There are models that are able to infer a tree during test time, while still need supervised signal on tree structure during training.
For example, (Socher et al., 2010; Alvarez-Melis & Jaakkola, 2016; Zhou et al., 2017; Zhang et al., 2015), etc.
Moreover, Williams et al.
(2017) did an in-depth analysis of recursive models that are able to learn tree structure without being exposed to any grammar trees.
Our model is also able to infer tree structure in an unsupervised setting, but different from theirs, it is a recurrent network that implicitly models tree structure through attention.
Apart from the approach of using recursive networks to capture structures, there is another line of research which try to learn recurrent features at multiple scales, which can be dated back to 1990s (e.g. El Hihi & Bengio (1996); Schmidhuber (1991); Lin et al.
The NARX RNN (Lin et al., 1998) is another example which used a feed forward net taking different inputs with predeﬁned time delays to model long-term dependencies.
More recently, Koutnik et al.
(2014) also used multiple layers of recurrent networks with different pre-deﬁned updating frequencies.
Instead, our model tries to learn the structure from data, rather than predeﬁning it.
In that respect, Chung et al.
(2016) relates to our model since it proposes a hierarchical multi-scale structure with binary gates controlling intralayer connections, and the gating mechanism is learned from data too.
The difference is that their gating mechanism controls the updates of higher layers directly, while ours control it softly through an attention mechanism.
Parsers are also related to our work since they are all inferring grammatical tree structure given a sentence.
For example, SPINN (Bowman et al., 2016) is a shift-reduce parser that uses an LSTM as its composition function.
The transition classiﬁer in SPINN is supervisedly trained on the Stanford
PCFG Parser (Klein & Manning, 2003) output.
Unsupervised parsers are more aligned with what our model is doing.
Klein & Manning (2004) presented a generative model for the unsupervised learning of dependency structures.
Klein & Manning (2002) is a generative distributional model for the unsupervised induction of natural language syntax which explicitly models constituent yields and contexts.
We compare our parsing quality with the aforementioned two papers in Section 6.3.
M OTIVATION
Suppose we have a sequence of tokens x 0 , ..., x 6 governed by the tree structure showed in Figure 1.
The leafs x i are observed tokens.
Node y i represents the meaning of the constituent formed by its leaves x l ( y i ) , ..., x r ( y i ) , where l ( · ) and r ( · ) stands for the leftmost child and right most child.
Root r represents the meaning of the whole sequence.
Arrows represent the dependency relations between nodes.
The underlying assumption is that each node depends only on its parent and its left siblings.
Directly modeling the tree structure is a challenging task, usually requiring supervision to learn (Tai et al., 2015).
In addition, relying on tree structures can result in a model that is not sufﬁciently robust to face ungrammatical sentences (Hashemi & Hwa, 2016).
In contrast, recurrent models provide a convenient way to model sequential data, with the current hidden state only depends on the last hidden state.
This makes models more robust when facing nonconforming sequential data, but it suffers from neglecting the real dependency relation that dominates the structure of natural language sentences.
In this paper, we use skip-connection to integrate structured dependency relations with recurrent neural network.
In other words, the current hidden state does not only depend on the last hidden state, but also on previous hidden states that have a direct syntactic relation to the current one.
Figure 2 shows the structure of our model.
The non-leaf node y j is represented by a set of hidden states y j
= { m i } l ( y j ) ≤ i ≤ r ( y j ) , where l ( y j ) is the left most descendant leaf and r ( y j ) is the right most one.
Arrows shows skip connections built by our model according to the latent structure.
Skip connections are controlled by gates g ti .
In order to deﬁne g ti , we introduce a latent variable l t to represent local structural context of x t :
• if x t is not left most child of any subtree, then l t is the position of x t ’s left most sibling.
• if x t is the left most child of a subtree y i , then l t is the position of the left most child that belongs to the left model sibling of y i .
and gates are deﬁned as:
[Equation]
Given this architecture, the siblings dependency relation is modeled by at least one skip-connect.
The skip connection will directly feed information forward, and pass gradient backward.
The parentto-child relation will be implicitly modeled by skip-connect relation between nodes.
The model recurrently updates the hidden states according to:
[Equation]
and the probability distribution for next word is approximated by:
[Equation]
where g t i are gates that control skip-connections.
Both f and h have a gating mechanism that takes g t i as input and forces the model to accumulatively forget about the longer term dependencies and focus on the latest information.
Since l t is an unobserved latent variable, We explain an approximation for g ti in the next section.
M ODELING S YNTACTIC S TRUCTURE 4. M ODELING L OCAL S TRUCTURE
In this section we give a probabilistic view on how to model the local structure of language.
A detailed elaboration for this section is given in Appendix B.
At time step t , p ( l t +1 | x 0 , ..., x t ) represents the probability of choosing one out of t possible local structures that deﬁnes the conditional dependencies.
We propose to model the distribution by Dirichlet Process:
[Equation]
where α t +1 j is parametrized in next section.
As shown in Appendix B, the expectation of gate value g t
+1 i is the Cumulative Distribution Function (CDF) of p ( l t +1 = t (cid:48) | x 0 , ..., x t ) .
Thus, we can replace the discrete gate value by its expectation:
[Equation]
With these relaxations, Eq. 2 and 3 can be approximated by using a soft gating vector to update the hidden state and the predict next token.
4. P ARSING N ETWORK
Inferring tree structure with Syntactic Distance
The masking vector g t +
i introduced in Section 4.1 determines the range of dependency, i.e., for the word x t
+1 we have g t
+1 i
= 1 for all t (cid:48) ≤ i < t
+ 1 .
This range is relaxed to a soft version in Eq.
All the words fall into the range t (cid:48) ≤ i < t is considered as x t ’s sibling or offspring of its sibling.
If the dependency ranges of two words are disjoint with each other, that means the two words belong to two different subtrees.
If one range contains another, that means the one with smaller range is a sibling, or is an offspring of a sibling of the other word.
However, if they partially overlaps, they can’t form a valid tree.
In that respect, we have to parameterize α tj carefully so that the structure given by g ti ’s approximates a valid tree.
The solution lies in the introduction of Syntactic Distance .
For a sentence with length K we deﬁne a set of K − 1 real valued scalar variables d 0 , ..., d k − 1 , each of them stands for a measure of the syntactic relation between a pair of adjacent words.
Given these syntactic distances, α tj can be deﬁned as:
[Equation]
which is basically the sign of comparing d t and d j +1 , scaled to the range of 0 and 1.
Then for each of its previous token the gate value g ti can be computed through Eq.
Now for a certain x t , we have
[Equation]
where
[Equation]
Now all the words that fall into the range t (cid:48) ≤ i < t are considered as either sibling of x t , or offspring of a sibling of x t (Figure 3).
The essential point here is that, under this parameterization, the dependency range of any two tokens won’t partially overlap.
Here we provide a terse proof in
Appendix C.
Further, to enable training through back-propagation, and also to make the model robust to ungrammatical sentences, we soften the sign( · ) function with hardtanh( · ) , and add a scaling factor τ to
Eq. 6:
[Equation]
where hardtanh( x ) = max( − 1 , min(1 , x )) .
τ is the temperature parameter that controls the sensitivity of α ti to the differences between distances.
This soft gating mechanism has some nice properties that both allow us to infer a tree structure from it and be robust to intermediate non-valid tree structures that the model may encounter during learning.
In Appendix D we list these properties and explained more on the meanings of their values.
Parameterizing
Syntactic Distance
The syntactic distance between a given token (which is usually represented as a vector word embedding e i ) and its previous token e i − 1 , is provided by a convolutional kernel over a set of consecutive previous tokens e i − L , e i − L +1 , ..., e i .
This convolution is depicted as the gray triangles shown in Figure 3.
Each triangle here represent 2 layers of convolution.
Formally, the syntactic distance d i between token e i − 1 and e i is computed by
[Equation]
[Equation]
where W c , b c are the kernel parameters.
W d and b d can be seen as another convolutional kernel with window size 1, convolved over h i ’s.
Here the kernel window size L determines how far back into the history node e i can reach while computing its syntactic distance d i .
Thus we call it the look-back range .
Note that, at time step t , we compute a temporary estimation of d t +1 :
[Equation]
From there we compute its corresponding α (cid:48)
t +1 and g (cid:48)
t +1 for Eq. 3.
Convolving h and d on the whole sequence with length K yields a set of distances.
For the tokens in the beginning of the sequence, we simply pad L − 1 zero vectors to the front of the sequence in order to get K − 1 outputs.
M ODELING L ANGUAGE 5. R EADING N ETWORK
The Reading Network generate new hidden states considering on input x t , previous hidden states m 0 , ..., m t − 1 , and gates g t 0 , ..., g tt − 1 , as shown in Eq. 2.
Similar to Long Short-Term MemoryNetwork (LSTMN) (Cheng et al., 2016), the Reading Network maintains two sets of vectors: a hidden tape H t − 1 = ( h t − N m , ..., h t − 1 ) , and a memory tape C t − 1 = ( c t − L , ..., c
t − 1 ) , where N m is the upper bound for the memory span.
The Reading Network captures the dependency relation by a modiﬁed attention mechanism: structured attention .
At each step of recurrence, the model summarizes the previous recurrent states via the structured attention mechanism, then performs a normal LSTM update, with hidden and cell states output by the attention mechanism.
Structured Attention At each time step t , the read operation attentively links the current token to previous memories with a structured attention layer:
[Equation]
where, δ k is the dimension of the hidden state.
Modulated by the gates in Eq.
5, the structured intra-attention weight is deﬁned as:
[Equation]
This yields a probability distribution over the hidden state vectors of previous tokens.
We can then compute an adaptive summary vector for the previous hidden tape and memory denoting by ˜ h t and ˜ c t :
[Equation] [Equation]
Structured attention provides a way to model the dependency relations shown in Figure 1.
Recurrent Update
The Reading Network takes x t , ˜ c t and ˜ h t as input, computes the values of c t and h t by the LSTM recurrent update (Hochreiter & Schmidhuber, 1997).
Then the write operation concatenates h t and c t to the end of hidden and memory tape.
5. P REDICT N ETWORK
Predict Network models the probability distribution of next word x t +1 , considering on hidden states m 0 , ..., m t , and gates g t +1 0 , ..., g t +1 t − 1 , as shown Eq.
In order to compute { g t +1 i } at timestep t , model estimate the value of d t +1 using Eq.
We parametrize f ( · ) function as:
[Equation] [Equation]
where h l : t − 1 is an adaptive summary of h l t +1 ≤ i ≤ t − 1 , output by structured attention controlled by g t +10 , ..., g t +1 t − 1 .
ˆ f ( · ) could be a simple feed-forward MLP, or more complex architecture, like ResNet, to add more depth to the model.
E XPERIMENTS
We evaluate the proposed model on three tasks, character-level language modeling, word-level language modeling, and unsupervised constituency parsing.
6. C HARACTER - LEVEL L ANGUAGE M ODEL
From a character-level view, natural language is a discrete sequence of data, where discrete symbols form a distinct and shallow tree structure: the sentence is the root, words are children of the root, and characters are leafs.
However, compared to word-level language modeling, character-level language modeling requires the model to handle longer-term dependencies.
We evaluate a character-level variant of our proposed language model over a preprocessed version of the Penn Treebank (PTB) and Text8 datasets.
When training, we use truncated back-propagation, and feed the ﬁnal memory position from the previous batch as the initial memory of next one.
At the beginning of training and test time, the model initial hidden states are ﬁlled with zero.
Optimization is performed with Adam using learning rate lr = 0 .
003 , weight decay w decay = 10 − 6 , β 1 = 0 .
9 , β 2 = 0 .
999 and σ = 10 − 8 .
We carry out gradient clipping with maximum norm 1.0.
The learning rate is multiplied by 0.1 whenever validation performance does not improve during 2 checkpoints.
These checkpoints are performed at the end of each epoch.
We also apply layer normalization (Ba et al., 2016) to the Reading Network and batch normalization to the Predict Network and parsing network.
For all of the character-level language modeling experiments, we apply the same procedure, varying only the number of hidden units, mini-batch size and dropout rate.
Penn Treebank we process the Penn Treebank dataset (Marcus et al., 1993) by following the procedure introduced in (Mikolov et al., 2012).
For character-level PTB, Reading Network has two recurrent layers, Predict Network has one residual block.
Hidden state size is 1024 units.
The input and output embedding size are 128, and not shared.
Look-back range L = 10 , temperature parameter τ = 10 , upper band of memory span N m = 20 .
We use a batch size of 64, truncated backpropagation with 100 timesteps.
The values used of dropout on input/output embeddings, between recurrent layers, and on recurrent states were (0, 0.25, 0.1) respectively.
In Figure 4, we visualize the syntactic distance estimated by the Parsing Network, while reading three different sequences from the PTB test set.
We observe that the syntactic distance tends to be higher between the last character of a word and a space, which is a reasonable breakpoint to separate between words.
In other words, if the model sees a space, it will attend on all previous step.
If the model sees a letter, it will attend no further then the last space step.
The model autonomously discovered to avoid inter-word attention connection, and use the hidden states of space (separator) tokens to summarize previous information.
This is strong proof that the model can understand the latent structure of data.
As a result our model achieve state-of-the-art performance and signiﬁcantly outperform baseline models.
It is worth noting that HM-LSTM (Chung et al., 2016) also unsupervisedly induce similar structure from data.
But discrete operations in HM-LSTM make their training procedure more complicated then ours.
[Table]
6. W ORD - LEVEL L ANGUAGE M ODEL
Comparing to character-level language modeling, word-level language modeling needs to deal with complex syntactic structure and various linguistic phenomena.
But it has less long-term dependencies.
We evaluate the word-level variant of our language model on a preprocessed version of the
Penn Treebank (PTB) (Marcus et al., 1993) and Text8 (Mahoney, 2011) dataset.
We apply the same procedure and hyper-parameters as in character-level language model.
Except optimization is performed with Adam with β 1 = 0 .
This turns off the exponential moving average for estimates of the means of the gradients (Melis et al., 2017).
We also adapt the number of hidden units, mini-batch size and the dropout rate according to the different tasks.
Penn Treebank we process the Penn Treebank dataset (Mikolov et al., 2012) by following the procedure introduced in (Mikolov et al., 2010).
For word-level PTB, the Reading Network has two recurrent layers and the Predict Network do not have residual block.
The hidden state size isunits and the input and output embedding sizes are 800, and shared.
Look-back range L = 5 , temperature parameter τ = 10 and the upper band of memory span N m = 15 .
We use a batch size of 64, truncated back-propagation with 35 time-steps.
The values used of dropout on input/output embeddings, between recurrent layers, and on recurrent states were (0.7, 0.5, 0.5) respectively.
[Table]
Text8 dataset contains 17M training tokens and has a vocabulary size of 44k words.
It has been recently used for language modeling and has a vocabulary of 44k words.
The dataset is partitioned into a training set (ﬁrst 99M characters) and a development set (last 1M characters) that is used to report performance.
As this dataset contains various articles from Wikipedia, the longer term information (such as current topic) plays a bigger role than in the PTB experiments (Mikolov et al., 2014).
We apply the same procedure and hyper-parameters as in character-level PTB, except we use
0.2) respectively
[Table]
In Table 2, our results are comparable to the state-of-the-art methods.
Since we do not have the same computational resource used in (Melis et al., 2017) to tune hyper-parameters at large scale, we expect that our model could achieve better performance after an aggressive hyperparameter tuning process.
As shown in Table 3, our method outperform baseline methods.
It is worth noticing that the continuous cache pointer can also be applied to output of our Predict Network without modiﬁcation.
Visualizations of tree structure generated from learned PTB language model are included in
Appendix A.
6. U NSUPERVISED C ONSTITUENCY P ARSING
The unsupervised constituency parsing task compares hte tree structure inferred by the model with those annotated by human experts.
The experiment is performed on WSJ10 dataset.
WSJ10 is the 7422 sentences in the Penn Treebank Wall Street Journal section which contained 10 words or less after the removal of punctuation and null elements.
Evaluation was done by seeing whether proposed constituent spans are also in the Treebank parse, measuring unlabeled F1 ( UF 1 ) of unlabeled constituent precision and recall.
Constituents which could not be gotten wrong (those of span one and those spanning entire sentences) were discarded.
Given the mechanism discussed in Section 4.2, our model generates a binary tree.
Although standard constituency parsing tree is not limited to binary tree.
Previous unsupervised constituency parsing model also generate binary trees (Klein & Manning, 2002; Bod, 2006).
Our model is compared with the several baseline methods, that are explained in Appendix E.
Different from the previous experiment setting, the model treat each sentence independently during train and test time.
When training, we feed one batch of sentences at each iteration.
In a batch, shorter sentences are padded with 0.
At the beginning of the iteration, the model’s initial hidden states are ﬁlled with zero.
When testing, we feed on sentence one by one to the model, then use the gate value output by the model to recursively combine tokens into constituents, as described in Appendix A.
[Table]
Table 4 summarizes the results.
Our model signiﬁcantly outperform the RANDOM baseline indicate a high consistency with human annotation.
Our model also shows a comparable performance with
CCM model.
In fact our parsing network and CCM both focus on the relation between successive tokens.
As described in Section 4.2, our model computes syntactic distance between all successive pair of tokens, then our parsing algorithm recursively assemble tokens into constituents according to the learned distance.
CCM also recursively model the probability whether a contiguous subsequences of a sentence is a constituent.
Thus, one can understand how our model is outperformed by DMV+CCM and UML-DOP models.
The DMV+CCM model has extra information from a dependency parser.
The UML-DOP approach captures both contiguous and non-contiguous lexical dependencies (Bod, 2006).
C ONCLUSION
In this paper, we propose a novel neural language model that can simultaneously induce the syntactic structure from unannotated sentences and leverage the inferred structure to learn a better language model.
We introduce a new neural parsing network: Parsing-Reading-Predict Network, that can make differentiable parsing decisions.
We use a new structured attention mechanism to control skip connections in a recurrent neural network.
Hence induced syntactic structure information can be used to improve the model’s performance.
Via this mechanism, the gradient can be directly backpropagated from the language model loss function into the neural Parsing Network.
The proposed model achieve (or is close to) the state-of-the-art on both word/character-level language modeling tasks.
Experiment also shows that the inferred syntactic structure highly correlated to human expert annotation.
APPENDIX A I NFERRED T REE S TRUCTURE
The tree structure is inferred from the syntactic distances yielded by the Parsing Network.
We ﬁrst sort the d
i ’s in decreasing order.
For the ﬁrst d i in the sorted sequence, we separate sentence into constituents (( x <i ) , ( x i , ( x >i ))) . Then we separately repeat this operation for constituents ( x <i ) and ( x >i ) . Until the constituent contains only one word. B M ODELING L OCAL S TRUCTURE
In this section we give a probabilistic view on how to model the local structure of language.
Given the nature of language, sparse connectivity can be enforced as a prior on how to improve generalization and interpretability of the model.
At time step t , p ( l t +1 | x 0 , ..., x t ) represents the probability of choosing one out of t possible local structures that deﬁnes the conditional dependencies.
If l t = t (cid:48) , it means x t depends on all the previous hidden state from m t (cid:48) to m t ( t (cid:48) ≤ t ).
A particularly ﬂexible option for modeling p ( l t +1 | x 0 , ..., x t ) is the Dirichlet Process, since being non-parametric allows us to attend on as many words as there are in a sentence; i.e. number of possible structures (mixture components) grows with the length of the sentence.
As a result, we can write the probability of l t +1 = t (cid:48) as a consequence of the stick breaking process 1 :
[Equation]
for 1 ≤ t (cid:48) < t , and
[Equation]
where α j = 1 − β j and β j is a sample from a Beta distribution.
Once we sample l t +1 from the process, the connectivity is realized by a element-wise multiplication of an attention weight vector with a masking vector g t +1 deﬁned in Eq.
In this way, x t becomes functionally independent of all x s for all s < l t +1 .
The expectation of this operation is the CDF of the probability of l , since
[Equation]
By telescopic cancellation, the CDF can be expressed in a succinct way:
[Equation]
for t (cid:48) < t , and P ( l t +1 ≤ t )
= 1 .
However, being Bayesian nonparametric and assuming a latent variable model require approximate inference.
Hence, we have the following relaxations
First, we relax the assumption and parameterize α t +1 j as a deterministic function depending on all the previous words, which we will describe in the next section
We replace the discrete decision on the graph structure with a soft attention mechanism by multiplying attention weight with the multiplicative gate:
[Equation]
With these relaxations, Eq.
(3) can be approximated by using a soft gating vector to update the hidden state h and the predictive function f .
This approximation is reasonable since the gate is the expected value of the discrete masking operation described above.
C P ROOF FOR N O P ARTIAL O VERLAPPING IN D EPENDENCY R ANGES
Proof.
Let’s assume that the dependency range of x v and x n partially overlaps.
We should have g ui = 1 for u ≤ i < v
and g ni
= 1 for m ≤ i < n .
Without losing generality, we assume u < m <
v < n so that the two dependency ranges overlap in the range [ m, v ] .
For x v , we have α vi = for all u ≤ i < v .
According to Eq.
6 and 5, we have d i < d v for all u ≤ i
< v Since u < m , we have d m < d v .
Similarly, for x n , we have d i < d n for all m ≤ i < n .
Since m < v , we have d v < d n .
On the other hand, since the range stops at m , we should also have d m > d n .
Thus d m > d v .
Items 1 and 2 are contradictory, so the dependency ranges of x v and x n won’t partially overlap.
D P ROPERTIES AND I NTUITIONS OF g ti AND d i
First, for any ﬁxed t , g ti is monotonic in i .
This ensures that g ti still provides soft truncation to deﬁne a dependency range.
The second property comes from τ .
The hyperparameter τ has an interesting effect on the tree structure: if it is set to 0, then for all t , the gates g ti will be open to all of e t ’s predecessors, which will result in a ﬂat tree where all tokens are direct children of the root node; as τ becomes larger, the number of levels of hierarchy in the tree increases.
As it approaches + inf , the hardtanh( · ) becomes sign( · ) and the dependency ranges form a valid tree.
Note that, due to the linear part of the gating mechanism, which beneﬁts training, when τ has a value in between the two extremes the truncation range for each token may overlap.
That may sometimes result in vagueness in some part of the inferred tree.
To eliminate this vagueness and ensure a valid tree, at test time we use τ = + inf .
Under this framework, the values of syntactic distance have more intuitive meanings.
If two adjacent words are siblings of each other, the syntactic distance should approximate zero; otherwise, if they belong to different subtrees, they should have a larger syntactic distance.
In the extreme case, the syntactic distance approaches 1 if the two words have no subtree in common.
In Figure 3 we show the syntactic distances for each adjacent token pair which results in the tree shown in Figure 1.
E B ASELINE M ETHODS FOR U NSUPERVISED C ONSTITUENCY P ARSING
Our model is compared with the same baseline methods as in (Klein & Manning, 2005).
RANDOM chooses a binary tree uniformly at random from the set of binary trees.
This is the unsupervised baseline.
LBRANCH and RBRANCH choose the completely left- and right-branching structures, respectively.
RBRANCH is a frequently used baseline for supervised parsing, but it should be stressed that it encodes a signiﬁcant fact about English structure, and an induction system need not beat it to claim a degree of success.
UPPER BOUND is the upper bound on how well a binary system can do against the Treebank sentences.
Because the Treebank sentences are generally more ﬂat than binary, limiting the maximum precision which can be attained, since additional brackets added to provide a binary tree will be counted as wrong.
We also compared our model with other unsupervised constituency parsing methods.
DEP-PCFG is dependency-structured PCFG (Carroll & Charniak, 1992).
CCM is constituent-context model (Klein & Manning, 2002).
DMV is an unsupervised dependency parsing model.
DMV+CCM is a combined model that jointly learn both constituency and dependency parser (Klein & Manning, 2004).