Latent Tree Learning with Ordered Neurons: What Parses Does It Produce? (2020)

Abstract: Recent latent tree learning models can learn constituency parsing without any exposure to human-annotated tree structures. One such model is ON-LSTM (Shen et al., 2019), which is trained on language modelling and has near-state-of-the-art performance on unsupervised parsing. In order to better understand the performance and consistency of the model as well as how the parses it generates are different from gold-standard PTB parses, we replicate the model with different restarts and examine their parses. We find that (1) the model has reasonably consistent parsing behaviors across different restarts, (2) the model struggles with the internal structures of complex noun phrases, (3) the model has a tendency to overestimate the height of the split points right before verbs. We speculate that both problems could potentially be solved by adopting a different training task other than unidirectional language modelling.

## introduction grammar induction is the task of learning the grammar of a target corpus without exposure to the parsing ground truth or any expert-labeled tree structures @xcite . recently emerging latent tree learning models provide a new approach to this problem @xcite @xcite @xcite . they learn syntactic parsing under only indirect supervision from their main training tasks such as language modelling and natural language inference. in this study, we analyze on-lstm @xcite , a new latent tree learning model that set the state of the art on unsupervised constituency parsing on wsj test @xcite when it was published at iclr 2019. the model is trained the interest-only securities were priced at n nn to yield n.n % the interest-only securities were priced at n nn to yield n.n % figure 1 : an example of an on-lstm's parse (top) disagreeing with a binary parse tree converted from a ptb gold-standard parse. on language modelling and can generate binary constituency parsing trees of input sentences like the one in figure 1 . as far as we know, though there is an excellent theoretical analysis paper @xcite of the on-lstm model that focuses on the model's architecture and its parsing algorithm, there is no systematic analysis of the parses the model generates. there are no in-depth investigations of (i) whether the model's parsing behavior is consistent among different restarts or (ii) how the parses it produces are different from ptb gold standards. answering these questions is crucial for a better understanding of the capability of the model and may bring insights into how to build more advanced latent tree learning models in the future. therefore, we replicate the model with 5 random restarts and look into the parses it generates. we find that (1) on-lstm has fairly consistent parsing behaviors across different restarts, achieving a self f1 of 65.7 on wsj test. (2) the model struggles to correctly parse the internal structures of complex noun phrases. (3) the model has a consistent tendency to overestimate the height of the split points right before verbs or auxiliary verbs, leading to a major difference between its parses and the penn treebank gold-standard parses. we speculate that both problems can be explained by the training task, unidirectional language modelling, and thus we hypothesize that training a bidirectional model on a more syntax-related task like acceptability judgement might be a good choice for future latent tree learning models. prpn @xcite and urnng @xcite are two of the stronger latent tree learning models that have comparable unsupervised parsing performance (@xmath0.8 and 52.4) with on-lstm (@xmath1.4). urnng is based on recurrent neural network grammar @xcite , a probablitic generative model; prpn is a neural language model that implicitly models syntax using a structured attention mechanism. @xcite analyze st-gumbel and rl-spinn. they find that though the two models perform well on sentence understanding, neither of the models induces consistent and non-trivial grammars. prior to this work, @xcite also analyze on-lstm. they raise doubts on the necessity of the model's novel gates and mathematically prove that it is impossible for the parsing algorithm used by @xcite to correctly parse a certain class of structures. in comparison, this study takes a more empirical approach that is similar to that of @xcite . ## data and model wsj dataset wsj is the wall street journal section of ptb @xcite , which is the most commonly used dataset for training and evaluating parsers including latent tree learning models @xcite . it is also the dataset on-lstm is originally trained on. we follow the traditional split of wsj: sections 0-21 as wsj train, section 22 as wsj dev, and section 23 as wsj test. we also use wsj 10, a subset of wsj that includes all sentences with @xmath2. in the experiments, the model is always trained on wsj train on language modelling, and evaluated on wsj test, wsj dev and/or wsj 10 on constituency parsing. models on-lstm is an lstm model @xcite plus a novel activation function, which causes the model to learn to store long-term information in high-ranking dimensions, implicitly encoding a constituency parse. the model is equipped with a master forget gate f t and a master input gate i t . at each timestep, f t is multiplied element-wise to the previous cell state c t-1 and thus controls to what extent the value of each dimension in the previous cell state can be forgotten; i t is multiplied element-wise to the candidate update values ĉt and thus controls how much new information can be written to each dimension in the cell state. the values of f t and i t are computed at each timestep based on the input token and the cell state. the model uses cumax() as the activation function of the master gates, where therefore, the values in f t are always monotonically increasing from 0 to 1, and the values in i t are always monotonically decreasing from 1 to 0. as a result, when a dimension is updated/erased, all of the dimensions whose ranks are lower than it are also updated/erased. intuitively, in an extreme and simplified example where f @xmath3, ..., 0, 1, ..., 1), the model is just picking a dimension d, erases all the dimensions from 1 to d -1 in c t-1 , and keeps @xmath4. as a result of this novel updating rule, the model will tend to store long-term information in highranking dimensions and short-term information in low-ranking dimensions so that when the model frequently erases and updates low-ranking dimensions of the cell state, the long-term information stored in high-ranking dimensions will stay unaffected. when the model is trained to perform language modelling, since a higher-level constituent always spans more words than its children, its related information will continuously be useful for word prediction in a longer term and will thus be stored in higher-ranking dimensions (see fig 2 for an example). therefore, intuitively, if a high-ranking dimension is erased/updated, it probably means that the currently processed input token is the start of a new high-level constituent. figure 2 : an example of the correspondences between a constituency parse tree and the hidden states of on-lstm. intuitively, when performing language modelling, the information related to the highest-level constituent "s" is useful when predicting both token x 2 and token x 3 , while the information related to the first "n" is only useful in predicting x 2 , and can be erased after the prediction of x 2 . in order to avoid removing "s" information when removing "n" information, the model will store "s" information in higher dimensions, and information of "n" in lower dimensions. image source: shen et al. (2019) based on this intuition, the master forget gates can be used to perform binary constituency parsing. in binary parsing, a constituent (which is initially the whole sentence) is recursively split into two constituents until each constituent contains only one word. therefore, each space between each pair of adjacent words is a split point the parsing algorithm will use at some point to make a split, and the order in which these split points are used decides what the resultant parsing tree will be like. in the case of on-lstm, the parsing algorithm uses the split points in the decreasing order of their "height", where the height of a split point between x t-1 and x t is usually defined as df t foot_0 , an estimate of the transition point foot_1 in f t from the 0-segment foot_2 where values are small and close to 0 to the 1-segment where values are large and close to 1. intuitively, the more information is forgotten at a timestep, the higher the split point before the token. we train on-lstm with 5 different random seeds on wsj train using hyperparameters shared by @xcite . note that the training objective is language modelling, so we only use the sentences from wsj train and the model never has access to the parsing trees in the dataset or any other tree structures. on wsj test, our models achieve an average perplexity of 56.33 (±0.06) on language modelling and average f1 score of 46.43 (±1.79) on unsupervised parsing, while the original paper reports 56.17 (±0.12) and 47.7 (±1.5). this shows that we roughly reproduce their work 4 . ## experiments to analyze the model's consistency, we use the 5 models we train to parse wsj test and wsj 10, calculate the self f1 and standard deviation on each dataset, and compare them to that of the random baseline. self f1 is the average of unlabeled binary f1 scores between every pairing of the five parses, each produced by one model. it shows to what extent each model agrees with the parsing decisions of the other four. to take a closer look at the parses generated by the model, we then use our 5 models to parse wsj dev, and report the average of the models' parsing accuracies on each constituent type. we use the constituent-level accuracy as a guide to analyze how the parses on-lstm produces are different from ptb gold standards. in the experiments, we include a simple random baseline that produces parses by recursively and randomly splitting the sequences to two halves. this is the same with on-lstm's parsing algorithm except that the baseline model chooses split points in a random order. ## conclusions and future work in summary, the model shows basic selfconsistency on the task of constituency parsing, and it is consistently able to correctly identify certain constituents (sq, vp, np). all these results show that the unique design of the model brings us closer to developing consistently powerful unsupervised parsing models. however, the experiments show that it (a) struggles with the internal structures of complex nps, and (b) often overestimates the height of the split points right before verbs. based on our analysis, we hypothesize that both of the failures can be at least partially attributed to the use of unidirectional language modelling as the training task. there are two potential problems with this training task. first, the motivation of language modelling generally does not perfectly match the target task constituency parsing, since cross-constituent hints are sometimes helpful, as revealed by (a). second, it is very hard for a unidirectional model to correctly identify some high-level constituents, as revealed by (b). therefore, we believe a promising research direction is to build latent tree learning models based on bidirectional model architectures like transformer @xcite and the task of acceptability judgement with a dataset like cola @xcite , which is a more syntax-related sentence-level task that requires the model to predict whether an input sentence is grammatically acceptable. another option to consider is masked language modelling because it is also a bidirectional task and is much easier to scale up compared to acceptability judgement since it is a self-supervised task.