Phrase-aware Unsupervised Constituency Parsing (2022)

Abstract: Recent studies have achieved inspiring success in unsupervised grammar induction using masked language modeling (MLM) as the proxy task. Despite their high accuracy in identifying low-level structures, prior arts tend to struggle in capturing high-level structures like clauses, since the MLM task usually only requires information from local context. In this work, we revisit LM-based constituency parsing from a phrase-centered perspective. Inspired by the natural reading process of human, we propose to regularize the parser with phrases extracted by an unsupervised phrase tagger to help the LM model quickly manage low-level structures. For a better understanding of high-level structures, we propose a phrase-guided masking strategy for LM to emphasize more on reconstructing non-phrase words. We show that the initial phrase regularization serves as an effective bootstrap, and phrase-guided masking improves the identification of high-level structures. Experiments on the public benchmark with two different backbone models demonstrate the effectiveness and generality of our method.

## introduction the hierarchical structure of natural language plays a key role in accurate language understanding, but can be unfortunately overlooked when text is treated as a plain sequence. to this end, considerable efforts have been made in integrating structural inductive bias into neural language models (lm) @xcite @xcite . despite different implementations, the general idea is to first apply a parsing module to induce the soft grammar tree of the input text, and then incorporate the induced tree into an encoding model (e.g., transformer @xcite ). the model is optimized in an unsupervised manner with masked language modeling (mlm) @xcite as a common proxy task. these models have shown inspiring success in inducing meaningful parsing trees without human annotation, but still face two challenging problems. firstly, the parsing module is randomly initialized at the beginning of the training process. suboptimal initial parsing accuracy can lead to problematic structural constraints in the encoder model, and further influence the training process and final performance @xcite . secondly, the token-level language modeling task encourages the model to focus on local structures, since the reconstruction of a masked word mainly relies on its local context. as a result, the learned model achieves high accuracy in local constituents, like noun phrases (np), but significantly worse accuracy in high-level, long-distance structures, such as subordinate clauses (sbar) and prepositional phrases (pp). on the ptb dataset, the most recent structured language model @xcite still falls behind neural probabilistic context-free grammar models (e.g., @xcite ) by over 4% in average sbar and pp recall. in this work, we revisit the lm-based unsupervised parsing models by providing a phrasecentered perspective. we model the reading process of a sentence in a stylized pipeline: when we try to parse a sentence, instead of handling each individual word, we first recognize the obvious phrases, for instance, names, concepts, slogans, etc. some phrases are known beforehand, while some are learned from the current context. we then treat each phrase as a complete unit, and only need to figure out the high-level structures that connect these phrases. following this intuition, we mimic the reading process with a three-stage learning framework. in the first stage, we identify the multigram phrases with the help of an unsupervised phrase tagging model. the extracted phrase set guides the parsing module to quickly manage the pattern of short constituents at the early training stage. the "warm-up" process does not require any external resource, and effectively improves and stabilizes the initial parsing accuracy. in the second stage, the model is optimized through the original mlm task. after this stage, the model is good at capturing local structures, as stated above. in the third stage, to push the model out of its comfort zone and force it to learn about high-level structures, we apply a simple and effective phrase-guided masked language modeling task. specifically, we extract short phrases in the training sentences as the local constituents identified by the model, which are relatively "easy cases" for the model. we then sample a part of the phrases, and exclude them from the mlm task, so we are basically downsampling intraphrase words in the reconstruction task, and emphasizing non-phrase words that connect phrases. the proposed method is general and can be applied to arbitrary lm-based parsers in a plug-and-play manner. contributions. the major contributions of this paper are summarized as follows: (1) we point out the major challenges faced by lm-based unsupervised constituency parsing, and revisit the problem with a phrase-centered perspective; (2) we propose a novel framework with phrase-regularized warmup and phrase-guided mask language modeling, that can be applied to general lm-based parsers for improvement; (3) experiments on the public benchmark with two different base models demonstrate the effectiveness of our method. code and data will be published for further research study. ## preliminary in this section, we present our problem formulation and briefly review the general framework of lm-based unsupervised constituency parsing, as illustrated in figure 1 . parsing as distance estimation. constituency parsing aims to assign an undirected constituency tree to the input sentence, which illustrates how different parts are hierarchically combined in the sentence @xcite . to enable end-to-end model learning, following prior works @xcite , the discrete parsing tree is represented as a distance sequence d θ (s) = {d 1 , d 2 , ..., d n-1 }, where d i is the distance score between adjacent words w i and w i+1 , parameterized by model θ. given the distance sequence, the tree structure can be induced in a greedy manner: starting from each single token as a leaf constituent, we recursively merge two constituents with the minimum distance score into a large constituent. the tree structure is hence uniquely determined by the relative order of the distance sequence. figure 1 shows a concrete example of the parse tree induction process from an estimated distance sequence. our goal is to learn a high-quality distance estimator d θ from unlabeled text corpus that induces accurate parsing trees. distance-guided model learning. for model learning, the generated distance sequence is injected into an encoding model (e.g., transformer) as structural bias to control information exchange between words. intuitively, two adjacent words with smaller distance score are more likely to belong to the same constituent, and will exchange more information to each other. the distance estimator d θ is jointly optimized with the distanceguided encoder from the masked language modeling (mlm) task as a proxy. formally, given a masking rate µ and a sentence @xmath0, w 2 , ..., w n }, a mask sequence is sampled from uniform bernoulli sampling, where m i is a binary variable with p(m @xmath1. we then get the masked sentence ŝ = { ŵ1 , ..., ŵn } by replacing w i with a mask token where m @xmath2. the mlm loss is computed as: ## framework overview in this work, we recognize and examine two major challenges of lm-based grammar induction: training corpus unsupervised phrase mining stage 1: phrase-regularized warm-up initial phrase set the longest river the biggest city … the fastest car … … … … guides stage 2: standard masked lm training stage 3: phrase-guided masked lm training local constituents the longest river in the world what [m] the [m] river in [m] world distance estimator what is the longest river in the world original sequence masked sequence distance d 6 d 6 d 1 d 1 d 2 d 2 d 3 d 3 d 4 d 4 d 5 d 5 d 7 d 7 distance-guided encoder output training loss what [m] the [m] river in [m] world distance estimator what is the longest river in the world distance-guided encoder the world in longest river the is what currently induced parse tree [c1] [c2] [c3] [c4] [c5] [c6] [c7] what [m] the longest river in [m] world distance estimator what is the longest river in the world distance-guided encoder sample sample for masking mlm loss phrase loss graph legends figure 2 : an overview of the proposed framework. given the training corpus, the training process consists of three stages. stage 1: phrase-regularized warm-up using the initial phrase set extracted by an off-the-shelf unsupervised phrase mining module (section 5); stage 2: standard masked language model learning; stage 3: extract a new phrase set with the local constituents identified by the model itself, and apply phrase-guided masked language model learning (section 6). to tackle the challenges, we revisit lm-based unsupervised constituency parsing from a phrasecentered perspective. we propose a three-stage training framework, as shown in figure 2 . in the first stage, we extract an initial phrase set using an off-the-shelf unsupervised phrase tagger. the extracted phrases serve as effective guidance to help warm up the distance estimator to boost its initial accuracy in the cold start phase. the model then gradually gets rid of the help from the initial phrase set and learns about local structures from the original mlm task in the second stage. in the third stage, we try to push the model out of its comfort zone by moving the focus from local structures to high-level structures. we extract a new phrase set from the local constituents identified by the model itself, which consists of "easy cases" for the model. we then downsample the intra-phrase words for the reconstruction task, and emphasize more on the relatively harder reconstruction of non-phrase words, which connect local constituents into highlevel structures. in following sections, we first introduce the base encoding models we experiment with, and then present more details of the proposed framework. ## distance-guided encoders our method can be applied to any encoder with a distance estimator and distance-constrained information aggregation. in this work, we examine our method on two recently developed models, treetransformer @xcite and struct-former @xcite , as our base models. both models extend the original transformer encoder @xcite by adding a structureaware attention term. specifically, the original transformer computes the attention matrix a as the two base encoders differ in their ways to parameterize the distance function d θ and to define the structure-based attention score q ij . treetransformer parameterizes the distance sequence with an additional attention module. the structure-based attention score q ij represents the probability that two words belong to the same constituent, and is defined as intuitively, words within a closer distance have more information exchange in treetransformer. structformer parameterizes the distance sequence with a convolutional neural network. struct-former uses a more complicated structure constraint: each constituent has a head word, and information can only be exchanged between the head word and remaining child words in the constituent. the structure-based attention score q ij stands for the probability that w i and w j can exchange information, which means w i is the head word of any constituent containing w j , or vice versa. q ij is jointly determined by the distance sequence and a syntacic height sequence. ideally, the height of each child word in a constituent should not exceed the boundary distances. more details can be found in the original paper @xcite . to summarize, the distance estimator d θ determines the attention matrix in the encoder. through the mlm task, the model learns to optimize d θ for more effective information aggregation. we then induce the parse tree from the distance sequence generated by d θ in the parsing process. in following sections, we introduce details about the proposed phrase-regularized warm-up and phraseguided masked language modeling, which jointly help train a better d θ . ## phrase-regularized warm-up given a target sentence, we first extract spans that are likely to be phrases. by definition, we seek word sequences that consistently occur "consecutively in the text, forming a complete semantic unit in certain contexts" @xcite . the extracted phrases are used as additional guidance for the distance estimator at the very beginning of the training process. specifically, we encourage the distance estimator to assign smaller intra-phrase distances than phrase boundary distances to draw a clear gap on the phrase boundaries. figure 3 shows a concrete example of intra-phrase and phrase boundary distances. here we introduce more details about the unsupervised phrase extraction process and phrase regularization for warm-up. ## phrase-guided masked language modeling the masked language modeling task mainly relies on the aggregation of local context information around the masked word. for instance, in the example sentence presented in figure 2 , the prediction of "longest" mainly depends on its neighbor "river". hence, the parser can quickly manage the structure of short phrases as they are closely related to the optimization proxy. high-level long constituents, however, can hardly be captured in this process. from this perspective, the sentence parsing task can then be divided into two parts: parsing the structures of short phrases, and capturing high-level long structures that connect short phrases. the former can be learned from the intra-phrase word reconstruction task, and the latter depends on the modeling of other non-phrase words. following this intuition, we propose simple and effective phrase-guided masked language modeling to emphasize the reconstruction of words outside of local constituents. specifically, we parse the training sentences with the learned model, and treat all local constituents (e.g., with fewer than 4 tokens) from the generated parsing trees. given a sentence with tagged local phrases, we first apply uniform bernoulli sampling on the phrases with probability µ p . the sampled phrases are excluded from the mlm task: words inside of the sampled phrases will not be masked. all rest words are sampled for masking with the original masking rate µ. formally, given a sentence s with the tagged phrase set p s , the probability of word w i being masked in the mlm task is computed as: by doing so, we try to push the model out of its comfort zone of local structure learning, and encourage it to focus more on how the local constituents are connected. discussion. another natural idea to achieve similar intuition is to apply phrase-level reconstruction through whole-phrase masking. namely, we mask the entire phrase so that the model cannot make prediction merely based on information aggregated through local structures, but can only rely on crossphrase structures to gather information. we test this intuition in two ways: (1) replace each token in the phrase with a mask token, and apply standard mlm; ## related work the study of unsupervised constituency parsing can be traced back to 50 years ago @xcite . we highlight some recent progresses that are closely related to our work: ## conclusion in this work, we study the role of phrases in language model-based unsupervised constituency parsing. we propose a phrase-centered framework with novel phrase-regularized warm-up and phraseaware masked language modeling. experiments with two different base models demonstrate the effectiveness of the proposed methods. comprehensive case study is conducted for straightforward understanding of the advantages of our model. although this work mainly focuses on the task of unsupervised parsing, the presented idea and observation can be valuable in more general context. we plan to follow this line of work and further incorporate our method in long-range structured language model learning in the future.