Contextual Distortion Reveals Constituency: Masked Language Models are Implicit Parsers (2023)

Abstract: Recent advancements in pre-trained language models (PLMs) have demonstrated that these models possess some degree of syntactic awareness. To leverage this knowledge, we propose a novel chart-based method for extracting parse trees from masked language models (LMs) without the need to train separate parsers. Our method computes a score for each span based on the distortion of contextual representations resulting from linguistic perturbations. We design a set of perturbations motivated by the linguistic concept of constituency tests, and use these to score each span by aggregating the distortion scores. To produce a parse tree, we use chart parsing to find the tree with the minimum score. Our method consistently outperforms previous state-of-the-art methods on English with masked LMs, and also demonstrates superior performance in a multilingual setting, outperforming the state-of-the-art in 6 out of 8 languages. Notably, although our method does not involve parameter updates or extensive hyperparameter search, its performance can even surpass some unsupervised parsing methods that require fine-tuning. Our analysis highlights that the distortion of contextual representation resulting from syntactic perturbation can serve as an effective indicator of constituency across languages.

## introduction constituency parsing is a fundamental task in natural language processing (nlp) that involves uncovering the syntactic structure of a sentence by identifying the constituents it is composed of. while supervised constituency parsing methods necessitate the utilization of a labeled dataset containing sentences and their corresponding constituency parses, unsupervised methods for generating syntax trees emerge because manual annotation is labor-intensive and requires specialized linguistic 1 our code is available at https://github.com/ jxjessieli/contextual-distortion-parser . figure 1 : example sentence and its constituency tree. we list perturbed sentences after substitution (1), decontextualization (2), and movement (3). in this paper, we propose a simple yet effective cpe-plm approach to bridge the performance gap between these two methods by input perturbations designed based on the intuition of constituency tests. linguists use constituency tests to determine whether a span of words forms a constituent in a sentence. one common constituency test is the substitution test which replaces the span of words with a single pronoun (such as "it" or "they") and checks if the sentence is still grammatical. for example, in figure 1 , the span "a film" can be replaced with the pronoun "it", resulting in the sentence "they watched it this afternoon," which is still grammatical. this suggests that "a film" is likely a constituent. our goal in this work is to maximally explore the capabilities of plms to induce grammar by themselves. specifically, we focus on masked lms and leverage the inherent properties of the mask token prediction pre-training objective. the main idea is to make pre-trained language models think like linguists, such that with constituency tests, span-level scores reflecting the likelihood of a span being a constituent can be obtained. the evaluation of constituency tests traditionally relies on grammaticality judgments. @xcite trained a classifier that can make grammaticality decisions with external data. in contrast, our approach assesses the degree of alternation in contextual representations resulting from manipulations akin to those used in constituency tests. we hypothesize that, when the context of a span is manipulated, the contextual representations of constituents will exhibit minimal alteration compared to those of distituents @xcite . we refer to these manipulations as perturbations, as our method measures the sensitivity of the representations to these changes. we define three perturbations and for each perturbation, we alter the input sentence and compare the representations of the perturbed sentences to that of the original. the three perturbations on an example span are illustrated in figure 1 . by applying perturbations to each span of words within the input sentence, we generate scores indicating the likelihood of each span being a constituent. to evaluate the effectiveness of our approach, we compare it with existing methods for extracting parse trees from plms without fine-tuning (section 4). our model improves over the previously published best result by a large margin. in a multilingual setting, our model surpasses the previous state of the art in 6 out of 8 languages. our model even outperforms some unsupervised parsing methods that require parameter updates, highlighting the effectiveness of our approach. our main contributions can be summarized as follows: instead of a generative model over sentences and trees, @xcite identified constituents based on span statistics. our method is relevant to the above in that constituents appear in constituent contexts. recent works have attempted to induce structural bias by constraining the flow of information in neural networks. examples include the parsing-reading-predict network (prpn) @xcite , the ordered neuron (on) model @xcite , and tree transformer @xcite . models with latent tree variables can also be seen as manipulating the information flow. the unsupervised recurrent neural network grammar (urnng) @xcite and the deep inside-outside recursive autoencoder (diora) @xcite optimized an autoencoder objective through latent tree variables. on the other hand, @xcite designed an unsupervised parser by specifying a set of transformations inspired by constituency tests and trained a classifier on the external raw text of 5 million sentences from english gigaword @xcite to make grammaticality decisions. our work builds upon this approach by utilizing constituency tests to obtain span scores. however, our method differs in that it is free from parameter updates and does not require the training of a grammaticality model on external data. ## conclusion in this work, we proposed a novel method for extracting constituency trees from masked lms without parameter updates. based on linguistic perturbations, we use the change in the contextual representation to reveal the constituency property of a span. through experiments on the english ptb and the multilingual spmrl dataset, we show that our method is robust and able to achieve state-ofthe-art performance across languages. notably, our method only requires a single hyperparameter, the layer index within the transformer architecture. our results indicate that our method is a simple yet effective approach to obtaining constituency trees, and future research includes exploring its application to broader plms beyond masked lms and the identification of other types of syntactic structures.