Heads-up! Unsupervised Constituency Parsing via Self-Attention Heads (2020)

Abstract: Transformer-based pre-trained language models (PLMs) have dramatically improved the state of the art in NLP across many tasks. This has led to substantial interest in analyzing the syntactic knowledge PLMs learn. Previous approaches to this question have been limited, mostly using test suites or probes. Here, we propose a novel fully unsupervised parsing approach that extracts constituency trees from PLM attention heads. We rank transformer attention heads based on their inherent properties, and create an ensemble of high-ranking heads to produce the final tree. Our method is adaptable to low-resource languages, as it does not rely on development sets, which can be expensive to annotate. Our experiments show that the proposed method often outperform existing approaches if there is no development set present. Our unsupervised parser can also be used as a tool to analyze the grammars PLMs learn implicitly. For this, we use the parse trees induced by our method to train a neural PCFG and compare it to a grammar derived from a human-annotated treebank.

## introduction pre-trained language models (plms), particularly bert @xcite and others @xcite @xcite based on the transformer architecture @xcite , have dramatically improved the state of the art in nlp. such models make it possible to train a large, generic language model on vast unannotated datasets, and then fine-tune it for a specific task using a small amount of annotated data. the success of plms has led to a large literature investigating the linguistic knowledge that plms learn implicitly during pre-training @xcite @xcite , sometimes referred to as bertology @xcite . bertology has been particularly concerned with the question whether bert-type models learn syntactic structure. typical approaches include test suites of sentences that instantiate specific syntactic structures @xcite , general probes (also known as diagnostic classifiers, belinkov and glass 2019) or structural probes @xcite . all of these approaches are limited: the first one requires the laborious compilation of languageand construction-specific suites of sentences; the second one sometimes fails to adequately reflect differences in representations @xcite @xcite ; the third one involves designing a novel extraction model that is not applicable to tasks other than probing @xcite . it is therefore natural to use a parsing task to test whether the representations learned by plms contain usable syntactic information. this enables us to test syntactic structure in general, rather than specific constructions, and doesn't require a specialized probe. in this paper, we will therefore use plm attention heads to construct an unsupervised constituency parser. previously, related approaches have been proposed under the heading of zero-shot constituency parsing @xcite ). 1 however, this prior work crucially relies on an annotated development set in order to identify transformer heads that are sensitive to syntactic structure. existing approaches therefore are not truly unsupervised. for most low resource languages, no such annotated data is available, and often not even an annotation scheme exists. thus, assuming a development set is not a realistic experimental setup @xcite . if a suitable development set is available, @xcite shows that an existing supervised parser trained on a few-shot setting can outperform all the unsupervised parsing methods by a significant margin. it strongly challenges tuning on an annotated development set for unsupervised parsing. in this paper, we propose a novel approach to build a plm-based unsupervised parser that does not require a development set: we rank transformer heads based on their inherent properties, such as how likely tokens are to be grouped in a hierarchical structure. we then ensemble the top-k heads to produce constituency trees. we evaluate our approach and previous zero-shot approaches on the english penn treebank (ptb) and eight other languages on the spmrl dataset. on the one hand, if the development set is absent, our approach largely outperforms previous zero-shot approaches on the english ptb. on the other hand, if previous zero-shot approaches are equipped with the development set, our approach can still match the parsing performance of these approaches using the single best head or layerwise ensembling. for the multilingual experiment, we take advantage of the top-k heads selected in english and directly parse other languages using our approach. surprisingly, on five out of nine languages, this crosslingual unsupervised parser matches previous approaches that rely on a development set in each target language with the single best head or layer-wise ensembling. however, our fully unsupervised method lags behind the previous state-of-the-art zero-shot parser if a top-k ensemble is used. furthermore, our approach can be use as a tool to analyze the capability of plms in learning syntactic knowledge. as no human annotation is required, our approach has the potential to reveal the grammar plms learn implicitly. here, we use the tree structures generated by our parser to train a neural pcfg. we evaluate the learned grammar against the english ptb on internal tags and production rules both qualitatively and quantitatively. ## related work recently, neural models have renewed interest in grammar induction. earlier work @xcite attempted to induce grammar by optimizing a sentence classification objective, while follow-up work @xcite @xcite showed that a language modeling objective performs better. latest work employed autoencoders or probabilistic grammars @xcite @xcite . a new line of work is zero-shot constituency parsing, whose goal is to automatically extract trees from plms in a parameter-free fashion. the top-down zero-shot parser @xcite utilizes the concept of syntactic distance @xcite , where trees are induced by an algorithm that recursively splits a sequence of words in a topdown manner. however, this approach suffers from its greedy search mode, failing to take into account all possible subtrees. the chart-based zero-shot parser @xcite applies chart parsing to address this problem. wu et al. (2020 ## zero-shot constituency parsing via plms in this section, we briefly review the chart-based zero-shot parser and then introduce our rankingbased zero-shot parser. ## grammar learning we are also interested in exploring to what extent the syntactic knowledge acquired by plms resembles human-annotated constituency grammars. for this exploration, we infer a constituency grammar, in the form of probabilistic production rules, from the trees induced from plms. this grammar can then be analyzed further, and compared to humanderived grammars. thanks to the recent progress in neural parameterization, neural pcfgs have been successfully applied to unsupervised constituency parsing @xcite . we harness this model foot_5 to learn probabilistic constituency grammars from plms by maximizing the joint likelihood of sentences and parse trees induced from plms. in the following, we first briefly review the neural pcfg and then introduce our training algorithm. ## experiments we conduct experiments to evaluate the unsupervised parsing performance of our ranking-based zero-shot parser on english and eight other languages (basque, french, german, hebrew, hungarian, korean, polish, swedish). for the grammars learned from the induced parse trees, we perform qualitative and quantitative analysis on how the learned grammars resemble the human-crafted grammar of the english ptb. ## conclusion in this paper, we set out to analyze the syntactic knowledge learned by transformer-based pretrained language models. in contrast to previous work relying on test suites and probes, we proposed to use a zero-shot unsupervised parsing approach. this approach is able to parse sentences by ranking the attention heads of the plm and ensembling them. our approach is able to completely do away with a development set annotated with syntactic structures, which makes it ideal in a strictly unsupervised setting, e.g., for low resource languages. we evaluated our method against previous methods on nine languages. when development sets are available for previous methods, our method can match them or produce competitive results if they use the top single head or layer-wise ensembling of attention heads, but lags behind them if they ensemble the top-k heads. furthermore, we present an analysis of the grammars learned by our approach: we use the induced trees to train a neural pcfg and evaluate the pre-terminal and non-terminal symbols of that grammar. in future work, we will develop further methods for analyzing the resulting grammar rules. another avenue for follow-up research is to use our method to determine how the syntactic structures inherent in plms change when these models are fine-tuned on a specific task.