Multilingual Chart-based Constituency Parse Extraction from Pre-trained Language Models (2021)

Abstract: As it has been unveiled that pre-trained language models (PLMs) are to some extent capable of recognizing syntactic concepts in natural language, much effort has been made to develop a method for extracting complete (binary) parses from PLMs without training separate parsers. We improve upon this paradigm by proposing a novel chart-based method and an effective top-K ensemble technique. Moreover, we demonstrate that we can broaden the scope of application of the approach into multilingual settings. Specifically, we show that by applying our method on multilingual PLMs, it becomes possible to induce non-trivial parses for sentences from nine languages in an integrated and language-agnostic manner, attaining performance superior or comparable to that of unsupervised PCFGs. We also verify that our approach is robust to cross-lingual transfer. Finally, we provide analyses on the inner workings of our method. For instance, we discover universal attention heads which are consistently sensitive to syntactic information irrespective of the input language.

## introduction constituency parsing is a classic task in natural language processing (nlp), whose goal is to construct a phrase-structure tree for a given sentence. as parse trees have long been recognized as being integral to the meaning of sentences, there has been an enormous amount of work in the literature to develop constituency parsers @xcite ; @xcite ; @xcite ; inter alia), resulting in the development of sophisticated neural supervised parsers @xcite @xcite . although it becomes possible to obtain such neural parsers of high-quality for a few languages, there remain many other languages which, for lack of resources or attention, have yet to benefit from the progress made in the field of constituency parsing. the main issue is that it is expensive and time-consuming to prepare adequate numbers of gold-standard trees essential for training parsers with supervision. considering this data scarcity problem, unsupervised constituency parsing methods have naturally arisen as an alternative for generating constituency trees. work on unsupervised parsing @xcite ; @xcite ; inter alia) has focused on devising linguistically-informed neural models, which are carefully designed to be more sensitive to the hierarchical nature of language structure and be able to learn this nature from raw text rather than gold-standard trees. despite the recent progress in implementing english unsupervised parsers with decent performance, it has been known as not trivial to reproduce such a success in multilingual environments @xcite . moreover, as most unsupervised parsers are typically trained on monolingual text for a specific target language, it is required to prepare separate instances of them to support different languages. on the other hand, a different line of work @xcite has proposed a new direction of inducing syntax trees, dubbed in this paper as constituency parse extraction from pre-trained language models (cpe-plm), by relying on the combination of (i) simple distance metrics and (ii) the representations obtained from pre-trained language models (plms). 1 the core assumption underlying the methodology is that plms hold enough syntactic knowledge to be utilized for predicting parse trees by themselves. although the cpe-plm framework has demonstrated that non-trivial trees resembling gold-standard annotations can be extracted from general plms even without fine-tuning on treebanks, it has been also reported that cpe-plm's freedom from task-specific training comes at the cost of its performance inferior to that of unsupervised parsers. furthermore, as is the case of the foregoing unsupervised parsing literature, it has yet to be verified that cpe-plm is also effective for languages other than english. in this paper, we first attempt to narrow the performance gap between unsupervised parsers and cpe-plm by introducing a novel method inspired by neural chart-based algorithms @xcite @xcite . in contrast to the top-down cpe-plm method @xcite , which focuses on detecting the boundary of two subspans in a phrase relying only on the knowledge from the two words around the boundary, our chart-based method considers all components in a phrase to judge how plausible the phrase is. furthermore, we introduce a simple but effective ensemble technique that utilizes the pre-defined set of attention heads which are confirmed as being effective by their performance on the validation set. we show that our chart-based method outperforms or is competitive to the top-down method and that the top-k ensemble plays a key role in boosting their performance. second, the limitation of most previous studies for both unsupervised parsing and cpe-plm is that they are heavily english-centric, leaving an open question whether they are universally applicable. to investigate this problem, we test cpe-plm on several other languages. specifically, we propose to introduce multilingual plms @xcite into cpe-plm to grant the framework an ability to deal with multiple languages simultaneously. we show that the cpe-plm methods built upon multilingual plms are able to induce reasonable parses for sentences in nine languages in an integrated and languageagnostic manner, achieving figures superior or comparable to ones from neural pcfgs @xcite . in supplementary analyses, we provide intuitive explanations about the inner workings of our method. for instance, we confirm the existence of universal attention heads which seem to be responsible for capturing syntactic information irrespective of the input language. ## background in this work, we focus on a variant of unsupervised constituency parsing, which we call constituency ## method among various approaches that belong to cpe-plm, we regard the top-down method proposed by @xcite as a starting point and aim to improve the method in several perspectives. ## experiments 4.1 general configurations to evaluate, we prepare the ptb @xcite dataset for english and the spmrl @xcite dataset for the eight other languages: basque, french, german, hebrew, hungarian, korean, polish, and swedish. we use the standard split of each dataset, and the datasets are pre-processed following @xcite and @xcite -removing punctuation marks. we leverage the unlabelled sentence-level f1 (percentage) score as a primary metric to evaluate the extent to which tree predictions resemble corresponding gold-standard trees. the hyperparameter k, which determines the number of attention heads engaging in the top-k ensemble, is decided by grid search on some reasonable candidates ({5, 10, 20, 30}). we empirically found that @xmath0. from now on, we employ the abbreviations td, cp, and cc to refer to the top-down (section 3.1; kim et al. ( 2020 )), chart-pair (section 3.2.1; our approach with s p ), and chart-characteristic method (section 3.2.2; our method with s c ) respectively. ## analysis we present several analyses that enrich our understanding about cpe-plm. we employ xlm-r as a backbone and the cc method as a parsing scheme. ## limitations and future work we here mention a few limitations of our approach and propose avenues for future work. first, analogous to several unsupervised parsers @xcite including pcfgs @xcite , the current form of our method relies on a few gold-standard annotations from the validation set to determine the best hyperparameters (i.e., the best choice for attention head selection). this dependency makes it hard to say that our approach is entirely unsupervised, although it steps aside from a typical way of learning parsers with supervision. a next, promising yet challenging, step will be therefore to develop a remedy that enables our method to be free from the annotations, similar to @xcite . note that our cross-lingual transfer experiments also shed some light on how to relieve such dependency. while we have shown that cpe-plm can be superior or comparable to pcfgs and that it can function as an effective tool for analyzing plms, its performance still falls short of expectations in terms of whether it can practically substitute standard parsers, similar to the case of unsupervised parsers. to improve its performance, we have a plan as future work to design an ensemble method that gathers information from heterogeneous plms. finally, chart parsing algorithm, whose time complexity is o(n 3 ), is inherently much more expensive than other efficient parsing strategies such as top-down parsing. therefore, when we need to decide which parsing algorithm to employ, we should keep in mind the trade-off between accuracy and efficiency. ## related work pre-trained language models (plms) now lie at the heart of many studies in the literature. following the trend, much effort has been made to develop english plms ( devlin et al. (2019); liu et al. (2019); radford et al. (2019); yang et al. (2019) , inter alia), to construct non-english plms @xcite ; i.a.), and to train multilingual variants @xcite . we have explored the potential use of these plms as parsers. the trees induced by our method can also be leveraged as a tool for probing plms, similar to recent work that attempt to explore the knowledge in plms @xcite . in particular, @xcite to multilingual settings, analogous to our work. still, it is different from ours in that it requires explicit supervision and devotes itself to dependency grammar. in this study, we have named a line of work @xcite @xcite ) that extracts trees directly from plms as cpe-plm. we have extended its application to multilingual scenarios as well as improving its performance. notably, @xcite developed an approach similar to ours, but they focused on transformers trained for machine translation rather than language models. work on neural unsupervised parsing @xcite ; @xcite ; @xcite , inter alia) also seeks to generate parse trees without supervision from gold-standard trees. it is worth noting that some work such as @xcite and @xcite attempt to evaluate unsupervised parsers in multilingual settings, akin to our work. the difference between ours and theirs is that our method does not require training a parser for each language, instead relying on off-the-shelf plms. ## conclusion in this work, we study constituency parse extraction from pre-trained language models (cpe-plm), a novel paradigm of inducing parses directly from plms. we introduce a chart-based method and top-k ensemble for improving performance, and extend the range of application of the paradigm to different languages by applying multilingual plms. we hope our work can function as the foundation for future research on (i) unsupervised constituency parsing for under-represented languages and (ii) probing the inner workings of multilingual plms.