Revisiting the Practical Effectiveness of Constituency Parse Extraction from Pre-trained Language Models (2022)

Abstract: Constituency Parse Extraction from Pre-trained Language Models (CPE-PLM) is a recent paradigm that attempts to induce constituency parse trees relying only on the internal knowledge of pre-trained language models. While attractive in the perspective that similar to in-context learning, it does not require task-specific fine-tuning, the practical effectiveness of such an approach still remains unclear, except that it can function as a probe for investigating language models’ inner workings. In this work, we mathematically reformulate CPE-PLM and propose two advanced ensemble methods tailored for it, demonstrating that the new parsing paradigm can be competitive with common unsupervised parsers by introducing a set of heterogeneous PLMs combined using our techniques. Furthermore, we explore some scenarios where the trees generated by CPE-PLM are practically useful. Specifically, we show that CPE-PLM is more effective than typical supervised parsers in few-shot settings.

## introduction with the increasing interest in the inner workings of pre-trained language models (plms; @xcite ; @xcite ; @xcite ; @xcite ), foot_0 much work that attempts to explore the inherent knowledge embedded in the models has been recently proposed. one of the main topics in this direction is to reveal whether plms understand syntactic knowledge of human language, usually represented as parse trees. while a line of work @xcite ; @xcite ) has investigated the existence of syntax in plms via structural probes with supervision from gold-standard parse trees, some studies @xcite @xcite ) have found that one can extract reasonable parse structures directly from the patterns presented in plms' hidden representations or attention distributions even without extra fine-tuning. in other words, the studies have shown that plms implicitly store their understanding of syntactic knowledge in their parameters, and that such information can be easily reformulated into syntactic trees with almost no additional cost. although the aforementioned approach, dubbed constituency parse extraction from pre-trained language models (cpe-plm; @xcite ), is undoubtedly a useful tool with many analytic uses @xcite , it still remains a research question whether this algorithm can also work for practical purposes. for instance, as cpe-plm is free from fine-tuning of plms, it may be appealing in few-shot settings, akin to in-context learning for natural language understanding @xcite . moreover, there exists a potential that the new parsing paradigm can substitute the role of supervised or unsupervised parsers for the case where an nlp model requires a parse tree as input. in this work, we focus on revealing the practical effectiveness of cpe-plm. specifically, we first rewrite the procedure of cpe-plm in a more rigorous form to clarify its working mechanism. we then introduce two new ensemble algorithms tailored for it, i.e., greedy and beam, making its parsing performance competitive with that of unsupervised parsers @xcite . we show that it is crucial to combine syntactic clues from heterogeneous plms for achieving comparable performance, and that this trend holds in not only english but also multilingual cases. equipped with the improved variants of cpe-plm, as the next step, we investigate some scenarios in which their outputs (i.e., generated trees) can be practically utilized. we show that (1) it is viable to introduce the trees from cpe-plm as auxiliary data for improving recurrent neural network grammars (rnng; @xcite ; kim et al. (2019c) ), that (2) on classification with tree lstms @xcite , the induced trees can replace gold-standard parses with a minimal loss, and that (3) cpe-plm can be even more data-efficient than supervised parsers in few-shot settings. ## proposed methods in this section, we additionally introduce two novel ensemble methods for cpe-plm, which are more effective than the previous counterparts in improving parsing performance. furthermore, we propose to allow several plms to collaborate with each other to provide cpe-plm with more diverse syntactic information (figure 2 ). ## experiments on parsing in this chapter, we aim to validate the effectiveness of the proposed ensemble algorithms, i.e., greedy and beam, which enable cle-plm to have a higher potential of being properly adopted for downstream tasks. ## experiments on downstream tasks in the previous section, we demonstrated that cpe-plm's performance can be significantly improved by introducing the techniques proposed in this work. we now turn our attention towards its outputs (i.e., generated parse trees) and investigate the utility of such trees in two application scenarios where tree structures are taken as input. ## discussion so far, we have focused on verifying the utility of cpe-plm through the lens of (1) its improved parsing performance and (2) the effectiveness of its models f 1 w/ ensemble (↑) inference greedy beam time (↓) unsupervised parsers/cpe-plm compound pcfg (kim et al., 2019b) 55.2 31 min. cpe-plm (all plms) 55.3 55.7 27 min. issues on the execution time. as identified in ## conclusion in this paper, we introduce two ensemble methods and multi-plm configurations for constituency parse extraction from pre-trained language models (cpe-plm). we demonstrate that the performance of cpe-plm can be competitive with that of unsupervised parsers with the aid of the proposed approaches, and that the parses induced by cpe-plm are practically useful in several applications where parse trees are required as input. we also propose solutions for mitigating some inherent limitations of cpe-plm. we anticipate that its potential will be further greater in the near future with the introduction of more sophisticated plms.