Construction Grammar Provides Unique Insight into Neural Language Models (2023)

Abstract: Construction Grammar (CxG) has recently been used as the basis for probing studies that have investigated the performance of large pretrained language models (PLMs) with respect to the structure and meaning of constructions. In this position paper, we make suggestions for the continuation and augmentation of this line of research. We look at probing methodology that was not designed with CxG in mind, as well as probing methodology that was designed for specific constructions. We analyse selected previous work in detail, and provide our view of the most important challenges and research questions that this promising new field faces.

## introduction in this paper, we will analyse existing literature investigating how well constructions and constructional information are represented in pretrained language models (plms). we provide context to support the argument that this is one of the most important challenges facing language models (lms) today, and provide a summary of the current open research questions and how they might be tackled. our paper is organised as follows: in section 2, we explain why lms must understand constructions to be good models of language and perform effectively on downstream tasks. in section 3, we analyse the existing literature on non-cxg-focused probing to determine its limitations in analysing constructional knowledge. in section 4, we summarise the existing probing work that is specific to cxg and analyse its data, methodology, and findings. in section 5, we argue that the development of an appropriate probing methodology for constructions remains an open and important research question ( §5.1), and highlight the need for data collection and annotation for facilitating this area of research ( §5.2). finally, in section 5.4, we suggest next steps that lms might take if cxg probing reveals fundamental problems. the funnier the example fixed the comparative phrase expressions being correlated the more citations the paper will have figure 1 : an example illustrating the complexity of a construction. it is an instance of the english comparative correlative (cc), with its syntactic features highlighted above the text and paraphrases illustrating its meaning below. ## motivation there has recently been growing interest in developing probing approaches for plms based on cxg. we see these approaches as coming from two different motivational standpoints, summarised below. ## established probing methods are only applicable to some aspects of cxg established probing methods have focused on different aspects of the syntactic and semantic knowledge of plms. in this section, we summarise the major approaches that were not designed specifically with constructions in mind. we show that although each of these methodologies deals with some aspect of cxg, and might even fully investigate some simpler constructions, none of them fully covers constructional knowledge as defined in section 1.1. ## cxg-specific probing we have argued that the most commonly used and straightforward probing methods are not sufficient for fully investigating constructional knowledge in plms. however, there have been several papers which have created new probing methodologies specifically for constructions. in this section, we will analyse them in terms of • which constructions were investigated? does the paper investigate specific constructions or does it use a pre-compiled list of constructions or restrain itself to a subset? paper language source construction example tayyar madabushi et al. (2020) english from automatically constructed list by dunn (2017) personal pronoun + didn't + v + how we didn't know how or why. li et al. (2022) english argument structure constructions according to bencini and goldberg (2000) caused-motion bob cut the bread into the pan. tseng et al. (2022) chinese from constructions list by (zhan, 2017) a + 到 + 爆, etc. 好吃到爆了！ it's so delicious! weissweiler et al. (2022) english mccawley (1988) comparative correlative the bigger, the better. for ease of reference, we provide an overview of the constructions investigated by each of the papers in ## research questions in this section, we lay out our view of the problems that are facing the emerging field of cxg-based probing and the reasons behind these challenges, and propose avenues for potential future work and improvement. ## conclusion we have motivated why probing large plms for cxg is a very important topic both for computational linguists interested in the ideal lm and for applied nlp scientists seeking to analyse and improve the current challenges that models are facing. we then summarised and analysed the existing literature on this topic. finally, we have given our reasons for why cxg probing remains a challenge, and detailed suggestions for further development in this field, within the realms of data, methodology, and fundamental research questions.