Using Collostructional Analysis to evaluate BERT ’s representation of linguistic constructions (2023)

Abstract: Collostructional analysis is a technique devised to find correlations between particular words and linguistic constructions in order to analyse meaning associations of these constructions. Contrasting collostructional analysis results with output from BERT might provide insights into the way BERT represents the meaning of linguistic constructions. This study tests to what extent English BERT’s meaning representations correspond to known constructions from the linguistics literature by means of two tasks that we propose. Firstly, by predicting the words that can be used in open slots of constructions, the meaning associations of more lexicalized constructions can be observed. Secondly, by finding similar sequences using BERT’s output embeddings and manually reviewing the resulting sentences, we can observe whether instances of less lexicalized constructions are clustered together in semantic space. These two methods show that BERT represents constructional meaning to a certain extent, but does not separate instances of a construction from a near-synonymous construction that has a different form.

## introduction neural language models have proven to be immensely successful in processing and generating language -especially transformer-based models such as bert have displayed phenomenal results @xcite . however, there is a glaring lack of transparency in the linguistic representations that these models rely on. while previously, language would be parsed according to specific formalisms grounded in linguistic theory, recent approaches rely on models inferring structures in an unsupervised way. this impedes our understanding of what structures a model should learn to accurately carry out natural language processing tasks, and dissuades theoretical linguists from researching them @xcite . recently there has been an increased interest in opening the black box of these language models @xcite showing that they capture various aspects of syntax, semantics and morphology, but these efforts are not always grounded in specific theories. one line of theoretical research that lends itself to these efforts is construction grammar @xcite . grounded in language acquisition research, construction grammar claims that human language is structured into learned pairs of forms and meaning, called constructions, ranging from specific words to abstract grammatical patterns. as long as a linguistic pattern carries semantic meaning restricted exclusively to that pattern, it's considered a construction. these constructions are acquired by human learners based on distributional data @xcite . this theory has proven itself to be compatible with data-driven, quantitative and distributional approaches to language @xcite as we can view each construction as having distributional meaning @xcite . it is not obvious that transformer models would learn constructions at all. their tokenizers already have certain built-in assumptions about the nature of linguistic units that may differ from those of construction grammar. furthermore, bert's training objectives are aimed at learning lexical-semantic information for these tokens, rather than for (partially abstracted) combinations of tokens or categories of tokens. lastly, the bert training process of course differs in significant ways from human language learning @xcite . nevertheless, bert's prediction accuracy indicates that it represents plenty of contextual information about the ways in which a token can be used. this raises the question, do bert's contextual meaning representations correspond to cxg constructions? a key idea within construction grammar is that specific words or phrases are more closely attracted to certain constructions than others, based on mean-ing associations. the method of collostructional analysis @xcite is used to quantitatively study such associations. such associations might be learned by bert models as well, leading to clusters that we might call constructions. an evaluation of bert output contrasted with a reproduced collostructional analysis results could reveal evidence of whether bert represents meaning beyond tokens in this way. this in turn will lead to more linguistically plausible intrinsic embedding evaluation and increased model transparency. we propose two tasks that are aimed at detecting meaning associations of specific constructions in an english-language bert model. firstly, by predicting the words within the open slots of constructions, we aim to evaluate to what extent bert represents the meaning of constructions. such associations between words and constructions are indicative of a construction's meaning according to the distributional hypothesis of meaning and studying them "provides an objective approach to identifying the meaning of a grammatical construction" (gries and @xcite . secondly, by finding similar sentences using bert's output embeddings, we evaluate whether those output embeddings contain the information necessary to distinguish constructions by both form and meaning. to ground our approach in the linguistic literature, we evaluate our results by comparing them to our reproduction of the results of gries and @xcite , whose method involves selecting exemplars based on part-of-speech annotation and manual filtering and can thus serve as a gold standard. ## construction grammar construction grammar (cxg), stemming from cognitive linguistics, deems constructions to be the fundamental components of written and spoken human language, and claims that the knowledge of an individual regarding language is solely defined by a complex set of constructions, labeled the constructicon @xcite . these constructions are form-meaning pairs, specifically, c is a construction iff def c is a form-meaning @xmath0, s @xmath1. @xcite so, rather than separating syntax and semantics like many other linguistic theories, cxg inextrica-bly links them. a construction is only a construction when it is impossible to deduce the form or semantic meaning entirely from the other elements of the construction. constructions range from very abstract to very specific -the past tense construction, for example is a highly abstract construction into which any verb can enter, while words are fully lexical, specific constructions. we discuss two constructions in more detail, and they are the constructions we will test bert for. we chose to analyse these two english-language constructions because they were among the constructions on which the method of collostructional analysis was first demonstrated (gries and stefanowitsch, 2003), allowing us to compare our results to established results from the literature. furthermore, they have been thoroughly studied from various theoretical perspectives. the two greatly differ in their level of abstractness, providing different methodological challenges. ## related work prior research seems to support the idea that bert learns some underlying structure beyond lexical semantics @xcite . such a structure might be construction grammar. however, the combination of construction grammar and large language models has not received much attention. @xcite present an overview of the use of neural networks to test usage-based theories of language acquisition, which sometimes use construction grammar representations. @xcite apply bert embeddings to study constructional change in the be-about futurate construction. a few works directly address construction grammar in bert. @xcite investigated whether bert is able to tell if two sentences contain the same construction. they took over 22000 different automatically identified constructions and let bert classify whether examples of them contain the same construction, obtaining 94% accuracy. however, due to automatic construction identification, their dataset may have contained many patterns that would not be considered constructions in cxg, and no comparison to results of linguistic analysis is carried out. tseng et al. (2022) two recent studies do focus on closely analysing specific construction types in bert. @xcite syntactically probe several bertbased models for the english comparative correlative construction in a cxg framework using minimal pair exemplars. they also perform an inferencing task and find that the models are not able to make inferences based on the meaning of the construction in a zero-shot setup. however, this may be due to the models' inadequate performance on inferencing tasks and logic in general rather than an inadequate intrinsic semantic representation. @xcite study argument structure constructions in transformer lms, including the ditransitive. in a sentence sorting task, they find that sentence embeddings of these constructions are clustered by their construction (ditransitive, resultative, caused-motion or removal) rather than by their verb in embedding spaces of several bertbased models. in a random word experiment, they fill the construction's slot with random real words and test whether the resulting context embeddings are close to averaged context embeddings of a verb prototypical of that construction. in this work we take the opposite approach, following gries and @xcite linguistic analysis method, focusing on what meanings appear in the slots to reveal the meaning of a construction. these studies conclude that constructions are or are not represented in bert models to varying degrees. we contribute novel evidence to this debate by 1) direct comparison to previous corpusbased linguistic analyses by @xcite of two specific constructions, 2) taking the content of open slots as indicative of the presence of constructional meaning in the embedding space using two different tasks and 3) investigating near-synonymous constructions to decide whether, if constructions are represented, they are clustered just by meaning or also by form. ## method to examine a bert model for the presence of constructional information and to perform collostructional analysis on these constructions we first need to obtain a set of representative exemplars of both constructions. to have a valid comparison with gries and @xcite , we use the british national corpus (bnc, bnc consortium, 2007) which is the corpus they derived their results from. ## discussion in construction grammar, constructions are defined as pairs of form and meaning. to claim that bert represents a construction, we need more than to find its constructional meaning in the model, as @xcite have done. it should be distinguishing form as well as meaning in its representations. as for meaning, our masked language modeling task showed that bert accurately represents the meaning of a highly lexicalized construction, matching a previous analysis from the linguistics literature. all eleven resulting words are nouns, and are felicitous in the x waiting-to-happen con-struction. accident and disaster are attributed high probability, and gries and @xcite collostructional analysis clearly shows that these are the strongest ranking collexemes of the construction. this is in line with @xcite finding that bert is better at identifying semantically specific constructions. however, this method cannot be used to study more abstract constructions or constructions where similar meanings may be expressed in different forms. our sentence embedding querying task showed that sentence context vectors derived from bert output embeddings yield both the target construction and its near-synonymous variant, providing an approximation of at least the meaning of the construction. this somewhat goes against @xcite conclusion that constructional information is not explicitly available in the output layers and should be brought out by tuning on a construction identification task. based on verb semantics, seventeen of the twenty nearest neighbour sentences to our prompt match the basic sense of the ditransitive or its extensions, with the remaining three only matching in terms of the lexical semantics of the verb(s). this similarly shows that bert clusters together sequences that signify a transfer in semantic space, though not perfectly. this leaves us with the question of whether constructional information in the embedding space is also separated by form. it is interesting that the near-synonymous, but distinct in form, prepositional dative construction was found only thrice using our double object construction prompt, compared to 14 docs. however, this appears to be an input frequency effect: sánchez (2018) found that in the british national corpus, of the 17081 instances of the dative alternation they identified through automated querying, 13921 (81.5%) are docs as opposed to pds, and in our top 20 result the proportion is 82%. this provides evidence that both of the near-synonymous ditransitive constructions participating in the dative alternation are clustered together in sbert embedding space. this means we found no evidence that bert clusters by constructional form when meaning is similar. the fact that the similarity search provided multiple ditransitive sentences with verbs not present in the input indicates that bert generalizes over instances of a construction. however, based on our evidence, we cannot claim that bert represents cxg constructions, defined as pairs of form and meaning, because the model does not fully distinguish constructions with similar meaning but different form. there are some caveats with this method. there is no evident explanation of why exactly some sentences are found to be similar. it is impossible to deduce from bert's output embeddings why phrases such as i said take the bus and he sent me his work are considered to be similar to each other. the two sentences clearly contain different constructions, and both semantically and syntactically they do not appear to be that similar besides both containing a verb of transfer. this black box problem extends to the waiting-to-happen construction. the predictions for the open slot are consistently nouns that fit the input exemplars of the construction used in the experiment but why bert predicts these exact nouns is unclear, as links to features of the training data are not preserved by transformer models. we also cannot fully exclude the influence of the sentence contexts that surround a construction, even when averaging the output distribution for multiple input sequences. ## conclusion overall, we have shown that the two methods we propose can retrieve substantial amounts of interpretable constructional information from bert. unlike previous work, taking the content of open slots as indicative of constructional meaning enabled us to make comparisons between the position in semantic space of embeddings potentially representing the construction, and supervised data from the linguistics literature. it appears that bert represents more lexicalized constructions better than more abstract constructions, and that bert output embeddings do contain constructional information. however, to claim that bert contains representations of cxg constructions it would be necessary to find evidence for separation in contextual embedding space of near-synonymous constructions. this was not tested in previous work and we did not find it in this study. in future work, it would be interesting to perform a contrastive analysis of two alternating constructions in sbert embedding space, while controlling for context, to further investigate the question of form. it would also be useful to compare methods for obtaining sentence context vectors for a construction that might target the construction more accurately as evaluated against traditional collostructional analyses. some further analyses might be insightful, such as viewing all instances of a construction as a cluster in embedding space and analysing the cluster coherence or the properties of its outliers. we could consider averaging vectors of larger or smaller parts of sentences containing the construction, or with more variation in terms of sentence lengths, lexical diversity or other factors that might affect the use of the target construction. it would also be interesting to scale up the analysis to larger corpora, although the results would be less comparable to those from the linguistic literature. it would also be interesting to see where bert positions ungrammatical exemplars, which of course did not occur in previous corpus studies. the methods we propose might be of interest to linguists, for example as a less transparent but more data-driven way of obtaining construction exemplars or studying collexemes, requiring less annotation effort. lastly, we hope to have shown that construction grammar can serve as a framework for studying meaning representations beyond the token in large language models, even for those not interested in construction grammar specifically.