C x GBERT : BERT meets Construction Grammar (2020)

Abstract: While lexico-semantic elements no doubt capture a large amount of linguistic information, it has been argued that they do not capture all information contained in text. This assumption is central to constructionist approaches to language which argue that language consists of constructions, learned pairings of a form and a function or meaning that are either frequent or have a meaning that cannot be predicted from its component parts. BERT’s training objectives give it access to a tremendous amount of lexico-semantic information, and while BERTology has shown that BERT captures certain important linguistic dimensions, there have been no studies exploring the extent to which BERT might have access to constructional information. In this work we design several probes and conduct extensive experiments to answer this question. Our results allow us to conclude that BERT does indeed have access to a significant amount of information, much of which linguists typically call constructional information. The impact of this observation is potentially far-reaching as it provides insights into what deep learning methods learn from text, while also showing that information contained in constructions is redundantly encoded in lexico-semantics.

## introduction and motivation the introduction of pre-trained contextual word embeddings has had a tremendous impact on natural language processing (nlp), resulting in significant improvements on several tasks such as translation and question answering, where the best automated systems are at par with or outperform humans @xcite . these models can broadly be classified based on the approach they use: the pre-training approach and the feature-based approach. while methods that rely on features, such as elmo @xcite require task specific architectures, those that rely on fine-tuning, such as generative pretrained transformer (openai gpt) @xcite , update all parameters during fine-tuning thus transferring their learning to new tasks with minimal fine-tuning. while not the first pre-trained model, bert @xcite was the first extremely successful one, utilising a deep bidirectional transformer model to provide significant improvements on the state-of-the-art of several tasks. the success of bert and its effectiveness in transfer learning, which is inherent to pre-trained models, could imply that these models have an understanding of the underlying language. this has led several researchers to focus on explaining what it is that these models understand about language. despite the inherent difficulty in understanding deep learning models, a new subfield of nlp, called bertology @xcite , has evolved to better understand what bert captures within its structure and how it is able to so effectively transfer that knowledge to so many different tasks through relatively quick fine-tuning. these efforts have shown that the bert embeddings capture linguistic information such as tense, parts of speech @xcite , entire parse trees @xcite in addition to encapsulating the nlp pipeline @xcite . bert is typically trained on two objectives: the masked language model (mlm) pre-training objective (which involves randomly masking some tokens and setting the objective to predict the original tokens), and next sentence prediction (nsp) objective (which consists of predicting if, given two sentences, one follows the other or not). these training objectives provide bert with access to a tremendous amount of lexico-semantic information. while lexico-semantic elements no doubt capture a large amount of linguistic information, it has been argued that they do not capture all information contained in text @xcite . this assumption is central to the constructionist approach which does not adhere to a strict division between lexical and grammatical elements, but rather argues that they form a continuum made up of constructions, which are learned pairings of form and function or meaning. constructions include partially or fully filled words, idioms or general linguistic patterns (e.g. idiom, partially filled: @xmath0. in order for a construction to be considered as such, its meaning must not be predictable from the sum of its parts (for example, in the case of idioms), or alternatively, if this meaning is predictable then a pattern is considered a construction if it occurs frequently @xcite . constructions occur at multiple levels of linguistic organisation (e.g., morphology, syntax) and at different levels of generalisation, starting with low level constructions, such as cat + -s (i.e., cats), progressing to more complex constructions such as noun + -s (the plural construction) or the xer the yer (e.g., the more i think about it, the less i like it), and finally the most abstract level which no longer contains any lexical elements such as the ditransitive construction (e.g. she gave me a book; i emailed her the details). speakers of a language are capable of making generalisations based on their encounter with only a few instances of a construction @xcite . the constructionist approach is based on theoretical principles developed through the study of language learners: it encompasses both the study of constructions (construction grammar: cxg, see section 2) and the idea that language learners construct their knowledge of language based on input (what people hear and read), subject to cognitive and practical constraints @xcite . while bertology has shown that bert captures certain important linguistic information, there have been no studies exploring the extent to which bert has access to constructional information. we hypothesise that should bert have access to constructional information, it should perform well on distinguishing constructions from each other. however, the ability to do so is not sufficient to conclude that bert has constructional information as we must fine-tune bert for it to perform this operation. additionally, bert could be using other information to distinguish constructions. if bert lacks constructional information, explicitly adding this information will significantly alter bert's ability to perform on downstream tasks or its ability to represent elements of language such as pos or parse trees, especially given the results of linguistic and cognitive linguistic experiments (sections 2 and 3.1). on the other hand, should bert already contain constructional information, the addition of this information would have no significant effect thus showing that bert does contain constructional information, or information that is functionally equivalent. thus, this work focuses on answering the following questions: answers to these questions will enable us to determine the extent to which bert has access to constructional information. it is crucial that we perform both these tests to rigorously test bert's knowledge of constructions as positive results for only one could be a result of some other variable. we make all of our code and experiment details including hyperparameters and pre-trained models publicly available in the interest of reproducibility and so others might experiment with these models. foot_0 2 construction grammar (cxg) construction grammars are a set of linguistic theories that take constructions as their primary object of study. a construction is a form-meaning pair (see figure 1 for an illustration and that is, it has a syntactic and phonological form which has a function or a meaning, albeit a (very) schematic one. constructions come in various shapes and sizes, from word forms to fully schematic argument structures. the common denominator is that either they are frequently occurring or that their meaning is not predictable from the sum of their parts. a famous example of this is the sentence she sneezed the foam off the cappuccino, which is an instance of the caused-motion construction. the verb sneeze on its own cannot be interpreted as a motion verb, nor is it usually used as a ditransitive verb, i.e. it does not normally take any complements. it is the caused-motion construction that activates or highlights the motion dimension of the verb "sneeze". other examples of the caused-motion construction include she pushed the plate out of the way or they moved their business to oklahoma. all these constructions share a similar syntactic pattern (form) and a meaning of caused motion. figure 1 : constructions and symbolic pairing of meaning and form. adapted from @xcite . rather than looking at words as individual tokens ordered in a sentence on the basis of syntactic rules, cxg assumes that syntax has meaning. according to cxg, linguistic knowledge is made up of chunks (i.e. constructions) such as the partially filled idiom "drive someone x", instances of which could be "drive someone crazy/nuts/bananas" etc. which is in turn an instance of the resultative construction "x makes y z" e.g sarah hammered the metal flat. cxg assumes that speakers are able to recognise a pattern after having encountered this pattern a certain number of times, with various lexical items. this is similar to merging n-grams (or chunks) through levels of schematicity or abstraction. it is assumed that speakers achieve this level of generalisation by abstracting over a number of similar instances. the examples given in she didn't understand how i could do so poorly. kiedis recalled of the situation: "he had such an outpouring of creativity while we were making that album that i think he really didn't know how to live life in tandem with that creativity." we didn't know how or why. one day she picked up a book and as she opened it, a white child took it away from her, saying she didn't know how to read. in a 1978 interview, dylan reflected on the period: "i didn't know how to record the way other people were recording, and i didn't want to. and it can be on my album, too, i just didn't realize how it worked. . . at first when i got this, people didn't know that i was an artist, so it was, like, 'oh, this songwriter bc.' for example, after encountering several sentences that are instances of the same construction as she put a finger on that, children associate the word 'put' with the pattern "x (she) causes y (finger) to move to z location (that)", even when the verb 'put' itself is not present, as in the sentence he done boots on. similarly, @xcite demonstrated that when people encounter words used in novel ways, they rely on constructions to decode their meanings. she crutched him the ball, for example, is interpreted to mean that she transferred the ball to him using the crutch, whereas she crutched him is interpreted as she hit him with a crutch. this phenomenon explains the human ability to generate and understand the infinite creative potential offered by language @xcite based on finite input. ## related work the study of how bert works, what it captures and what it is capable of -called bertology -has been gaining momentum since the introduction of bert @xcite . of particular relevance to us is the work associated with probing. probing is the use of a supervised model to establish if a particular encoding of a sentence contains certain information @xcite , such as say pos information @xcite or sentence length @xcite . it has been used to establish that bert seems to capture more detailed information such as the entire classical nlp pipeline @xcite and measure bert's ability to reason @xcite . probing techniques have also been used to understand the linguistic information captured by contextual representations as in the recent work by @xcite . they have also been used to establish that multilingual bert learns representations of syntactic dependency labels which largely agree with the universal dependencies taxonomy @xcite . of particular interest to this work is a probing technique called edge probing, which consists of tasks, inspired by traditional structured nlp, designed to evaluate how models encode sentence structure across syntactic, semantic, local and long-range phenomenon @xcite . some of this work has extended prior work that similarly investigated rnns, such as whether rnns can learn to predict subject-verb agreement @xcite . ## experimental set-up we design two sets of experiments to answer each of our questions: a) how does the addition of constructional information affect bert and, b) how effective is bert in identifying constructions? thus, the first set of experiments (described in section 4.1) are aimed at adding constructional information to bert, testing the resultant model using probing techniques and on downstream tasks and, the second set, described in section 4.2 consist of probing standard bert models using a variety of probes to test how effective bert is at identifying constructions. given the nature of our experiments, and the fact that we use non-standard datasets, we establish baselines for each of our tasks and perform an ablation study to better understand the relation between cxgs and bert. for use in both sets of experiments, we classify all sentences in the wikitext-103 corpus, which consists of a subset of verified "good" and "featured articles" on wikipedia @xcite into different "documents". this classification is based on the constructions each sentence is an instances of which is extracted using a modified version of dunn's (2017) work. their system provides a list of over 22 thousand constructions and we use this pre-calculated list of constructions to classify sentences. it should be noted that a single sentence can be classified as being an instance of several constructions. we modify their system to vastly speed it up so as to process all sentences from wikitext-103, which consists of 30,000 articles and approximately 4.6 million sentences. examples of some sentences instantiating one construction so processed are listed in this results in between 0 and over 50,000 sentences instantiating each of the constructions defined. we call the collection of "documents", each consisting of sentences from wikitext-103, that are instances of a particular construction "cxg wikitext". some sentences may not be classified as an instance of any construction so are discarded. both sets of experiments are designed based on the premise that there is significant linguistic information encoded in the knowledge that two sentences are instances of the same construction. as discussed in section 3.1, there is significant theoretical and applied linguistic evidence to support this claim. this reduces the problem of classifying sentences into over 22 thousand constructions into a binary classification problem of whether or not two sentences are instances of the same construction. ## empirical evaluation this section details the results of the experiments described in experimental set-up (section 4). we do not lowercase any of the input, either during evaluation, probing or fine-tuning and also use the upper case version of bert base when using pre-trained models. additionally, the results are the maximum of five runs to account for variation due to random initialisation except where otherwise stated. ## discussion in an attempt to determine to what extent bert has access to constructional information, we set out to answer two questions, the first of which was: how does the addition of constructional information affect bert? overall, our experiments show that the addition of constructional information has little impact on bert. the fact that neither cxgbert nor bert base clone comes out ahead shows that the information available to both models is similar. however, it is interesting that the linguistic information available to a generalisation engine (such as a neural network or person), regardless of whether the input is a set of wikipedia articles (as in the case of bert base clone) or sentences instantiating the same construction (as in the case of cxgbert), is very similar. one need only study the diverse set of sentences that can be instances of the same construction ( the second question we aimed to answer was: how effective is bert in identifying constructions? we find that the constructions that are most easily predicted by bert (using the same cxg prediction task) also tend to have a lower number of sentence instances. a manual inspection of these constructions revealed that these constructions (that have fewer sentences as instances), such as personal pronoun + didn't + v + how illustrated in those that have several sentences as instances, such as noun -s, which bert finds harder to predict, tend to be so general that they are less useful. the extremely high accuracy on the set of constructions that have less than 10,000 sentences as instances seems to suggest that bert has a substantial amount of information pertaining to semantically specific constructions and the fact that bert, when frozen, performs rather poorly seems to suggest that this information is contained within the internal layers of bert and not in its output. it is striking that these constructions can be predicted by bert with an 85% accuracy after training on just 500 training elements. this is particularly surprising given that there are over 21 thousand constructions which are contained in this set -so 500 training elements are not even enough to provide one sample from each construction. the combination of these results allows us to conclude that bert does indeed have access to a significant amount of constructional information. this information, as with other information such as pos information or parse trees, is not explicitly available in the output layer, but can be accessed from within the internal layers of bert. the impact of this observation is potentially far reaching as it not only further shows the capabilities of deep learning methods, but also shows that information that is typically called constructional can be learned form exposure to lexico-semantic information. this is expected given the redundancy inherent in language: words and constructions constrain each other mutually. we note that this work is limited by the constructions that are available to us. a manual inspection of the constructions seems to indicate that some constructions -even those that have few sentences as instances -seem to be relatively short, even if the sentences that contain them are themselves long. for example, one of the constructions extracted is modal + be + past-participle and sentences that are instances of this construction include a) these complexes, though their origins may be found as early as the 19 th century, snowballed considerably during the cold war. and b) when the counterrevolution became stronger moreno called the junta and, with support from castelli and paso, proposed that the enemy leaders should be shot as soon as they were captured instead of brought to trial. however, we also note that verifying if a set of sentences are instances of a higher level construction is cognitively demanding, and so it is possible that we were only able to identify the shorter, more obvious, patterns. this might be contributing to bert's high accuracy in predicting them, yet does not fully explain bert's ability to predict similarity amongst over 21 thousand constructions using just 500 training examples. ## conclusions and future work in this work, we set out to explore the extent to which bert might have access to constructional information by use of several probes and extensive experimentation. our results allow us to conclude that bert does indeed have access to a significant amount of information, much of which linguists typically call constructional information. we hope that this work will inspire greater interaction between neural network research and linguistics, as has been suggested before @xcite . an analysis of constructions that have several sentences as instances showed that they tend to contain generic labels (such as "preposition + his") and do not constrain the meaning of the construction or its slots much. it is doubtful whether such patterns would be considered constructions on a construction grammar approach, since these patterns have form, but lack a clear mapping of that form to a specific function or meaning. @xcite describes a method of filtering constructions that might address this issue and we believe it to be an interesting future direction of work. finally, these results also suggest that bert could be used by linguists for creating an inventory of constructions. there have so far been some attempts at creating such an inventory, usually called a constructicon @xcite , but the tools used for these endeavours do not match cxgbert's potential @xcite . for example, rather than a purely manual analysis, or a mere frequency split, one could use bert's ability to disambiguate a construction as indicative of how "informative" it is, yet another possible direction of future work.