The better your Syntax, the better your Semantics? Probing Pretrained Language Models for the E nglish Comparative Correlative (2022)

Abstract: Construction Grammar (CxG) is a paradigm from cognitive linguistics emphasising the connection between syntax and semantics. Rather than rules that operate on lexical items, it posits constructions as the central building blocks of language, i.e., linguistic units of different granularity that combine syntax and semantics. As a first step towards assessing the compatibility of CxG with the syntactic and semantic knowledge demonstrated by state-of-the-art pretrained language models (PLMs), we present an investigation of their capability to classify and understand one of the most commonly studied constructions, the English comparative correlative (CC). We conduct experiments examining the classification accuracy of a syntactic probe on the one hand and the models’ behaviour in a semantic application task on the other, with BERT, RoBERTa, and DeBERTa as the example PLMs. Our results show that all three investigated PLMs are able to recognise the structure of the CC but fail to use its meaning. While human-like performance of PLMs on many NLP tasks has been alleged, this indicates that PLMs still suffer from substantial shortcomings in central domains of linguistic knowledge.

## introduction the sentence "the better your syntax, the better your semantics." contains a construction called the english comparative correlative (cc; @xcite . paraphrased, it could be read as "if your syntax is better, your semantics will also be better." humans reading this sentence are capable of doing two things: (i) recognising that two instances of "the" followed by an adjective/adverb in the comparative as well as a phrase of the given structure (i.e., the syntax of the cc) express a specific meaning (i.e., the semantics of the cc); (ii) understanding the semantic meaning conveyed by the cc, i.e., understanding that in a sentence of the given struc-ture, the second half is somehow correlated with the first. in this paper, we ask the following question: are pretrained language models (plms) able to achieve these two steps? this question is important for two reasons. firstly, we hope that recognising the cc and understanding its meaning is challenging for plms, helping to set the research agenda for further improvements. secondly, the cc is one of the most commonly studied constructions in construction grammar (cxg), a usage-based syntax paradigm from cognitive linguistics, thus providing an interesting alternative to the currently prevailing practice of analysing the syntactic capabilities of plms with theories from generative grammar (e.g., @xcite . we divide our investigation into two parts. in the first part, we examine the cc's syntactic properties and how they are represented by plms, with the objective to determine whether plms can recognise an instance of the cc. more specifically, we construct two syntactic probes with different properties: one is inspired by recent probing methodology (e.g., @xcite and draws upon minimal pairs to quantify the amount of information contained in each plm layer; for the other one, we write a context-free grammar (cfg) to construct approximate minimal pairs in which only the word order determines if the sentences are an instance of the cc or not. we find that starting from the third layer, all investigated plms are able to distinguish positive from negative instances of the cc. however, this method only covers one specific subtype of comparative sentences. to cover the full diversity of instances, we conduct an additional experiment for which we collect and manually label sentences from c4 @xcite that resemble instances of the cc, resulting in a diverse set of sentences that either are instances of the cc or resemble them closely without being instances of the cc. applying the same methodology to this set of sentences, we observe that all examined plms are still able to separate the examples very well. in the second part of the paper, we aim to determine if the plms are able to understand the meaning of the cc. we generate test scenarios in which a statement containing the cc is given to the plms, which they then have to apply in a zero-shot manner. as this way of testing plms is prone to a variety of biases, we introduce several mitigating methods in order to determine the full capability of the plms. we find that none of the plms we investigate perform above chance level, indicating that they are not able to understand and apply the cc in a measurable way in this context. we make three main contributions: ## syntax our investigation of plms' knowledge of the cc is split into two parts. first, we probe for the plms' knowledge of the syntactic aspects of the cc, to determine if they recognise its structure. then we devise a test of their understanding of its semantic aspects by investigating their ability to apply, in a given context, information conveyed by a cc. ## conclusion we have made a first step towards a thorough investigation of the compatibility of the paradigm of cxg and the syntactic and semantic capabilities exhibited by state-of-the-art large plms. for this, we chose the english comparative correlative, one of the most well-studied constructions, and investigated if large plms have learned it, both syntactically and semantically. we found that even though they are able to classify sentences as instances of the construction even in difficult circumstances, they do not seem to be able to extract the meaning it conveys and use it in context, indicating that while the syntactic aspect of the cc is captured in pretraining, the semantic aspect is not. we see this an indication that major future work will be needed to enable neural models to fully understand language to the same degree as humans.