Negation, Coordination, and Quantifiers in Contextualized Language Models (2022)

Abstract: With the success of contextualized language models, much research explores what these models really learn and in which cases they still fail. Most of this work focuses on specific NLP tasks and on the learning outcome. Little research has attempted to decouple the models’ weaknesses from specific tasks and focus on the embeddings per se and their mode of learning. In this paper, we take up this research opportunity: based on theoretical linguistic insights, we explore whether the semantic constraints of function words are learned and how the surrounding context impacts their embeddings. We create suitable datasets, provide new insights into the inner workings of LMs vis-a-vis function words and implement an assisting visual web interface for qualitative analysis.

## introduction recently, tremendous progress has been observed in the development of contextualized language models (lm). after the introduction of contextualized embeddings like elmo @xcite and bert @xcite , earlier static word embeddings @xcite have been sidelined, and new standards have been set for the state-of-the-art. particularly, lms have been shown to learn task-agnostic properties of language (e.g., @xcite @xcite and linguistic properties that imitate the classical nlp pipeline @xcite . despite this success, lms cannot be taken to understand language the way humans do, as they fail to generalize on unseen data and tasks requiring compositionality @xcite . research efforts concentrate on shedding light on what these models learn and consequently how they can be improved. one strand of work relies on probing/diagnostic tasks, where classifiers are * equal contribution. in this paper, we attempt to fill this gap by taking a closer look at function words, i.e., words with little lexical meaning. function words have been traditionally ignored in nlp, being dismissed as stopwords. since the rise of contextualized embeddings, function words are not dismissed anymore but do not receive any special treatment either; instead, they are contextually learned like any other word. however, the linguistically-motivated datasets created in this work and tested within the masked-language-modeling (mlm) task show that this context-based learning does not effectively capture the functionality of such words and, thus, that the weaknesses that lms show in semantic tasks can be attributed to these ill-learned representations. particularly, our work sheds light to the type and quality of masked-word predictions when negation, coordination, and quantifiers are involved. this work focuses on english masked lms, but the basic findings should be extendable to other languages since the training process of most masked lms is relatively similar. with this, the contributions of this work are fourfold. first, we show the semantic constraints of three of the most studied classes of function words -negation, coordination, and quantifiers. second, we provide linguistically-motivated datasets that capture the semantic constraints of these function words. based on these datasets, we offer new quantitative and qualitative insights into the treatment of function words in masked lms. the qualitative insights are facilitated by an openly-available web interface that visualizes lm predictions and allows researchers to easily test their hypotheses. ## related work most related work, i.e., probing studies and adversarial testing has focused on specific nlp tasks and on the learning outcome of the models. however, recently, research has also concentrated on exploring the learning procedure per se and understanding the lm inner workings as per the way of their training. with this, the study of contextualization has emerged. the embeddings are learned based on the co-occurrence of words in particular contexts. thus, the lm embeddings are contextualized, i.e., a word has different representations based on the context it is found in. particularly, @xcite shows that the embeddings become more contextualized, i.e., more context-specific, in the upper layers of bert, and that contextualization is not entirely driven by polysemy: (non-polysemous) stopwords such as @xcite to' also become increasingly contextualized in the upper layers, and their representations are among the most context-specific ones. rather, he finds that contextualization seems to be driven by the variety of contexts a word appears in. these findings are further explored by @xcite , who show that contextualization is neither driven by polysemy nor by pure context variation, but by the combination of functionality, sense variation, syntactic variation, and semantic context variation: bert can efficiently model polysemous, homonymous and monosemous words, and also words that appear in semantic contexts of high and low variation and independently of their polysemy. but it cannot model words that have a semi-functional/semi-content nature (e.g., modals, quantifiers, temporal adverbials). in this work, we take up this research direction and further explore what is exactly learned during the learning of function words and how their surrounding context influences their learned embeddings. ## experiments in this work we focus on the three functional categories discussed and set out to complement the existing research on the treatment of function words within contextualized lms. first, we create datasets that can be used to evaluate how lms have learned to capture the semantic constraints of these functional categories. second, we generate quantitative insights into the contextualization of function words, which we qualitatively investigate through a user-friendly web interface, which also allows researchers to evaluate their own datasets. ## results and discussion the results of our experiments can be found in tables 1 and 2 . ## conclusion this paper presented the extent to which lms learn semantic constraints of function words. based on theoretical linguistic literature, we created new datasets for testing three functional classes, and showed that popular masked lms make problematic predictions. the visualizations in the developed web interface highlighted potential reasons for this poor performance. in this work, we set out to shed light on a subset of function word categories. future work shall look into other function word categories that ought to be challenging for lms and uncover additional reasons for the poor model performances.