Evaluating language models for the retrieval and categorization of lexical collocations (2021)

Abstract: Lexical collocations are idiosyncratic combinations of two syntactically bound lexical items (e.g., “heavy rain” or “take a step”). Understanding their degree of compositionality and idiosyncrasy, as well their underlying semantics, is crucial for language learners, lexicographers and downstream NLP applications. In this paper, we perform an exhaustive analysis of current language models for collocation understanding. We first construct a dataset of apparitions of lexical collocations in context, categorized into 17 representative semantic categories. Then, we perform two experiments: (1) unsupervised collocate retrieval using BERT, and (2) supervised collocation classification in context. We find that most models perform well in distinguishing light verb constructions, especially if the collocation’s first argument acts as subject, but often fail to distinguish, first, different syntactic structures within the same semantic category, and second, fine-grained semantic categories which restrict the use of small sets of valid collocates for a given base.

## introduction language models (lms) such as bert @xcite , and its variants spanbert @xcite , albert @xcite , roberta @xcite , etc. have proven extremely flexible, as they behave as unsupervised multitask learners @xcite , and can be leveraged in a wide array of nlp tasks almost out-of-thebox (see, e.g., the glue and superglue results in @xcite and @xcite , respectively). they have also been harnessed as supporting resources for knowledge-based nlp @xcite , as they capture a wealth of linguistic phenomena @xcite . recently, a great deal of research analyzed the degree to which they encode, e.g., morphological (edmiston, 2020), syntactic @xcite , or lexico-semantic structures @xcite . however, less work explored so far how lms interpret phraseological units at various degrees of compositionality. this is crucial for understanding the suitability of different text representations (e.g., static vs contextualized word embeddings) for encoding different types of multiword expressions @xcite , which, in turn, can be useful for extracting latent world or commonsense information @xcite . one central type of phraselogical units are lexical collocations, defined as restricted cooccurrences of two syntactically bound lexical items @xcite , such that one of the items (the base) conditions the selection of the other item (the collocate) to express a specific meaning. for instance, the base lecture conditions the collocates give or deliver to express the meaning 'perform', the base applause conditions the selection of the collocate thunderous to express the meaning 'intense', and so on. lexical collocations are of high relevance to lexicography, nlp and second language learning alike, and constitute a challenge for computational models because of their heterogeneity in terms of idiosyncrasy and degree of semantic composition @xcite . in this paper, we analyze a suite of lms in the context of two tasks that involve lexical collocation modeling. first, unsupervised collocate retrieval, where we mask a collocation's collocate (e.g., "heavy" in "heavy rain"), and quantify how well a lm of choice (bert in particular) predicts, via its masked language modeling (mlm) objective, a valid collocate for that particular base ({"heavy", "torrential", "violent", . . . } for the base "rain" and the meaning intense). second, su-pervised in-context collocation categorization, where we fine-tune lms on the task of predicting a semantic category of a collocation in terms of its lexical function (lf), given its sentential context; cf. section 3.1. modeling, recognizing, and classifying collocations in corpora has obvious applications for automatically creating and expanding lexicographic resources, as well as for various downstream nlp applications, among them, e.g., machine translation @xcite , word sense disambiguation @xcite , or natural language generation @xcite . the two main contributions of this paper thus are: our results suggest that modeling collocations in context is a challenge, even for widely used lms, and that this is particularly true for less semantic (and thus less compositional and more idiosyncratic) collocations. we also find that jointly recognizing the semantics and the syntactic structure (e.g., whether the collocate acts as subject or object in verbal constructions) of a collocation also constitutes non-trivial challenges for current architectures. moreover, as a byproduct of our analysis, we also find an interesting behaviour in lms when modeling antonymy in adjectives, specifically that their representations undergo substantial transformations as they flow through bert's transformer layers, with many contextualized embeddings clustered together in the tip of a narrow cone that seems to represent adjectives in collocations denoting intensity ("heavy" rain) and weakness ("minor" issue). ## related work in this section, we discuss related works in two methodological areas that are relevant to this paper, namely conditioning mlms (section 2.1) and recognition of multiword epressions (mwes) (section 2.2). ## experiment 2: collocation categorization in the second experiment, we test the performance of a number of well-known lms for the task of lf categorization using the train/test splits we sampled and annotated from gigaword (section 3). this experiment serves two purposes. first, we expect to learn about the predictability of lfs in context, which is a long-standing problem in computational lexicography and the cornerstone of automatic construction of collocation resources. second, previous work has shown that some lfs are quite easy to distinguish, without @xcite and with sentential context @xcite . however, it is still unclear whether, by focusing exclusively on the phenomenon of collocations, and excluding, e.g., idiomatic expressions or non-compositional phrasal verbs (which are not only semantically but, more importantly, syntactically different from collocations), an lm can indeed be used to construct a resource for second language learners, or whether (and to what extent) an lm can be trained to select appropriate collocates. our setting is essentially a sentence-pair classification problem, where the second sentence is the lexical collocation itself. specifically, a training instance is a @xmath0, collocation, label>, as in example (3) (where we use ; as a wildcard concatenation token, as these are different across lms): we use as labels the lfs listed in ## conclusions we have analyzed lms in tasks revolving around modeling, recognizing and categorizing lexical collocations. we conclude that some prominet types of lvcs require little context to be well encoded, as opposed to other lfs involving, e.g., nouns and adjectives, and that predictability of lfs is challenging, not a function of training data, and that syntax plays a major role. in the future, we will make this work multilingual using linguistic equivalences as anchors, in the spirit of cross-lingual embedding research, in order to align collocations of the same lf across languages (e.g., in english and norwegian we take a nap, in german, we 'make' it, in portuguese we 'pull' it, in spanish, we 'throw' it, etc.). we would also like to explore the idea of "semantic masking" for collocate discovery, where we would train models for dynamically masking (or removing) idiosyncratic information such that only the semantics of the collocate remain, thus largely corresponding to a latent abstraction over the lf. this approach has been applied recently in the lexical substitution task, with the limitation, however, that the dropout rate was tuned in a validation set, whereas a promising avenue to explore would be to automatically learn the embedding dropout in a fully supervised setting. finally, motivated by the observed large gap in performance between the categorization of, e.g., oper1 and oper2, bon and antibon, ver and antiver, we plan to investigate in more depth the codification of collocational information in pretrained lms.