Multilingual Extraction and Categorization of Lexical Collocations with Graph-aware Transformers (2022)

Abstract: Recognizing and categorizing lexical collocations in context is useful for language learning, dictionary compilation and downstream NLP. However, it is a challenging task due to the varying degrees of frozenness lexical collocations exhibit. In this paper, we put forward a sequence tagging BERT-based model enhanced with a graph-aware transformer architecture, which we evaluate on the task of collocation recognition in context. Our results suggest that explicitly encoding syntactic dependencies in the model architecture is helpful, and provide insights on differences in collocation typification in English, Spanish and French.

## introduction native speech is idiosyncratic. of special prominence are syntactically-bound restricted binary cooccurrences of lexical items, in which one of the items conditions the selection of the other item. consider a cnn sports headline from 02/15/2021: rafael nadal eases into australian open quarterfinals, remains on course for record-breaking grand slam (cnn.com). in this short headline, we see already three of such co-occurrences: ease [into] quarterfinals, remain [on] course, and record-breaking grand slam. quarterfinals conditions the selection of [to] ease [into], course of remain @xcite , and grand slam of record-breaking. the idiosyncrasy of these cooccurrences becomes obvious when we look at them from a multilingual angle. thus, in french, instead of the literal translation of ease [into], we would use se qualifier 'qualify @xcite ', in spanish, remain [on] will be translated as seguir @xcite 'continue in', and in italian record-breaking will be da record, lit. 'of record' -while the translation of quarterfinals, course, and grand slam will be literal. in lexicology, such binary co-occurrences are referred to as collocations @xcite @xcite , with the conditioning item called the base and the conditioned item the collocate. collocations in this sense are of high relevance to second language learning, lexicography and nlp alike, and constitute a challenge for computational models because of their heterogeneity in terms of idiosyncrasy and degree of semantic composition @xcite . research in nlp has already addressed a number of collocation-related tasks, in particular: (1) collocation error detection, categorization, and correction in writings of second language learners @xcite @xcite ; (2) creation of collocation-enriched lexical resources @xcite @xcite ; (3) use of knowledge on collocations in downstream nlp tasks, among them, e.g., machine translation @xcite , word sense disambiguation @xcite , natural language generation @xcite , or semantic role labeling @xcite ; (4) probes involving collocations for understanding to which extent language models are able to identify non-compositional meanings @xcite ; and (5) detection and categorization of collocations with respect to their semantics @xcite @xcite . it is this last task which is the focus of this paper. in general, collocation identification and categorization tend to be treated as two disjoint tasks. most of the research deals only with collocation identification @xcite @xcite @xcite . some works deal with the categorization of manually precompiled lists of collocations, either in isolation @xcite @xcite or with their original sentence-level contextual information @xcite . only a few works in the early phase of the neural network era of nlp address the problem of collocation identification and semantic categorization as a joint task in monolingual settings @xcite . accordingly, the performance of the models put forward in these works is still rather low. in this paper, we propose a sequence tagging framework for simultaneous collocation identification and categorization, with respect to the taxonomy of lexical functions (lfs) @xcite . the proposed framework is based on mono-and multilingual bert-based sequence taggers, which are enhanced by a graphaware transformer @xcite in order to ensure that the specific syntactic dependencies between the base and the collocate are taken into account. the sequence taggers are executed as part of a multitask learning setup, which is complemented by a sentence classification task, which predicts the occurrence of an instance of a specific lf instance in the sentence under consideration. our results for english, french and spanish show the flexibility of our framework and shed light on the multilingual idiosyncrasies of collocations. ## background on collocations although widely used in lexicology in the sense defined above, the term collocation is ambiguous in linguistics. as introduced by @xcite , it refers to common word co-occurrences in discourse in general. thus, cast and vote, strong and tea, but also public and sector, night and porter, supermarket and price form collocations in english in firth's sense. in computational linguistics, firth's definition has been taken up, e.g., by @xcite @xcite @xcite @xcite . to avoid confusion between the two different senses, @xcite proposed to use the narrower term lexical collocation to refer to restricted binary lexical item co-occurrences. in what follows, we will use this term to refer to the definition underlying our work. lexical collocations can be typified with respect to the meaning of the collocate and the syntactic structure formed by the base and the collocate. practical collocations dictionaries such as, e.g., the oxford collocations dictionary foot_1 or the mcmillan collocations dictionary foot_2 , already offer a coarsegrained semantic typification. however, their typification still does not make a distinction between, e.g., control and cut in co-occurrence with expenditure or between cavernous and palatial in cooccurrence with room -distinctions which are essential in the context of both second language learning and nlp. to the best of our knowledge, lexical functions (lfs) @xcite are the most fine-grained taxonomy of lexical collocations. a lexical function (lf) is defined as a function f (b) that delivers for a base b a set of synonymous collocates that express the meaning of f . lfs are assigned latin abbreviations as labels; cf., e.g., "oper1" ("operare" 'perform'): oper1(condolences) = {convey, express, extend}; "magn" ("magnum" 'big'/'intense'): magn(grief) = {deep, inconsolable, great, . . . }. each lf can also be considered as a specific lexico-semantic relation between the base and the collocate of a collocation in question @xcite . as seen in ## related work previous works that consider collocations in a firthian sense look at word adjacency in terms of n-grams @xcite , although most often, statistical measures of co-occurrence are used; cf. pearce et al. (2002); pecina and schlesinger (2006); pecina (2010); garcia et al. (2019) . some complement statistical measures by morphological @xcite and/or syntactic @xcite @xcite patterns. in view of the asymmetrical nature of the relation between the base and the collocate, e.g., gries (2013) proposes to investigate "directional measures" as an addition to association measures; @xcite explicitly encode this asymmetry in terms of npmi @xcite , which is a normalized version of pmi; see also @xcite . in the collocation classification task, substantial research focused on the identification of light verb constructions, which are captured by the oper-(and partially by the real-) families of lfs; cf., e.g., @xcite @xcite @xcite more recently, word embeddings have been successfully applied in unsupervised setups, e.g., rodríguez fernández et al. (2016a) use simple vector arithmetic. in supervised setups, we find, first, the "collocate retrieval" approach proposed by @xcite , who train a linear transformation to go from a "base" to a "collocate" vector space, exploiting regularities in multilingual word embeddings @xcite , and second, espinosa anke et al. (2019), who train an svm on a dedicated relation vector space for base and collocate. embeddings have also been used in multilingual english/spanish (rodríguez fernández et al., 2016b) and english/portuguese/spanish he played the piano. udpipe parser cls he played the piano obj root nsubj . punct det ... ... sentence classifier bio sequence tagging graph-to-collocation transformer cls he played the piano . sep 1 2 3 4 5 6 7 -pron verb det noun punct - ## graph-to-collocation transformer we propose a graph-to-collocation transformer (g2c-tr) to: (1) cast collocation identification and classification as a sequence tagging problem: as pointed out above, lexical collocations are lexicosemantic relations, and relation extraction has been recently successfully addressed as sequence tagging @xcite ; (2) boost performance by enabling multitask learning via joint sentence classification and lf-instance bio tagging; and (3) capture the asymmetric semantic and syntactic dependency between the base and the collocate by the use of a modified attention mechanism. the g2c-tr is implemented as a suite of bertbased models for joint sentence classification and sequence tagging. the syntactic dependency graph of the sentence is input to a g2c-tr model through its attention mechanism. figure 1 illustrates the framework of our model. given the input sen-tence @xmath0, w 2 , ..., w n ), we first use a pretrained dependency parser dp() to build the dependency graph g, and part-of-speech (pos) tags @xmath1, p 2 , ..., p n ). due to the fact that each lf is characterized by the pos of its lexical items and the syntactic dependency between them, this information is of significant importance. then, g2c-tr predicts the tagged sequence @xmath2, y 2 , ..., y n ) as follows: ## results in what follows, we first present the outcome of the sentence classification and collocation extraction and categorization experiments for the three datasets and then analyze the performance with respect to the individual lfs. ## conclusions and future work we have proposed an architecture for joint collocation extraction and lexical function typification by explicitly encoding syntactic dependencies in the attention mechanism. our experiments show that our proposed architecture drastically improves over its language model-only counterparts, and that joint multilingual training is a promising direction for less resourced languages. for the future, we would like to extend these experiments to other languages and explore zero or few-shot prompt-based methods.