Lexicalised and non-lexicalized multi-word expressions in W ord N et: a cross-encoder approach (2023)

Abstract: Focusing on recognition of multi-word expressions (MWEs), we address the problem of recording MWEs in WordNet. In fact, not all MWEs recorded in that lexical database could with no doubt be considered as lexicalised (e.g. elements of wordnet taxonomy, quantifier phrases, certain collocations). In this paper, we use a cross-encoder approach to improve our earlier method of distinguishing between lexicalised and non-lexicalised MWEs found in WordNet using custom-designed rule-based and statistical approaches. We achieve F1-measure for the class of lexicalised word combinations close to 80%, easily beating two baselines (random and a majority class one). Language model also proves to be better than a feature-based logistic regression model.

## introduction recognition of multi-word expressions (mwes) is one of the main tasks in the field of natural language processing (nlp) and lexicography, notably in the development of custom-designed mwe lexicons for various nlp tools or compilation of dictionaries, respectively @xcite . however, mwes are not homogeneous and there is a plethora of their definitions and operationalizations in specialized literature. for example, according to @xcite , the range of mwes is very broad (including idioms, proper names, fixed phrases, compound nouns, collocations, to name but a few), as any "idiosyncratic interpretations that cross word boundaries (or spaces)" are considered to be mwes. these idiosyncratic interpretations of a given word combination can be related to various linguistic criteria (formal, pragmatic, statistical or psycholinguistic ones), e.g, morphosyntactic patterns, constituent substitutability, semantic compositionality, frequent/recurrent use, reproducibility, collocational strength, conventionalization, pragmatic function @xcite , and any idiosyncrasy/irregularity/nonstandardness in those criteria may imply that we are potentially dealing with a mwe that is lexicalised. we treat lexicalisation as a gradable syntax-tolexicon process whereby a purely compositional word combination (a syntactic unit) comes to be treated as a single semantic or pragmatic unit (a lexical unit), exhibiting word-like behaviour @xcite @xcite , or -in other words -as "a conventionalized association of a contentful sense with a form at the level of the lexicon" @xcite ). as we argue that lexicalisation is best described on a continuum, the range of multi-word expressions (mwes) is rather wide, starting with purely compositional word combinations created ad hoc on one end, through collocations, to fixed phrases and idioms on the other end @xcite . however, in the theory and practice of lexicography, it is often difficult to determine which mwes should or should not be recorded in a dictionary, i.e., treated as vocabulary/lexical units rather than mere word combinations created ad hoc in speech or writing. lexicographers have to make a binary decision: either this is a bona fide lexical unit or not. traditionally this status was indicated in a dictionary by place of an item in the entry and typography. more precisely, when making this binary choice, lexicographers rely on their linguistic intuition, linguistic experience and competence, contemporary and previous sources of information @xcite to decide which mwes to record in a dictionary, and these decisions may also differ across lexicographic traditions. for example, we looked into selected dictionaries of english and polish and found that english lexicographers tend to record semantically compositional word combinations much more often than their polish counterparts @xcite . in this study, we assume that the same practical lexicographic problems apply to wordnets, as not all mwes recorded in the princeton wordnet can be indisputably considered to be lexicalised, e.g., such items as elements of the wordnet taxonomy (biological group, animal group etc.), quantifier phrases (piece of furniture, article of furniture), collocations (rich people, psychology department). for this reason, we need clear and operational procedures for deciding which mwes should be included in a wordnet and which should not. by analogy to lexicography, where lexical entries in dictionaries are treated as lexical units, in this study we use the label 'multi-word lexical units' (mwlus) for those lexicalised mwes that should indisputably be recorded in a wordnet. hence, our proposed procedure, combining rulebased and statistical approaches, would help us filter out mwlus from the broad pool of mwes recorded in wordnet (or pwn/enwn) and, in consequence, facilitate making the aforementioned dichotomous choice. the findings may help fine-tune the list of wordnet mwes, which are often used as gold standard for nlp applications @xcite @xcite . finally, we believe that our findings will help us better understand how word-net developers @xcite tackled the problem of recording mwes when compiling that lexical resource. ## sample annotation from princeton wordnet and enwordnet we chose all word combinations that contained at least one space. we ruled out all proper names, as well as chemistry and biological taxonomy terms, just like we did in our previous experiment @xcite 1 . after the filtering, we got 39,406 mwes. in order to verify the potential mwlu status of the sampled 387 word combinations, we checked how they are described in 6 dictionaries of english; we assume that if a word combination was given the headword status in the dictionaries then that indicates they are treated as multiword lexical units by native speakers of english -lexicographers -whose lexical competence surpasses that of any native speaker of english. we treat data from dictionaries thus as native speakers' response to a question: is this expression a mwlu? in other words, we believe that lexical units with headword status in dictionaries are end products of lexicalization. we are going to mention some problems with this belief below. the dictionaries are all from established publishing houses, and will be mainly identified as such; they are: new oxford dictionary of english (node, british) foot_0 , merriam-webster collegiate (m-w, usa) foot_1 , collins dictionary (ced, british) foot_2 , new world dictionary (n-w, usa), collins cobuild (cobuild) foot_3 , longman dictionary of contemporary english (longman) 7 . four of those dictionaries (node, m-w, n-w, ced) are so-called medium, or desktop, dictionaries that are intended to be used primarily by educated native speakers of english, and two are so-called pedagogical dictionaries (cobuild, longman), that are intended to be used primarily by advanced learners of english or non-native speakers of english @xcite . we used online versions, as they are updated quite regularly in contrast to printed versions. these dictionaries were selected to ensure that we have a multi-faceted approach, and this can be shown as follows. first, the selection was based on the needs of the intended user, as described above, but it was also based on the size and comprehensiveness of coverage. desktop dictionaries include most of the vocabulary that educated native speakers can find in texts of english and which they may not know (that is why they reach for a dictionary), though they do not use them on their own. we used dictionaries that are meant to be used by both american or british english speakers. pedagogical dictionaries include vocabulary of high frequency that native speakers have in their active vocabulary, the needs of a non-english user, especially from outside european culture, are not quite predictable. they have a balanced selection of british and american items, therefore we did not describe them as being british or american. each mwe in our sample was manually verified in terms of its occurrence as a lexical entry in any of the six dictionaries on the basis of elimination tests, starting with m-w, followed by longman, cobuild, ced, n-w, and concluding with node. in the sample we treated cobuild, ced and n-w as one source. for example, if a mwe was recorded in m-w, then its occurrence was not checked in the remaining dictionaries, and its status as mwlu was labeled as true (t). conversely, if the mwe was not recorded in any dictionary, then its mwlu status was labelled as false (f). we denote those non-lexicalised mwes with the 'non-mwlu' label. finally, in the 387 mwes sample we obtained 144 non-lexicalised mwes and 243 multi-word lexical units. ## methodology we capitalize on and extend our earlier research @xcite , where we developed and applied a method (rule-based and statistical one using ridge logistic regression) of distinguishing between lexicalised ('mwlus') and non-lexicalised word combinations in wordnet, taking into account selected lexicality features. in the rule-based approach, we used i-synonymy and cascade dictionary equivalents, while in the statistical approach we used mwe length measured in characters, the cosine of the angle between embedding vectors calculated for wordnet glosses and mwe lemmas, mwe sense ordering in wordnet, and the existence of equivalents in each constituent cascade dictionary. we extracted the subset of mwlus from wordnet and its extension, enwordnet with high precision (> 70%), yet the completeness of both approaches varied. using the rule-based approach, we obtained approximately 25% of all mwlus, and using the statistical approach we extracted nearly 50% of the mwlus, which translates into absolute lemma hypernym, definition label jest at mock, subject to laughter or ridicule 0 take back disown, take back what one has said 1 we fine-tuned the setu4993/smaller-labse pretrained model one hundred times in a loop (with four epochs in each turn) for the need of the .632 bootstrap estimator @xcite . in each iteration, we sampled with replacement n m w @xmath0. in order to balance the training sample, we also additionally resampled 99 (= 243 -144) examples without replacement from the set of just resampled non-lexicalised mwes. the remaining (i.e. not selected) word combinations were assigned to the evaluation (testing) data set. thus, in the training data set both classes were balanced, while in the testing data set they were not. within the bootstrap loop, we calculated precision (p ), recall (r) and f 1 measure from confusion matrices for the language model, as well as for random and majority baselines. the results were further tested for significance with the non-parametric .632 bootstrap method @xcite 9 . the confusion matrices were obtained from efron's .632 bootstrap rule: the number of epochs in each training iteration was arbitrarily set to 4. for bert-like models, the number should be sufficient, although not optimal. for bert itself, @xcite recommend 2-4 epochs for fine-tuning. we selected the biggest number from that range, as we had assumed that the smaller-labse model would have needed more time to optimize its weights due to a rather small annotated sample size. our assumption was later verified with accuracy gain/loss results for each iteration (fig. 2 ). the posterior evaluation revealed that setting the number of epochs to 4 almost always resulted in the highest accuracy scores. this excludes overfitting, but still our approach is prone to the problem of underfitting. we used default settings for other training parameters. ## results the difference between the language model and the majority class baseline was insignificant, when we compared the f1-measure for the 'mwlu' class (@xmath3.32 in the test). the recall for the 'mwlu' class was, of course, lower than the 100% of the baseline. comparing efficacy of smaller-labse cross-encoder with a feature-based approach @xcite , we find that current f1 measure for the 'mwlu' class is much better (78% vs. 58%, @xmath4.01), while the measure for the 'non-mwlu' class is not worse (62% vs. 61%, @xmath5.31). 10 we retrained the model on all manual annotations and applied the fine-tuned cross-encoder to wordnet data set of 39k word combinations. out of them, 25.5k were found to be lexicalised by the language model. ## conclusions in a bootstrap cross-validation, we have found that the smaller-labse cross-encoder performed very well on a manually annotated sample of nearly 400 word combinations. both precision and recall for multi-word expressions were close to 80%, 10 please note that for the comparison with results from the previous experiment, we used bootstrap point estimation on mean logistic regression values, instead of paired bootstrap. while the statistics for non-lexicalised mwes were higher than 60%. the discrimination between lexicalised and non-lexicalised expressions worked better than two random baselines (simple uniform distribution and majority class baselines). the usage of the language model, i.e. the smaller-labse cross-encoder, also improved the results obtained in @xcite with a more traditional feature-based method. interestingly, the cross-encoder model was given no more than bare lemmas and their synset definitions enriched only with hypernyms. no corpus frequency (a feature important in mwe recognition) was provided. we assume that the smaller-labse cross-encoder (the black box par excellance) relied on semantic discrepancies between a word combination and its semantic description in the definition, that is, on semantic opacity/compositionality. but this assumption should be further verified in consecutive experiments in the future. the rationale for our experiment is pivoted on lexicographic descriptions taken manually from dictionaries. a few words must be said to address possible shortcomings of this approach. native-speaker dictionaries are often constricted by the tradition of monolingual dictionaries in english and, what follows, by the expectations of users. this is the reservation that we voiced in section 2: native-speaker dictionaries can include items because these items were included in some dictionaries that had been published earlier and which were quite influential. and these items are not lexical units, even though they are quite frequent in texts but the users might expect them in a dictionary. m-w and oxford dictionaries are such influential dictionaries. in contrast, editors of pedagogical dictionaries are not constrained by tradition and one may believe that the items they include are genuine lexical items. unfortunately, this also works in the other direction: a mwlu that is not very rare in texts may not be recorded in dictionaries because no previous dictionary recorded it. clearly there is room for improvement both for wordnets and for "traditional" dictionaries. one obstacle for changing traditional dictionaries has been removed: they are not constrained by space, as they do not have to be printed, and may freely include mwlus, which until recently have not been covered adequately because there was no sufficient space for them.