Vector Spaces for Quantifying Disparity of Multiword Expressions in Annotated Text (2024)

Abstract: Multiword Expressions (MWEs) make a goodcase study for linguistic diversity due to theiridiosyncratic nature. Defining MWE canonicalforms as types, diversity may be measurednotably through disparity, based on pairwisedistances between types. To this aim, wetrain static MWE-aware word embeddings forverbal MWEs in 14 languages, and we showinteresting properties of these vector spaces.We use these vector spaces to implement theso-called functional diversity measure. Weapply this measure to the results of severalMWE identification systems. We find that,although MWE vector spaces are meaningful ata local scale, the disparity measure aggregatingthem at a global scale strongly correlateswith the number of types, which questions itsusefulness in presence of simpler diversitymetrics such as variety. We make the vectorspaces we generated available.

## quantifying diversity one may argue that, in nlp, many situations can benefit from having a higher diversity, the main example being the quality of the training set for its impact on system quality @xcite . thus diversity is often desirable and there is a research objective of formally measuring it. more precisely, there is an interest in measuring the diversity of specific linguistic phenomena in corpora. diversity can be understood through variety, balance, and disparity @xcite . to understand these three aspects, let us consider the two following examples that tackle specifically the diversity of vmwes. 2 example 1 "i just got of @xcite the phone with hai and he told me how to make @xcite an adjustement [2a] on a day to day basis [...] and p&l would still somehow work out [3] because adjustments @xcite would be made @xcite .", (typos from original text) example 2 "does this mean that for june [...] we should not do anything and just make adjustments @xcite on a going forward [2] basis (and assume everything will work out [3] at month end)?" in these examples, items (i.e., individual instances) are underlined. the first example contains 4 items, while the second example contains 3. however, items may be clustered into types based on some shared identity, such as make [...] adjustment 'to make an adjustment' and adjustments [...] made 'to make an adjustment' in the first example. both examples thus contain 3 types: to get off, to make an adjustment, and to work out for the first example, to make an adjustment, to go forward, and to work out for the second example. diversity is measured on types. variety concerns itself with the number of types; as both examples have 3 types, they are equally varied. balance concerns itself with the evenness in the distribution of types; as the first example has a type with more items than others, it is less balanced than the second example in which every type has the same number of items. disparity concerns itself with the fundamental differences between types; as two types are shared between the two examples (to make an adjustment and to work out), the question here is which of to get off and to go forward is more different (or, phrased otherwise, more distant) from the shared types. variety, balance, and disparity are general dimensions: a number of concrete measures exists for each @xcite . variety is often trivial, as it concerns itself with the number of types, such as richness n (lion-bouton et al., 2022) or species count n -1 @xcite . ## related works: mwe vector spaces distributional semantic models represent text units as vectors of real numbers in a multidimensional space. vector representations for mwes in particular can be obtained from word co-occurrence matrices after dimensionality reduction (schulte im @xcite or neural networks trained by self-supervision @xcite . in the latter case, the vectors, called embeddings, can be trained on the level of characters, words or documents, and can be static (notably word2vec) or contextual (most often obtained with transformers). static mwe-aware word embeddings (wes), on the one hand, require a corpus which is re-tokenized so that all occurrences of mwes (and of other phrases of interest) are merged into single tokens @xcite @xcite . crosslingual embeddings can also be obtained by aligning monolingual mwe-aware static wes @xcite . a contextual embedding of an mwe, on the other hand, can be obtained straightforwardly from generic transformer models (trained on a corpus with no mwe-aware tokenisation) by combining the vectors for (sub)tokens occurring in the mwe in a precise context @xcite . this eliminates the requirement of having identified mwes in advance in the training corpus. nevertheless, hashempour and villavicencio (2020) show that merging mwes into single tokens in the train corpus enhances performances of in mwe-related tasks, also with contextual embeddings. one of the parameters for training mwe-aware embeddings is the method used to identify mwes in the train corpus, prior to their fusion into single tokens. in the simplest case, a handcrafted controlled list of phrases (including mwes), possibly lemmatized, is straightforwardly matched against the corpus. 3 most of the compositionality prediction experiments cited below, as well as @xcite and @xcite , use this technique. the embeddings for mwes are then available only for the mwes from the controlled list. in a more elaborate case, a generic mwe identifier is used to tag mwes in a large raw corpus. in this case precision may be preferred over recall by favoring mwes seen in the training corpus. embeddings have been efficiently used in mwespecific nlp tasks, most notably in automatic prediction of the degree of compositionality of a mwe. the hypothesis here is that this degree coincides with the distance between the vector representing the whole mwe and the combination of the vectors of its components (or of its synonyms and paraphrases). this principle was applied to 2-word noun phrases (ivory tower) in english @xcite , french and portuguese @xcite . verb-particle constructions (set off ) were also approached in this way in english @xcite and in german @xcite . more recent work by @xcite on persian, a low-resourced language, extends this idea to various vmwes, which are harder to model due to their morphosyntactic variability @xcite . static wes for mwes were also successfully combined with embeddings representing hypernymy relations @xcite and multimodal text-image associations @xcite . interestingly, "simple" word2vec embeddings are reported by a number of authors @xcite @xcite as outperforming more elaborate contextual wes in this precise task. another mwe-specific task is machine translation of mwes. a mwe in the source language can be translated by selecting the closest, in terms of (static) cross-lingual wes, target language word or mwe. this technique proved efficient for 10 typologically different languages in @xcite . but more recent mwe-specialized translation engines rely on transformers, fine-tuned on parallel mwe datasets @xcite or pretrained on monolingual idiom corpora @xcite . yet another task, mwe disambiguation, consists in distinguishing literal and idiomatic occurrences of a potential idiomatic expression (pie), like to take the cake 'be the most remarkable of its kind'. systems were developed notably in english, german, portuguese, galician and japanese. while static wes proved useful @xcite , contextual wes occurred more efficient (hashempour and villavicencio, 2020). 4 thus, recent best performing methods rely on pre-trained transformer models, either frozen or fine-tuned, to generate contextual phrase or sentence embeddings prior to binary classification @xcite @xcite . it is also worth noting that generic static embeddings (trained with no particular attention paid to mwes) proved useful to model the compositional/literal meanings of mwes in tasks such as translating mwes and collocations (gamallo and garcia, 2019), detecting synonyms of terminological mwes (hazem and daille, 2018), and mwe identification @xcite . to sum up, while contextual representations of mwes and their contexts outperform static wes in tasks focusing on mwe occurrences (disambiguation, translation and identification), those concerning types (compositionality prediction) seem to be solved more efficiently with static mwes. ## overview of our approach we address the task of vmwe identification with a novel perspective on evaluation: quantifying the diversity of vmwes in annotated text. while @xcite address variety and balance of annotated vmwes, they do not cover disparity. here, we bridge this gap by using a disparity measure to assess how diverse the types of vmwes found in annotated text are. this requires a measure of distance between types, and we propose to define it in terms of distance between vmwe embeddings. since the task is type-oriented, we use static vmwe-aware word embeddings, as suggested by the above soa. to this aim: ## vector spaces this section describes steps 1 through 3 of the above overview ( §4). ## disparity functions we've tested multiple disparity functions from the literature and the one we found to be most discriminant, while not in its logic related to the number of types, is the functional diversity proposed by @xcite . they present a generalisation of @xcite numbers for species diversity (corresponding to variety and balance only, as it does not include distances between types), functional diversity (relying on property-wise distances between types) and phylogenetic diversity (based on distances in a tree, i.e., the evolution tree). as we are interested specifically in the functional aspect (species diversity does not cover disparity, and phylogenetic diversity is out of scope here), we shall use their functional hill number n func α based on the generalised entropy in which @xmath0, p @xmath1,≤1 the relative proportion of the ith type, d ij (∈ @xmath2,≤2 for cosine distance) the distance between the ith and the jth types, and α ∈ @xmath3. @xmath4 as the weighted average of distances. n func α and h func α have limiting cases with b representing the logarithmic base (e in our case). these equations are parametric with α, which conditions how strongly the proportion of a pair of types should be considered; α = 0 entails the same consideration for all pairs independently of proportion, while an increasing α entails an increasing relative consideration for high-frequency pairs. this behavior may be visualised in figure 1 of @xcite . this will be relevant as we will give results for multiple values of α. for distance between types we shall use cosine distance where ⃗ v i is the vector of the ith type. ## results and discussion we use the functional diversity measure from ( 2 ) and ( 5 ) to estimate the disparity of vmwe identification systems from the parseme shared task 1.2. like in (lion-bouton et al., 2022), we estimate disparity of true positives only. we focus on polish and we use v s m t lb rather than v s s2s because we need vectors for all or most vmwes identified by all the systems. 9 to ensure that annotations of a specific system are not substantially different from that of other systems in terms of raw distances between types (on a macroscopic scale), we also provide the mean µ dist and standard deviation σ dist of the distance matrix. we note that zipfian curvature (s) and distance matrix properties (µ dist and σ dist ) are stable across systems. therefore, the outcome of formula (2) or ( 5 ) can grow in only two cases: (i) the system system recognizes more types (n grows), (ii) the system more frequently annotates types which tend to be distant from other types (so that d ij p i p j grow). we claim that (ii) has few influence on disparity. this is because ( §6), with α = 0, all d ij are considered equally, no matter p i p j , while with an increasing α, the most frequent pairs of types are increasingly favoured, to the detriment of least frequent pairs of types. if (ii) dominantly mattered, we would expect different values of α to give different rankings of diversity. but this not the case: we see that the rankings for n func α with different α in ## conclusion we have proposed methods to quantify semantic distances among vmwes and single words, via vss. on this basis we performed experiments in evaluating the task of vmwe identification along a novel dimension: disparity of the systems' results. due to huge computational costs of these experiments, not all possible scenarios were implemented. namely, v s m t lb was necessary to have a large coverage of vmwes used in disparity experiments. but v s m t lb was trained for polish and french only, due to its high computational cost. to mitigate this, v s s2s were trained (with a much lower cost) for 14 languages. similarities between v s m t lb and v s s2s on the one hand, and similarities between v s s2s for various languages on the other hand, allow us to hypothesise that the conclusions from the diversity experiments probably also apply to languages other than polish and french. thus, we may provide the following answers to our initial research questions r1 and r2. firstly, across various languages, vmwes are sensibly positioned in the vss relative to standard tokens as well as vmwe constituents. similarity and analogy testing reveals such vss have reasonable quality vmwe-wise. pairwise distances between vmwes display normal-like behavior. secondly, using formal disparity measures on these vmwes does not allow for sensible distinctions. there appears to be no link between joint probability and distance, and as distances are nearequal in high-dimension vss, disparity in this context is non-discriminant and strongly linked to the number of types n. this questions its usefulness in presence of simpler diversity metrics such as variety.