Evaluating Diversity of Multiword Expressions in Annotated Text (2022)

Abstract: Diversity can be decomposed into three distinct concepts, namely: variety, balance and disparity. This paper borrows from the extensive formalization and measures of diversity developed in ecology in order to evaluate the variety and balance of multiword expression annotation produced by automatic annotation systems. The measures of richness, normalized richness, and two variations of Hill’s evenness are considered in this paper. We observe how these measures behave against increasingly smaller samples of gold annotations of multiword expressions and use their comportment to validate or invalidate their pertinence for multiword expressions in annotated texts. We apply the validated measures to annotations in 14 languages produced by systems during the PARSEME shared task on automatic identification of multiword expressions and on the gold versions of the corpora. We also explore the limits of such evaluation by studying the impact of lemmatization errors in the Turkish corpus used in the shared task.

## introduction diversity of naturally occurring phenomena and artefacts is a desirable property of many environments and systems. it has been modelled and measured in many domains, including linguistics but has rarely been formalized with respect to particular linguistic phenomena within one language. this paper addresses diversity of one particular phenomenon: multiword expressions (mwes), which are combinations of words, such as out of the blue or pay a visit, exhibiting idiosyncratic properties at lexical, morphological, syntactic and/or semantic level @xcite . automatic annotation of multiword expression occurrences in textshenceforth referred to as mwe identification following nomenclature from @xcite has been the focus of many works, among which the parseme shared tasks @xcite @xcite . during these tasks, performances of participating systems were evaluated on the precision, recall, and f1-score of their annotations. in order to get a better understanding of how participating mwe identifiers behave, performances were also measured for specific subtasks such as the annotation of light verb construction occurrences (pay a visit, give a lecture), or the annotation of mwes that were not seen during training. such analysis of the resulting annotation stems from a need to make sure that the systems getting the best scores are not simply performing well on a few easier, more rewarding subtasks while ignoring others. thus, these evaluation scenarios implicitly address some aspects of data and system diversity. in this paper we follow the same objectives but we address diversity explicitly and formally. we suggest that diversity measures could later be used to put performance measures in perspective, by favoring nlp tools which cover diverse types and not only easy and repetitive cases. the paper is organized as follows. we first present related work in estimating diversity in linguistics and nlp (sec. 2). we present the data used in experiments (sec. 3). we formalize diversity with respect to the mwe phenomenon and propose concrete measures (sec. 4). then we experiment with these measures to estimate diversity in mwe-annotated corpora and in annotations produced by systems participating in the parseme shared task 1.2 (sec. 5) and we offer a discussion of the results (sec. 6). finally, we conclude and suggest perspectives for future work (sec. 7). ## related work in linguistic diversity measuring diversity, -often along its three dimensions: variety, balance and disparity (sec. 4) -has been practiced in domains such as ecology, economy, public policy, information theory, social media, etc. @xcite . diversity is also a central notion in linguistic debates. @xcite oppose to the hypothesis of the existence of language universals @xcite and suggest that linguistic research should rather use diversity as a starting point. quantifying linguistic diversity has been performed for decades. greenberg (1956) measured the probability of monolingual members of a population to speak the same language. nettle (1999), cited by @xcite , modelled language diversity in terms of richness (the number of different languages in a given geographical area), phylogenetic diversity (the number of different lineages in the phylogenetic tree of languages) or structural diversity (variation among structures within languages). the terralingua initiative foot_0 suggested that linguistic diversity should be regarded, from a holistic perspective, as part of biocultural diversity, and proposed indices to follow the number of world's active languages, the distribution of mother-tongue speakers among them and the rate of language extinction @xcite . also socio-linguistic diversity was measured in terms of the probability of using more than one common language in multilingual communication, as well as the degree of diversity of language policies @xcite . all these measures are inter-linguistic. we are, conversely, interested in intra-linguistic measures which would represent diversity of linguistic phenomena within one language, and more precisely within nlp artefacts such as language resources and outcomes of nlp tools. the need for diversity in training data and its impact of the performance of nlp tools has been stressed in parsing @xcite or question answering @xcite . in these works, however, the notion of diversity was used loosely and in ad hoc manner (e.g. to describe adding noise to training data, or using multiple knowledge sources and topics) rather than formally defined. in other works more precise diversity measures do occur. this is especially the case in natural language generation (nlg), where the so-called quality-diversity tradeoff problem is observed (systems reduce the potential diversity of their generated outputs to better fit the reference). we notice, however, that the use of diversity in nlg is not standardized. @xcite calculate the number of distinct unigrams and bigrams in generated text and zhang et al. (2020) use shannon's entropy, measures which relate to richness and balance, respectively. @xcite define word embedding similarity, i.e. the average cosine distance between utterance embeddings. @xcite use selfbleu, i.e. the bleu measure applied to generated utterances rather than to the reference. @xcite mix the 2 previous measures with jaccard, i.e. the average word overlap across utterances. these are distance measures which might be used to model disparity (if items and types are properly defined). some of those measures are also implemented in nlg toolboxes @xcite . thus, diversity estimation is becoming an inherent component of nlg models. on the other hand, complexity, a notion somewhat similar to diversity has also been measured in nlp @xcite , e.g. for the sake of language learning or text simplification. we, conversely, are interested in diversity (a notion larger than complexity) and in its promotion in language resources and tools. in this paper, we mainly focus on two of the three aspects of diversity, variety and balance. especially the latter seems to have rarely been formalised and measured in nlp. ## data before going into details on what our diversity measures will be, we take a look at the data and the ways we use them in our experiments. the parseme shared task 1.2 on automatic identification of verbal mwes (vmwes) was conducted on 14 languages. the corpus of each language, annotated for morpho-syntax and vmwes, was split into three corpora, train, dev and test. the trains, devs, and blind version of tests corpora (with annotation for vmwe hidden) were given to the participants of the shared task, which were tasked to annotate the blind tests with their systems. we will apply our diversity measures to gold annotations (vmwes manually annotated in the test corpora) and system annotations (vmwes automatically annotated by systems participating in the shared task 1.2). in addition, in section 5.1 we use the french corpus sequoia @xcite which is annotated in similar fashion, not only for vmwes, but for all syntactic types of mwes. diversity measures for system annotations make sense mainly for correctly annotated mwe occurrences. a mwe occurrence is considered correctly annotated in a given corpus if all its tokens annotated in the corresponding gold corpus, and only those tokens, have been annotated as part of the same mwe occurrence. trivially, this definition also applies to mwes from the gold corpus itself, which are all considered correctly annotated. for example, if sentence 1 below is considered the gold annotation, where paid visit and out of the blue are respectively annotated as mwe a and b, then in sentence 2 only mwe d out of the blue is correctly annotated and in sentence 3 no mwe is correctly annotated. in parseme shared task, identification systems were also tasked to assign a category (lightverb construction, verbal idiom, inherently reflexive verb, etc.) to each annotated mwe, however, we will not look at these annotations. a mwe can therefore be considered correctly annotated by a system even if its attributed category was wrong. in this paper, we will be interested in the diversity both of gold annotation and of system annotations. in the latter case, we will consider only correctly annotated mwes. ## diversity measures the concept of diversity is usually divided into three distinct notions: variety, balance and disparity @xcite . each of these notions goes by other names and, adding to the confusion, is sometimes referred to as diversity. the notion of diversity relies heavily on the concepts of items and types. in ecology, items usually refer to specimens/individuals, and types refer to the species these specimens are affiliated to. in this paper, items will refer to the correctly annotated mwe occurrences and types will refer to what parseme calls mwe types, meaning the multisets of lemmas of the annotated mwe occurrences. formally, we define items and types as follows: let i be a set of items, t a set of types, and τ : @xmath0. we define items @xmath1. sec. 3), and types @xmath2: here, an item i is seen as a sequence of the annotated wordforms of the mwe occurrence, and lemma(w) the function returning the lemma of a wordform. thus, under this definition, two correctly annotated mwe occurrences (items) are of the same type if their component words have the same lemmas, e.g. pay visit and visits paid are items of type {pay, visit}. we note here that such a definition of a type, relying on the notion of a lemma, does not perfectly capture what we would instinctively refer to as mwe types. for instance, two senses of put down: 'execute' and 'belittle' are assigned to the same type despite their different meanings. sec. 5.3 brings to light other issues related to the data quality. still, we consider our approximation of types good enough to be useful. given these definitions of items and types, we will see variety as a measure of the number of types in a set of items, that is to say, a measure of how many different mwe types are present in an annotation. the more mwe types an annotation has, the more varied it is. balance is seen as a measure of the equilibrium of the distribution of items per type, meaning that it scores distribution based on how close the mwe types are from being equally represented. disparity is seen as a measure of the distance among the types present in a set of items. while some authors, such as stirling (1998), advocate for a complete diversity measure -meaning that all three aspects of diversity are taken into account jointly -we take the opposite stance and aim to measure each aspect of diversity as independently of the others as possible. we consider that such an approach would be easier to interpret and sidesteps the issue of finding the right aggregation of our notions of diversity. henceforth, we will no longer address the notion of disparity in this paper and leave it for future work instead. let us only mention two main challenges behind this concept: ## results the previous section introduced formalization of measures for two diversity facets: variety and balance. in this section, we apply these measures on mwe-annotated corpora and mwe identifiers (sec. 3) so as to assess their appropriateness to the mwe phenomenon (sec. 5.1), use them as evaluation scenarios in the parseme shared task framework (sec. 5.2), and show one of their limits related to data quality (sec. 5.3). 5 ## discussion in section 4 we define items as correctly annotated mwe occurrences, meaning that incorrect annotations are ignored during diversity measurements. this was motivated by the fact that variety would otherwise be artificially increased by wrong mwe types and balance affected by the likely abundance of wrong mwe types with very few occurrences. as a consequence the variety of systems' annotations cannot be higher than that of the gold. this is however not the case for balance (as can be seen in figure 2 ). this raises the question of whether it is preferable for a system annotation to be more balanced than its target (gold) annotation or to be as close to its gold annotation as possible. when training identification systems the aim is usually to approach gold annotations as close as possible. this is however already measured through scores such as precision and balance. when used in conjunction with performance measures we believe that balance should be simply interpreted as "higher is better". in section 4 we choose to define our types through a lexical approach of mwes. this decision was motivated by multiple reasons: (i) it is easy to implement, (ii) non-parametric, (iii) it results in clear-cut types (non-fuzzy), (iv) it offers a good granularity (for balance measures), (v) it is very similar to the parseme notion of mwe types and (vi) it is a quite adequate approximation of what we would consider linguistically motivated mwe types (mwes sharing lexemes and mean-ing). in future work we might use alternative definitions of types, e.g. cluster mwes having the same syntactic structure (verb-object, subject-verb, adjective-noun), the same semantics (to kick the bucket, to bite the dust), or the same mwe categories (light verb construction, verbal idiom). ## conclusions and future works in this paper, we borrowed the formalization of the notion of diversity from the literature in ecology. we focused on two out of the three main aspects of diversity, namely variety and balance. our contribution is to apply these measures to assess intralinguistic diversity, focusing on the particular phenomenon of multiword expressions. we not only formalize variety and balance measures in this context but we also put forward methods for selecting those variants of these measure which fit the nature of the mwe phenomenon. this validation methodology is based on corpus sampling with variable sample size. as a result, we retain richness and the e 2,1 evenness as the optimal variety and balance measures for mwes (among those studied by us). we apply these measures to the corpora and system results in the parseme shared task on automatic identification of mwes. the results show that richness of the correct annotations produced by the systems is roughly consistent with their f-measure performances. however, their balance is much less correlated with more traditional measures. we also display the limits of the richness and balance measures, when calculated on automatically annotated data, due to incorrect approximation of types under improper lemmatization in a morphologically rich language. further investigation, particularly on the evenness, is needed as an impressive amount of evenness measures have been proposed throughout the years. furthermore, the notion of disparity was only briefly touched upon in this paper. disparity measures might be based on lexical overlap between types, similarity of syntactic structures or distributional semantics. these directions constitute future work.