A Survey of MWE Identification Experiments: The Devil is in the Details (2023)

Abstract: Multiword expression (MWE) identification has been the focus of numerous research papers, especially in the context of the DiMSUM and PARSEME Shared Tasks (STs). This survey analyses 40 MWE identification papers with experiments on data from these STs. We look at corpus selection, pre- and post-processing, MWE encoding, evaluation metrics, statistical significance, and error analyses. We find that these aspects are usually considered minor and/or omitted in the literature. However, they may considerably impact the results and the conclusions drawn from them. Therefore, we advocate for more systematic descriptions of experimental conditions to reduce the risk of misleading conclusions drawn from poorly designed experimental setup.

## introduction the task of identifying multiword expressions (mwes) in texts, as defined by @xcite , can be modeled using several paradigms: syntactic parsing @xcite , compositionality prediction of mwe candidates @xcite @xcite , or sequence annotation @xcite . the sequence annotation paradigm has been recently popularised by the dimsum shared task @xcite , and by three editions of the parseme shared tasks @xcite @xcite . automatic methods designed to solve mwe identification (mwei) seen as sequence annotation range from more traditional structured sequence tagging @xcite to more free-form recent transformer-based token classification @xcite . while the sequence annotation paradigm makes it possible to analyse various idiosyncratic aspects of mwei in full text, empirical model evaluation is still a challenge. our survey focuses on experimental design choices that are not always clearly described and discussed in the literature @xcite . the data used to learn, tune and evaluate mwei models can influence a study's conclusions. for instance, the parseme corpora contain only verbal mwes; evaluations based on it favour systems that can manage discontinuities ( § 3). moreover, annotation schemes have different approaches to deal with discontinuity, variability, nesting, and overlaps, which are particular to mwes. traditionally, variations of bio labelling were used to represent some of these aspects @xcite . parseme proposes a generic corpus format, taking these above-mentioned phenomena into account. however, the lack of standardisation with the selection and application of labelling schemes leaves the door open for system developers to decide how they want to model mwes ( § 4). another important aspect of evaluation is the choice of the evaluation metrics used to assess system performance. while global exact and fuzzy metrics based on precision, recall and f-score are traditionally employed @xcite , they ignore a model's capability to deal with challenging traits like mwe discontinuity, seen/unseen mwes, and their variability. from edition 1.1, parseme designed focused measures to evaluate for these aspects @xcite . we discuss and compare these metrics, and the way systems report and discuss them in papers ( § 5). furthermore, most related work does not assess whether a superior performance is likely due to chance, that is, whether observed performance differences are statistically significant. thus, we propose a framework, a free implementation, and report significance analyses on the parseme 1.2 shared task results ( § 6). finally, we look at whether and how mwei papers report error analysis ( § 7). in short, we shed some light on these apparently minor aspects which actually can have a great impact on results and conclusions. we look at corpus constitution and split, pre-and post-processing, mwe tagging, evaluation metrics, statistical significance of system comparison, and error analyses. we compare the experiments of 40 mwei papers and discuss best practices in designing experimental setup and evaluation. ## survey scope our survey covers a total of 40 papers selected according to the following criteria: our selection is not exhaustive, disregarding influential mwei articles with experiments on other corpora, e.g. @xcite ; @xcite , and recent papers on in-context compositionality prediction, e.g. @xcite ; tayyar @xcite . to keep the number of papers manageable, we arbitrarily disregard papers published in venues absent from the acl anthology, e.g. @xcite . 1 moreover, our sample is certainly biased towards over-represented languages (e.g. english for dimsum) and mwe categories (e.g. verbal mwes for parseme). nonetheless, we believe that it represents a large fraction of work in the mwe annotation paradigm, and could be complemented by a larger survey in the future. the goal of our survey is to base our discussion on quantitative data extracted from the papers. thus, intuitions can be confirmed and concrete proposals can be made for clearly identified gray zones. thus, for each of the surveyed papers, we systematically answered the following questions: hereafter, we distinguish the 27 papers submitted to one of the four recent shared tasks (st papers) from the 9 standalone papers, not submitted to a shared task (non-st papers). moreover, 4 of the papers are overall shared task description papers. for the others, we will use the terms systems and models interchangeably, as these papers describe experiments using a system that relies on a proposed model or family of models. ## corpus constitution and selection the first aspect that we look at is the corpora used in the mwei experiments. languages the languages of the corpora used mostly depend on the data available for sts. the semeval dimsum st provided corpora in english @xcite , whereas parseme sts provided corpora for 18 languages in edition 1.0 @xcite , 19 languages in edition 1.1 @xcite , and 14 languages in edition 1.2 @xcite . the dimsum corpus is based on streusle @xcite and is annotated for most major mwe categories (nominal, verbal, adverbial, functional), but does not include category labels. the parseme corpora, on the other hand, contain fine-grained mwe category annotations, but only cover verbal mwes. figure 1 shows the distribution of papers across the 24 languages considered by our paper sample. the reasons that lead to choosing a given corpus and/or set of languages in non-st works are various: language diversity @xcite , corpus domain @xcite , and corpus quality and size @xcite . conversely to the number of papers per language, we can also look at the number of languages addressed by each paper. most papers (26 out of 40) address more than one language, with the following distribution: 1-3 languages: 15 papers; 4-10 languages: 6 papers, 11 languages or more: 19 papers. among the 9 non-st papers, 6 cover only one language, whereas 3 are multilingual. only 2 papers reported limiting their predictions to a subset of mwe categories @xcite , otherwise the target mwe categories are by default all those present in the corpora. the prevalence of multilingual systems is probably due to the large amount of available corpora in the parseme collection, and to the use of largely language-independent methods based on these corpora. on the other hand, high cross-lingual variability is observed in most mwei experiments. this can be due to the heterogeneity in the corpora and/or in the mwes in each language (and how mwei methods model them). language-specific parseme corpus description papers not covered here can provide details, e.g. for basque @xcite , chinsese @xcite , english @xcite , irish @xcite , italian (monti and di buono, 2019), polish (savary and waszczuk, 2020), portuguese @xcite , romanian @xcite , turkish @xcite , among others. domains corpus domain may play an important role in mwei. dimsum includes texts from 3 domains: web reviews, ted talk transcriptions, and tweets, and the st paper analyses results per domain. one paper in our sample focuses on tweets, using this corpus @xcite . parseme corpora contain mostly newspapers, with a few exceptions (e.g. french contains also wikipedia, transcripts, and drug notices). one interesting case is that of the parseme hungarian corpus, which contains barely any idioms, due to its highly specialised nature (law texts). thus, systems using this corpus tend to report good performance, since this difficult category is under-represented @xcite . @xcite report cross-corpus (thus cross-domain) experiments using fine-tuned pre-trained language models with fine mwe+supersense labels. ## evaluation metrics evaluation strategies for structured tagging tasks are less straightforward than that of classification. system performance is determined based on the correct prediction for sets of labels (e.g. for all tokens in raining cats and dogs). the strict matching between the labels of all components of an mwe in the gold data and its correspondents in the predicted data is measured using mwe-based precision, recall and f1 measures in parseme. the same measures are referred to as exact match in dimsum. nevertheless, in order to reward systems for partially correct predictions, parseme uses token-based precision, recall and f1 measures and dimsum @xcite introduces link-based measures which are computed based on links (correct use of tags) between consecutive tokens in an expression. 8 20 out of 21 parseme st papers focused on reporting mwe-based f1 (with the focus of 6 parseme 1.2 papers being on unseen expressions only), and only one sys7 no dimsum st paper applied this method. 8 the linked-based measures only work for dimsum data, where the mwe tags exactly follow their tagging scheme in which there is no big o label in between mwe components and no single-token mwe. single-token mwes are allowed in parseme to account for tokenisation problems, e.g. spanish abstenerse (lit. 'abstain oneself'), which occurs as such although ideally it should be tokenised as abstener se. standard machine learning approaches optimize systems towards the best f1-measure. depending on the target task, precision or recall might be more beneficial. @xcite boost mwe-based precision by modifying the output of their transformer-based system by filtering out the predictions that involve tokens that are not connected in dependency trees. focused measures introducing focused measures in parseme 1.0, 1.1, 1.2 developed over time, motivated by related work. for example, al @xcite showed the negative correlation between system performance and the number of unseen mwes. seen/unseen identifying unseen expressions became the focus of parseme 1.2, resulting in interesting insights. word embeddings trained on extra unannotated data @xcite proved successful in detecting unseen expressions and not surprisingly pre-trained language models @xcite were the best. while rule-based syntactic patternmatching based on association measures @xcite failed at capturing unseen expressions, it showed promising results in detecting various forms of a seen mwe. all 6 parseme edition 1.2 papers and 3 papers from previous editions focused on reporting performance on unseen expressions diversity evaluating a system's capability to identify variants of existent mwes is possible thanks to one of parseme's additional focused measures. only two parseme papers reported these focused measures. a more recent study by @xcite expanded on the above analysis, and proposed two new measures, namely richness and evenness, for evaluating diversity in models' predictions. in the experiments on mwe identification with parseme datasets, they showed that f1-measure performance roughly correlates with the richness of models' predictions but not with their evenness. discontinuity mwes pose a unique challenge to nlp due to the discontinuity that often occurs between the words that make up the expression. this challenge distinguishes mwes from other similar phrasal structures, such as keyphrases or multiword named entities, making their processing more difficult. parseme's sts introduce additional evaluation measures focused on discontinuity. five out of 27 studies on parseme datasets reported results on discontinuous mwes separately. most of them use dependency parse grammatical structure to identify the relationships between constituents of an mwe @xcite . @xcite propose a model which benefits from combining attention mechanism with graph convolutional network to improve identifying discontinuous mwes. we believe that these focused measures can be generalized to other nlp tasks to alleviate more thorough evaluation. ## hypothesis testing and significance system (or model) comparison has been one of the most important methodological tools, driving progress in nlp for the last 30 years. in this paradigm, we conclude that system a is superior to system b if it obtains a better evaluation score than system b on some given test set(s). the previous sections discussed data ( § 3) and evaluation metrics ( § 5) usually employed in the context of mwe identification. however, several papers throughout the decades have shown that there is a probability that this conclusion is false in general, because the test set is a limited-size sample of the actual language (text) on which the systems will be applied in production @xcite @xcite . fortunately, statistic tools can estimate this probability given the characteristics of the test set, and in particular its size. in a nutshell, hypothesis testing can be used to assume no difference between two systems as the null hypothesis to reject. then, a statistical method can be used to estimate the p-value, that is, the probability of type-i error. 10 in other words, a p-value estimates the probability of wrongly rejecting the null hypothesis (i.e. concluding that the systems are indeed different) when there is actually no difference between the systems. one can consider that the difference between the systems is statistically significant if the p-value is lower than a confidence threshold (usually set to 0.05).then, if we claim that system a is superior to b, there is a probability of at most 5% that this conclusion is wrong. in mwe identification, comparison is based on precision, recall, and f-score, which prevents the use of simple parametric tests like student's t-test @xcite . thus, non-parametric tests such as the bootstrap @xcite should be employed. however, our survey showed that p-values were reported for only 2 papers. the dim-sum st paper compares system predictions using the non-parametric mcnemar's test. the official st ranking shows three systems tied in first position since their results are not significantly different from each other. however, as discussed by @xcite , this test is not very powerful, and this result may fall into type-ii error, that is, not being able to reject the null hypothesis when it is actually true. then, @xcite report significance using randomized approximation, which is a more appropriate test in this case since it is both non-parametric and powerful. significance analysis given the lack of systematic significance analysis in our paper sample, we propose a new tool and a first analysis of the system predictions of the parseme st 1.2. we have re-implemented the st evaluation script using the cupt library. 11 on top of it, we have added an option to compare two systems, estimating the p-value of their difference for all calculated metrics (global and phenomenon specific). p-values are estimated using the bootstrap method which resamples @xmath0,000 new test sets with replacement from the original test set. 12 the pvalue is estimated as the relative frequency of extreme results, that is, the proportion of samples for which the difference between the system scores is at least twice as large as the difference observed on the whole test set. our tool is available at https://gitlab.com/parseme/significance . in practice, significance is more relevant when the differences between systems are small and/or test sets are small. this is the case for many languages and system pairs in the 1.2 edition of the parseme st. 13 our analyses were performed on each language individually, running the signifi the results table contains 2,728 p-values in total, which we cannot exhaustively present here. thus, only a sample of the results is gathered here, trying to cover test sets of different sizes, since sample size is known to influence the significance of results. in in our analysis is not exhaustive, and other mwe identification papers did report significance in the past, e.g. @xcite . nonetheless, our analyses show that this methodological precaution is mostly neglected in the field. we hope that our survey can contribute to raising awareness on this issue for future publications. ## error analysis error analysis, when conducted properly, can help to identify particularly challenging cases for mwei, whether because of intrinsic properties of the mwes, the dataset, or the language, or because of weaknesses in the model, as demonstrated by the survey. 33 out of 40 papers carried out some degree of error analysis; certain properties of mwes, languages, or corpus phenomena are investigated in particular. comparisons of model performance across languages (sometimes including examination of the linguistic features or mwe categories particular to that language) are carried out in 11 papers @xcite , while reporting the model results across the focused measures highlighted in § 5 are carried out in 15 papers. the parseme 1.1 and 1.2 papers usually report and discuss focused metrics, as these metrics were implemented in the st evaluation scripts @xcite . analyses tended to take one of two forms: example-based analysis reporting individual instances where the model performed better or worse than usual @xcite , and automatic metrics aggregated across particular properties or phenomena. among the focused metrics, some papers pay special attention to discontinuities @xcite @xcite and seen/unseen mwes @xcite @xcite . some studies analyse the model's features and modules via ablation experiments @xcite @xcite . crosslanguage performance was also discussed, especially in the first editions of parseme @xcite . more original aspects discussed less often include pos sequence patterns @xcite , the use of external lexicons @xcite , syntactic dependencies between components @xcite , pre-trained embedding representations @xcite , and tagging schemes, as discussed in § 4 @xcite . in short, although quite heterogeneous, error analyses are usually present in mwei papers, and tend to uncover interesting research questions for future work. ## conclusions and open issues this paper provides a survey on experimental conditions reported and discussed in recent works on identifying mwes. analysis of the details of methodological choices by authors helps researchers and practitioners understand the performance of different models and identify areas for improvement. while sts help benchmark many of such experimental designs and evaluation criteria, tight schedules and less attention to task description papers cause many such details still to be neglected. this survey focuses on two shared tasks on identifying mwes and consequent systems designed based on their task definitions, datasets, and evaluations. as common-sense best practices, we advocate reporting on experimental choices such as corpus constitutions and selections, pre-and postprocessing, evaluation metrics and significance testing of performance, and some error analysis performed in related work. we encourage the introduction of focused measures that facilitate error analysis, as is done in the later parseme editions. for statistical significance testing, we propose a tool that can automatically run such analyses on standard parseme-formatted predictions. however, our analyses are not exhaustive and there are other methodological details to be discussed in the papers. one aspect that we only skim over in our discussion of the use of dev sets is hyper-parameter tuning. which hyper-parameters were tuned, on which selection of the datasets, and what strategy (if any) was taken (e.g. grid search, random, etc.) are aspects that only very few of the papers clearly reported, and future work should encourage authors to report these. currently, most evaluation techniques are automatic. one open issue is whether there is a place in which manual evaluation of detected mwes should be performed, (e.g. in the context of downstream tasks). new evaluation protocols can be considered in the future, towards answering other questions, e.g. whether some categories of mwes are more important than others. we expect that our survey can contribute to the gradual adoption of methodological standards and best practices, both for shared tasks and independent research work in our community.