Verbal Multiword Expression Identification: Do We Need a Sledgehammer to Crack a Nut? (2020)

Abstract: Automatic identification of multiword expressions (MWEs), like ‘to cut corners’ (to do an incomplete job), is a pre-requisite for semantically-oriented downstream applications. This task is challenging because MWEs, especially verbal ones (VMWEs), exhibit surface variability. This paper deals with a subproblem of VMWE identification: the identification of occurrences of previously seen VMWEs. A simple language-independent system based on a combination of filters competes with the best systems from a recent shared task: it obtains the best averaged F-score over 11 languages (0.6653) and even the best score for both seen and unseen VMWEs due to the high proportion of seen VMWEs in texts. This highlights the fact that focusing on the identification of seen VMWEs could be a strategy to improve VMWE identification in general.

## introduction multiword expressions (mwes) are word combinations idiosyncratic with respect to e.g. syntax or semantics @xcite . one of their most emblematic properties is semantic noncompositionality: the meaning of the whole cannot be straightforwardly deduced from the meanings of its components, as in cut corners 'do an incomplete job '. 1 due to this property and to their frequency @xcite , mwes are a major challenge for semantically-oriented downstream applications, such as machine translation. a prerequisite for an mwe processing is their automatic identification. mwe identification aims at locating mwe occurrences in running text. this task is very challenging, as signaled by @xcite , and further confirmed by the parseme shared task on automatic identification of verbal mwes @xcite . one of the main difficulties stems from the variability of mwes, especially verbal ones (vmwes). that is, even if a vmwe has previously been observed in a training corpus or in a lexicon, it can re-appear in morphosyntactically diverse forms. examples (1-2) show two occurrences of a vmwe with variation in the components' inflection (cutting vs. cut), word order, presence of discontinuities (were), and syntactic relations (obj vs. nsubj). however, unrestricted variability is not a reasonable assumption either, since it may lead to literal or coincidental occurrences of vmwes' components @xcite , as in (3) and (4), respectively. 2 (3) start with :::::: cutting one :::::: corner of the disinfectant bag. (4) if you ::: cut along these lines, you'll get two acute ::::::: corners. this work is licensed under a creative commons attribution 4.0 international license. license details: http:// creativecommons.org/licenses/by/4.0/ . our paper addresses vmwe variability, so as to distinguish examples (1-2) from (3-4). we focus on a subproblem of vmwe identification: the identification of previously seen vmwes. section 2 describes the corpora and best systems of the parseme shared task 1.1, sections 3 and 4 motivate and describe our system seen2020 dedicated to the task of seen vmwe identification. experimental results are shown in section 5, an interpretation is proposed in section 6 and we conclude in section 7. ## parseme shared task 2018 vmwe identification recently received attention, especially in the parseme community, with the development and release of multilingual annotation guidelines and annotated corpora. 3 these data underlie three editions of the parseme shared task, dedicated to vmwe identification. this section discusses the second edition (1.1); the third edition (1.2) was ongoing at the time of writing. ## filter-based seen2020 system our system, called seen2020, operates in the closed track and focuses on the seen vmwe identification task with a two-step approach: candidate (i.e. potential vmwe) extraction, followed by candidate filtering. this method will be illustrated with examples in french. first, the extraction phase provides vmwe candidates in dev (while training) or test (while testing) based on the previously seen lemma multisets (so as to follow hypothesis h1 from sec. 3). multisets allow extracting candidates in any order (as opposed to tuples) while keeping track of repeated lemmas (as opposed to sets). assume that a french corpus contains train sentences (5-9) and test sentences (10-19). 8 focusing on (fr) faire verb la det lumière noun 'to make the light'⇒'to shed light' seen once in train (5), the (non-exhaustive) list of candidates in tab. 2 would be extracted, as they contain the three lemmas faire, la and lumière. the extraction guarantees high recall at the expense of precision, e.g. @xmath0.013 in italian. thus, more than half candidates in tab. 2 are no vmwes but coincidental (11-12;15-a,b,c) or literal (19) occurrences. many spurious candidates may be found in the same sentence (15) because of frequent determiners like la 'the'. 9 the extraction recall is not perfect either: ex. (10) is not extracted since the determiner la disappears in the negative form. errors in automatic lemmatization also affect recall. second, the filtering phase of seen2020 aims at increasing precision. to this end, 8 filters f1 to f8 take morphosyntactic properties of vmwe components into account. like in f1, a drawback in f2-f3 is to examine the pos sequences independently of the lemmas, which may (rarely) affect vmwes with repeated pos, like (fr) donner verb sa det langue noun au adp+det chat noun 'to give one's tongue to the cat'⇒'to give up'. as highlighted in the last column of tab. 2, the activation of all 8 filters is not enough to distinguish vmwes from non-vmwes, since it results both in false positives (19) and in true negatives (14;16-17). the overall goal is, thus, to determine, for each language, the optimal set of filters to activate on test among the 2 8 = 256 combinations. to achieve this, we consider as the best combination the one that maximizes the seen vmwe identification f-score obtained on dev, as described in section 5. ## results we perform experiments on 11 languages of the parseme shared task 1.1. the dev corpora are used for tuning the 8 parameters, i.e. choosing the optimal combination (among 2 8 =256 combinations) of activated filters for each language. the best combination of filters is evaluated on the test corpora, and compared with the state-of-the-art systems trained and evaluated on the same data, as described is sec. 2. ## interpretability and generalization our method proves encouraging. not only does it outperform state-of-the-art systems, even those in the open track, but also it is interpretable. first, it is straightforward to identify the filters responsible for errors made by the system, enabling incremental development and customization to specific needs. second, the filters are based on well known and pervasive linguistic properties of vmwes. third, these properties are generic enough to allow us to put forward cross-language interpretations and perspectives. for instance, the differences observed between the corpora in fr, pt, ro on the one hand and pl on the other hand could be due to the fact that the filters were initially conceived for a romance language (fr). consequently, they may tend to perform better for languages from the same family than for slavic ones, where fine case-number inter-dependencies occur in nouns (bg, even if slavic, exhibits no case inflection). also, our filters are less relevant to languages with a large number of single-token vmwes, like hu (where our system's performances are still very high). we may also observe interesting cross-language tendencies, such as a performance drop in bg, el, eu, it and pt at the beginning of the second part of fig. 1 and 3 . it occurs with the simultaneous activation of filters f2, f4 and f5 while f3 and f6 are inactive. this might be due to the fact that this configuration favors the nearest components without looking at their syntactic connection and uses f2 which is less generic than f3 to determine the allowed pos sequences. globally the most discriminative filters across all 11 corpora are: f8 (embedded sequences) selected for most languages (8); f4 (category-specific maximal linear distance between components) and f6 (syntactic distance) for 7 languages; f5 (minimal linear distance favored) for 6 languages; f2 (sequence of pos of lexicalized components) for 4 languages; f7 (noun inflection) for 3 languages and f3 (sequence of pos including discontinuities) for 2 languages. f1 (pos disambiguation) only appears in the best configuration for french. the single optimal cross-language configuration -based on the highest macro-averaged f-score on dev (@xmath1.79) -is obtained when f2, f4, f5 and f6 are activated. 19 in other words, the distance between components, either linear or syntactic, appears as much more discrim-inative than nominal fixity. this should however be tempered by the fact that not all vmwes contain a unique noun, making f7 much more specific than those filters based on distance. the last-minute results mentioned in sec. 5.4 show that our system generalizes well to new languages and remains competitive even when compared to contextual embedding models. it might be argued that its generalization power heavily depends on high-quality predicted pos tags and dependency trees, extensively used in filters, but potentially noisy (h3). however, the parseme corpora rely on universal dependencies @xcite and on the tools trained on them, whose quality is increasing. recent results show pos tagging and parsing accuracies mostly exceed 90% and 80% for the languages of the parseme corpora. 20 moreover, syntax is often approximated by pos sequences (in f2-f5) rather than used directly (f6). finally, language-specific tuning of the filters mitigates low-quality parsing. it could also be argued that filters such as f7 could cover all types of inflection (e.g. verb tense, mood). we have extensively tested this hypothesis in previous work, automatically generating all possible feature-value combinations and selecting the most discriminant ones @xcite . however, the results of this (computationally intensive) process were worse than those of seen2020. more insight into generalization might stem from covering new languages and additional training data. ## conclusions and future work we presented seen2020, a system for the identification of seen vmwes based on a combination of morphosyntactic filters. based on state-of-the-art results, our method deliberately avoids complex machine learning and deep learning techniques for several reasons. firstly, the type-level idiosyncrasy and strong lexicalization of mwes (cf. sec. 3), evidenced in many languages, suggest that distributional semantics might not be a strong ally in distinguishing mwes from non-mwes. this is confirmed by the recent results of deep-learning-based mwe identification systems, which -even if partly competitive with respect to the state of the art -still remain largely unsatisfactory. secondly, the recent mwe identification results show: (i) the critical difficulty of generalizing over unseen data (ii) considerable room for improvement with respect to morphosyntactic variants of seen mwes, (iii) the very low frequency of literal readings of mwes. this means that focus on seen data and linguistic characterization of their variants should be enough to boost the global mwe identification results. thirdly, we wish to avoid the complex architectures and non-interpretable results of neural methods. our system's main contributions are: (i) its simplicity since no complex tuning is necessary, (ii) higher performances (for 9/11 languages) than the best closed-track systems of a recent shared task and even higher than the best open-track system for 7 languages, (iii) ability to highlight linguistic properties of vmwes, (iv) interpretability of the results and easy incremental development based on error analysis, (v) adaptability as new filters can be easily added. the system's main drawbacks are its high sensitivity to scarcity and the quality of the vmwe annotations (despite its ability to mitigate errors in other annotation layers, such as pos tagging or syntax). as future work, we would like to further enhance our system's design, e.g. by representing vmwe as multisets of lemma-pos pairs rather than of lemma and pos multisets separately, which will help avoid some errors, as discussed above. deeper per-language analyses of the results might also bring more insights on the generalization of our method and to the nature of vmwes in general. for instance, we can study the generalization of our system on out-of-domain data, since we heavily rely on the lemmas of seen vmwes, which may vary considerably across topics, registers and domains.