Edition 1.2 of the PARSEME Shared Task on Semi-supervised Identification of Verbal Multiword Expressions (2020)

Abstract: We present edition 1.2 of the PARSEME shared task on identification of verbal multiword expressions (VMWEs). Lessons learned from previous editions indicate that VMWEs have low ambiguity, and that the major challenge lies in identifying test instances never seen in the training data. Therefore, this edition focuses on unseen VMWEs. We have split annotated corpora so that the test corpora contain around 300 unseen VMWEs, and we provide non-annotated raw corpora to be used by complementary discovery methods. We released annotated and raw corpora in 14 languages, and this semi-supervised challenge attracted 7 teams who submitted 9 system results. This paper describes the effort of corpus creation, the task design, and the results obtained by the participating systems, especially their performance on unseen expressions.

## introduction multiword expressions (mwes) such as to throw someone under the bus 'to cause one's suffering to gain personal advantage' are idiosyncratic word combinations which need to be identified prior to further semantic processing (baldwin and kim, 2010; calzolari et al., 2002). the task of mwe identification, that is, automatically locating instances of mwes in running text (constant et al., 2017) has received growing attention in the last 4 years. progress on this task was especially motivated by shared tasks such as dimsum (schneider et al., 2016), and two editions of the parseme shared tasks, edition 1.0 in 2017 (savary et al., 2017), and edition 1.1 in 2018 (ramisch et al., 2018). previous editions of the parseme shared task focused on the identification of verbal mwes (vmwes), because of their challenging traits: complex structure, discontinuities, variability, ambiguity, etc. (savary et al., 2017). the problem is addressed from a multilingual perspective: editions 1.0 and 1.1 covered 18 and 20 languages, respectively. the annotation guidelines and methodology are unified across languages, offering a rich playground for system developers. the framework proposed by the (closed track of) previous shared tasks was tailored for supervised learning. an annotated training corpus for each language was made available for system developers. the systems, building mostly on statistical and deep learning techniques, were then able to identify mwes in the test data based on regularities learned from the training corpora. the strength of supervised machine learning approaches lies in (a) contextual disambiguation and (b) generalisation power. in other words, the identification of ambiguous expressions should be conditioned on their contexts, and new expressions or variants should be identified even if they were not observed in the training corpus. however, corpus studies show that supervised methods can take limited advantage of these strengths for vmwe identification. firstly, even if a number of studies have been dedicated to contextual disambiguation (between idiomatic and literal occurrences of mwes), recent work shows that this task is quantitatively of minor importance, because literal readings occur surprisingly rarely in corpora. namely, based on manual annotation in german, greek, basque, polish, and brazilian portuguese, savary et al. (2019b) discovered that most expressions are potentially ambiguous, but the vast majority of them never occur literally nor accidentally. secondly, mwe idiosyncrasies manifest at the level of types (sets of occurrences of the same expression) and not at the level of tokens (single occurrences). this fact, in addition to mwe's zipfian distribution and low proliferation rate, makes it unlikely to detect new mwes based on a few instances of known ones (savary et al., 2019a). thus, the generalisation power of supervised learning only applies to variants of expressions already observed in the training data. these two findings motivated the current edition of the parseme shared task focusing on the identification of unseen vmwes. a vmwe annotated in the test set is considered unseen if the multi-set of lemmas of its lexicalised components was never annotated in the training data. foot_0 differently from edition 1.1, by training data we understand all the gold data released before the training stage, i.e. both the subset meant for training proper (train) and the one meant for development/fine-tuning (dev). therefore, the main novelties in this edition are: ## manually annotated corpora the corpus used in the shared task and the underlying cross-lingually unified and validated annotation guidelines result from continuous efforts of a multilingual community since 2015. @xcite the 1.2 guidelines mostly follow those from edition 1.1, with decision flowcharts based on linguistic tests, allowing annotators to identify and categorise candidates into the following categories: foot_4 it intendersi di (lit. 'understand of') 'to know about' the only changes to these guidelines are language-specific additions: (i) a chinese-specific decision tree for mvcs, (ii) two swedish-specific sections about identifying multiword tokens and distinguishing particles from prepositions and prefixes. the manually annotated corpus for edition 1.2 covers 14 languages: german (de), basque (eu), greek (el), french (fr), irish (ga), hebrew (he), hindi (hi), italian (it), polish (pl), brazilian portuguese (pt), romanian (ro), swedish (sv), turkish (tr) and chinese (zh). @xcite new languages the underlined languages in the list above are those whose corpora are new or substantially increased with respect to editions 1.0 and 1.1. @xcite chinese is the first language in the parseme collection in which word boundaries are not spelled out in running text. thus, tokenisation constitutes a major challenge. we used previously tokenised texts from the chinese ud treebank and some raw texts from the conll 2017 parsing shared task corpus. @xcite the latter was tokenised automatically and manually corrected when segmentation errors affected the right scope of a vmwe. about 48% of the annotated vmwes consist in a single (multiword) token. irish is our first language of the celtic genus, with new vmwe-related challenges. firstly, frequent contractions of prepositions with personal pronouns make it hard to annotate iavs. the preposition is usually lexicalised while the pronoun is not, as in ga chuir sé orm (lit. 'put he on-me') 'he bothered me'. however, since these contractions are seen in ud as inflected prepositions, they are represented as single words and lemmatised into the preposition alone. @xcite therefore, the only possible vmwe annotation is to consider the pronoun as an inflectional ending, i.e. part of the lexicalised preposition (chuir sé orm). secondly, some copula constructions, like ga x is ainm dom (lit. 'x is name to-me') 'my name is x', are idiomatic and would normally find their place among the vids. this is, however, currently not possible because, according to our guidelines, a vmwe (in its syntactically least marked form) has to be headed by a verb. however, following the ud lexicalist morphosyntactic annotation principles, the head of a copula construction is the predicative noun (ainm 'name') rather than the copula (is 'is'). swedish had a small annotated corpus in edition 1.0, but the new corpus was annotated from scratch. the main challenge was related to particle-verb combinations occurring as single tokens. some of them can be seen either as unique words, i.e. no vmwe candidates, or as multiword tokens (mwts), i.e. potential vpcs. this depends on whether they can occur both in the joint (one-token) and in the split (two-token) configuration, with the same or a different meaning. for instance, sv pågå (lit. 'on-go') 'be in progress' can be split but only with a changed meaning sv gå på (lit. 'go on') 'keep bringing the same issue up'. in sv överleva (lit. 'over-live') 'survive' the particle (över) is easily distinguished from the verb but the split configuration never occurs. other compound verbs, like sv sysselsätta (lit. 'activity-put') 'put into work', cannot be split either. currently, all such cases are considered mwts and annotated as vpcs or vids. about 49% of the annotated vmwes contain a single (multiword) token. ## corpus splits some datasets in edition 1.1 contained very few unseen vmwes. @xcite using them as is would lead to statistically unreliable assessment of systems' performance on unseen vmwes. thus, we had to design a strategy to re-split the corpora controlling for the distribution of unseen vmwes. our two prerequisites were to: (i) ensure a sufficient absolute number of unseen vmwes for each language (ii) adapt the strategy to the (7 out of 14) languages with no new annotated data compared to previous editions. hence we could not use the strategy of the wnut2017 shared task on novel and emerging entity recognition, which would consist in annotating new texts, pre-filtered so as not to contain the vmwes already present in the existing data (derczynski et al., 2017). therefore, we decided to split the whole annotated data for each language by randomly placing sentences in the training (train), development (dev) or test sets. we considered several splitting methods differing in the parameters that were controlled. apart from the absolute number of unseen vmwes, the unseen/all vmwe ratio, as well as the test/whole corpus size ratio, seemed like desirable parameters of the splitting method. however, these three parameters interact. figure 2 , which plots the average unseen ratio as a function of the train+dev size (in terms of the number of sentences), shows that unseen ratios greatly vary across languages, even when controlling for train+dev size. furthermore, we can see that this ratio depends on the relative size of the train+dev/test sets. so while the unseen ratio may well depend on some traits intrinsic to the language, it clearly depends on other, external, factors (e.g. the chosen text genres and the particular split). on the other hand, the unseen vmwe ratio was proved to better (inversely) correlate with mwe identification performance than with the training set size alone (al saied et al., 2018). the analysis above dissuaded us from controlling for a "natural" (i.e. close to the average across random splits) unseen ratio. therefore two options were considered: (1) perform random splits using predetermined proportions for train/dev/test sets and pick a split that best approaches the "natural" unseen ratio for that language, while reaching a sufficient absolute number of unseen vmwes in the test set; (2) target roughly the same absolute number of unseen vmwes per language, while the test size and unseen ratio follow from it naturally. both options restrict the unseen ratio (which still varies depending on the specific split). we preferred the second one because it gives equal weights to each language in system evaluation. ## conclusions and future work the contributions of the parseme shared task 1.2 can be summarised as: (1) the creation and enhancement of vmwe-annotated corpora including three new languages, (2) an evaluation methodology to split the corpora ensuring the representativity of the target phenomenon, and (3) encouraging results hinting at improvements on the identification of unseen vmwes. in the future, we would like to implement continuous corpus development, with frequent releases independent of shared tasks, so that new languages can join at any time and system developers benefit from latest corpus versions. additionally, our longterm aim is to increase the coverage of mwe categories, including nominal expressions, adverbials, etc. finally, we would like to pursue our efforts to design innovative setups for combining (unsupervised) mwe discovery, automatic and manual lexicon creation, and supervised mwe identification.