PARSEME corpus release 1.3 (2023)

Abstract: We present version 1.3 of the PARSEME multilingual corpus annotated with verbal multiword expressions. Since the previous version, new languages have joined the undertaking of creating such a resource, some of the already existing corpora have been enriched with new annotated texts, while others have been enhanced in various ways. The PARSEME multilingual corpus represents 26 languages now. All monolingual corpora therein use Universal Dependencies v.2 tagset. They are (re-)split observing the PARSEME v.1.2 standard, which puts impact on unseen VMWEs. With the current iteration, the corpus release process has been detached from shared tasks; instead, a process for continuous improvement and systematic releases has been introduced.

## introduction the difficulty in automatically identifying multiword expressions (mwes) in texts has been acknowledged for a while (sag et al., 2002; baldwin and kim, 2010), and confirmed through results of experiments, many of which conducted as part of shared tasks (schneider et al., 2016; savary et al., 2017; ramisch et al., 2018, 2020). mwes, especially verbal ones (vmwes), have been the focus of the parseme community since the homony-mous cost action took place @xcite and are now paid further attention, in correlation with syntactic annotation and language typology, within the unidive cost action foot_1 . training, tuning, and testing the systems that are able to identify vmwes in texts need corpora annotated with such expressions. within parseme, guidelines for annotating vmwes were created and then improved with feedback provided during annotation. when we compare the differences between v. 1.0 of the guidelines foot_2 and their v. 1.1 foot_3 , we notice that the latter came with a refined vmwes typology and an enhanced decision tree ensuring the consistent treatment of the phenomenon in a multilingual environment. the guidelines contain the following types foot_4 of vmwes, established with respect to their pervasiveness in the languages under study. universal types include: (i) vid (verbal idiom) e.g. (de) schwarz fahren (lit. 'black drive') 'take a ride without a ticket', (ii) lvc (light verb construction), which has two subtypes: lvc.full, e.g. (hr, sr) držati govor (lit. 'hold a speech') 'give a talk' and lvc.cause, e.g. (ro) da bȃtȃi de cap (lit. 'give strikes of head') 'give a hard time'. quasi-universal types contain: (i) irv (inherently reflexive verbs), e.g. (pt) se queixar 'complain', (ii) vpc (verb-particle construction), with two subtypes: vpc.full, e.g. (en) do in and vpc.semi, e.g. (en) eat up, (iii) mvc (multi-verb construction), e.g. (fr) laisser tomber (lit. 'let fall') 'give up'. language-specific types -so far, only italian has defined such a type: icv (inherently clitic verb): (it) smetterla (lit. 'quit it') 'knock it off'. experimental category -iav (inherently adpositional verbs), e.g. (es) entender de algo (lit. 'understand of something') 'know about something' -is annotated optionally. whenever language-specific characteristics demand it, the decision trees are adjusted to reflect those characteristics, as in the case of italian or hindi. the initiative of collecting and annotating corpora following common guidelines was initially joined by 18 language teams. with each new edi-tion of the corpus, some teams remained active, some others were on standby and some new teams joined. in total, until edition 1.2, corpora for 26 were created but not unified within one single edition. with this new release (v.1.3) which is the topic of this paper, our objectives are: (i) to release all past 26 languages 6 in a unified format, i.e. morphosyntactic annotation in universal dependencies 7 (ud) (nivre et al., 2020) format, (ii) to detach the corpus releases from shared tasks, and (iii) to define a process of continuous improvement and systematic releasing (following the ud model). this describes the novelties concerning the annotated data (sec. 2-4), their underlying morphosyntactic annotation layers (sec. 5), and their split (sec. 6). then, the statistics of the resulting corpus are provided (sec. 7). we also describe recent developments of the technical infrastructure at the service of the corpus development (sec. 5-9). we provide results of two vmwe identifiers trained on the new release, which establishes new state of the art for many languages (sec. 10). we finally conclude and evoke perspectives for future work (sec. 11). the corpus is available for download at http://hdl.handle.net/11372/lrt-5124 . ## new languages we have two new languages on board: arabic and serbian. the previous dataset for arabic was created by hawwari in parseme 1.1 (ramisch et al., 2018). however, this corpus has never been published under an open license, being restricted to the shared task participants. the arabic corpus in parseme 1.3 is a new corpus created from scratch. more than 4,700 vmwes have been annotated in about 7,500 sentences taken from the ud corpus prague arabic dependency treebank (padt) (hajic et al., 2004), containing newspaper articles. this new annotated corpus is already available in the parseme repository under the cc-by v4 license. the serbian language was not represented in the previous versions of the parseme corpus. the @xcite the 26 languages and their corresponding language codes are: arabic (ar), bulgarian (bg), czech (cs), german (de), greek (el), english (en), spanish (es), basque (eu), farsi (fa), french (fr), irish (ga), hebrew (he), croatian (hr), hungarian (hu), hindi (hi), italian (it), lithuanian (lt), maltese (mt), polish (pl), portuguese (pt), romanian (ro), slovene (sl), swedish (sv), serbian (sr), turkish (tr), chinese (zh). @xcite universaldependencies.org first step in preparing the serbian parseme corpus consisted of the preparation of the large set of examples required for the guidelines. @xcite through this work, it became clear that the types of vmwes to be encoded in serbian texts were: lvc (full and cause), vid, and irv. the serbian corpus in parseme 1.3 consists of 3,586 sentences of newspaper texts covering mostly daily politics, and a small part dealing with fashion. the morphosyntactic annotation of texts was done using udpipe (straka, 2018). more than 1,300 vmwes (approx. 640 different types) were annotated in it by one annotator. for the next edition of the corpus, we will try to recruit at least one more annotator for the same text. ## enlarged corpora three of the languages already present in previous editions were further enhanced with new annotated data: greek, swedish, and chinese. in the first edition of the parseme corpus, the greek (el) dataset was rather small and we have been committed since to adding new data in view of ultimately providing a corpus of adequate size. the new dataset comprises newswire texts (c. 26k sentences) also from sources that are characterized as bearing an informal register, lifestyle magazines, and newspapers, in order to account for new types of vmwes. only a fraction of the greek dataset bears manual annotations at the lemma, pos, and dependency levels, namely the one originating from the ud initiative; the rest was completed automatically using udpipe. vmwes annotation was performed by two annotators; during the annotation process, extensive discussions were aimed at manually correcting common errors and avoiding inconsistencies. the swedish data set is expanded in comparison to parseme release 1.2. the swedish annotations now cover the complete ud swedish-talbanken treebank, increasing the total size from 4,304 to 6,026 sentences. the swedish corpus includes the manual morphosyntactic annotations from ud, now updated from version 2.5 to version 2.11. the new annotations were done in connection to the parseme 1.2 annotation campaign, by two trained annotators. as an extra decision support, the annotators were given access to the report from the consistency check for swedish parseme 1.2, which both annotators reported as being very useful. in this edition, the chinese data includes 9,000 newly annotated sentences from the conll 2017 shared task (zeman et al., 2017). the columns were updated with the new udpipe model to make the data consistent with the standard of ud 2.11. all the sentences were double annotated and the decisions were made by a trained linguistics student for the disagreed ones. ## enhancements of the existing data the croatian parseme annotations were, long overdue, transferred to the source hr500k dataset (ljubešić et al., 2016) @xcite . sentences in hr500k that were annotated with parseme annotations are those that are annotated with gold ud linguistic annotation. with the parseme annotation transfer into hr500k, we enabled the gold ud annotations, which are being continuously improved, to be transferred back to the croatian parseme dataset. the percentage of sentences that went through some change is rather staggering: from 3828 sentences, only 374 (9.8%) stayed identical as in parseme version 1.1, while the remaining sentences went through some sort of improvement in the linguistic annotation, either ud error correction or ud standard enhancement. the romanian corpus contained annotation of the vid, lvc.full, lvc.cause, and irv types of mwes in its previous releases. the new version contains annotation of iavs, a type that was experimental in the shared task 1.2. working with this type raised a few challenges, given that the class of such verbs seems to be heterogeneous with respect to the presence of the preposition in various syntactic structures in which the verb occurs. on the other hand, the test for identifying this type has proven insufficient in the case of some verbs, which shows the need for revisiting it. given the frequency of this type in the corpus (a third of all vmwes in the romanian corpus is represented by iavs, see in some languages, manual revision of previous annotations was performed. thus, in english, the 1.1 version of the corpus went through a thorough process of consistency checks (savary et al., 2018). in polish, a number of controversial or inconsistent annotations were spotted by a new team member. grew-match was also used to identify potential errors. revealed errors were manually fixed. in the irish corpus, a controversial category was removed (irv), with mwes of this type re-categorised as iav or vid. morphosyntactic annotations were also updated to be consistent with ud v2.11. the turkish corpus was improved in its morphosyntactic annotations. it was manually reviewed by one annotator and the incorrect annotations from the previous release were corrected. this resulted in changes in the form, lemma, upos, and xpos fields in, respectively, 15, 2480, 1250, and 1266 tokens. the number of morphological features changed in the features field is 6451. for two languages, czech and maltese, parseme corpora were released in version 1.0 only. the 1.0-to-1.1 upgrade of the parseme annotation guidelines foot_7 involved a few major changes, including a redesigned set of vmwe categories. thus, upgrading 1.0 corpora to version 1.1 requires some manual intervention. further upgrades to versions 1.2 and 1.3 were minor and mostly automatically applicable. for the present release, we could achieve a partial upgrade from version 1.0 to 1.3 in czech and maltese. future work includes manual annotation of the lvc.cause category, which emerged in v 1.1. ## compatibility with universal dependencies syntactic and semantic properties of mwes are deeply intertwined. @xcite therefore, the parseme corpus has, since its beginnings, been released with annotations for both vmwes and morphosyntax for most languages. the morphosyntactic annotations have not been produced by parseme annotators but rather extracted from existing treebanks or generated by parsers. to this end, we have been increasingly relying on the ud framework (de marneffe et al., 2021), treebank collection (nivre et al., 2020) and ud-pipe parser (straka, 2018), as parseme largely shares ud's objectives and principles of universality and diversity. since edition 1.1, the parseme corpus uses the .cupt format, which extends the ud' conll-u format with a vmwe annotation layer. @xcite since edition 1.2, we have strongly advocated compatibility with ud version 2. this objective has been finally achieved in the current 1.3 edition. in 11 languages, we have at least partly manual morphosyntactic annotations. when those stem from ud treebanks, we synchronised them with the most recent ud release (2.11 from november 2022) @xcite . in 16 languages, at least part of the morphosyntactic data had been automatically generated and we updated them using the most recent udpipe models (mostly v 2.10). @xcite whenever several models per language existed, tagging/parsing performances and the genre of the training corpus were used as choice criteria. as a result, all 26 language corpora now use the ud-2 tagsets (most often in the 2.11 version) for pos, morphological features, and dependency relations. @xcite the readme files were updated with details of the above updates and a change log now documents the history of releases. in the future, the procedure for synchronising morphosyntactic annotations with recent ud releases or updating them with udpipe should be made fully automatic. in the long run, we plan gradual convergence with ud, so as to possibly integrate the parseme annotations into ud treebanks (savary et al., 2023). ## corpus re-split the parseme shared task edition 1.2 involved dividing the annotated corpora provided by the task organizers into three subsets: training, development, and test (train/dev/test). the training data is used to train the mwe identification systems, the development data is used to perform model selection and fine-tuning, and the test data is used to evaluate the performance of the final models. since new languages were added and others updated, we decided to follow the 1.2 standard (ramisch et al., 2020) to re-split the annotated corpus for each language participating in the 1.3 release. this splitting method is based on two key parameters: the number of unseen vmwes in the test data compared to the combined train and dev data, and the number of unseen vmwes in the dev data compared to the train data. the latter ensures that the dev data is similar to the test data, thereby making it possible for systems that are tuned on the dev data to perform well on the test data. just as in the shared task 1.2, we set the number of unseen vmwes in the test to 300 and the number of unseen vmwes in the dev to 100. this configuration has been established to ensure a balanced split that meets the input specifications while preserving the natural distribution of the data, particularly the ratio of unseen to all vmwes. this particular attention paid to unseen vmwe is motivated by the observation from shared task 1.1 that the performances of the vmwe identification systems correlate weakly with the size of the training data but strongly with the proportion of unseen vmwe in the test data. the statistics for the train/dev/test splits across 26 languages can be found in ## statistics of the corpus over 127,000 vmwes are annotated across all 26 languages. the most frequent categories are lvc.full, irv and vid. the universality (understood as existence in all languages under study) is confirmed for vids and lvc.full. lvc.cause, deemed universal, is not annotated in czech and turkish. in czech this is due to the fact that the corpus development was on standby since edition 1.0 in which the lvc.cause category was not defined (cf. sec. 4). in turkish we might face a language-specific understanding of the guidelines. the (quasi-universal) irv category is present in all slavic and romance languages of the collection. among germanic languages, irvs are present in german and swedish but not in english. vpc.full is a pervasive category in hungarian and in all 3 germanic languages. it also occurs in arabic, greek, hebrew, irish, and italian. vpc.semi is the dominating category in chinese and is observed in germanic languages, hungarian, irish, and italian. iavs are present in some languages and not others -this is not due to the nature of the language but rather to the fact that this category is considered experimental and has been annotated optionally. mvcs are pervasive in chinese and in hindi. their high frequency in spanish is probably due to a language-specific understanding of the guidelines. @xcite finally, ls.icv is an italian-specific category and obviously occurs in this language only. all corpora are currently being released under various flavors of the creative commons license. their publication via the lindat/clarin platform is upcoming. ## annotation guidelines one important aspect of the parseme guidelines is the database of examples in multiple languages. currently, the guidelines feature 232 example identifiers, each covering up to 28 languages. however, not all languages have examples for all example identifiers: we have a total of 1,980 examples, whereas, in theory, we could include up to 232 × 28 = 6, 496 examples. in edition 1.2, the guidelines contained 1,801 examples; the newly added examples concern mostly serbian and arabic, i.e. the languages for which new corpora have been created for this release. figure 1 shows a histogram with the number of examples per language, ranging from 188 for spanish to only 1 example for turkish, hebrew, and lithuanian. @xcite the examples in the guidelines are complex, including their form in the original language, lexicalised components in bold, literal, and idiomatic translations, as well as explanations, comments, negative counter-examples, etc. their addition by language experts is a time-consuming and errorprone process that required much energy. one of the latest improvements on the parseme guidelines is a system for online example editing. the original xml language used to edit the examples on a shared online spreadsheet was replaced by an online editing system illustrated in figure 2 . we expect that this system will allow for a much quicker and more autonomous editing of examples by language teams. ## versioning, documenting and querrying in order to help the maintenance of the different corpora, a new infrastructure was set up. all existing corpora, gathered from different previous releases were put in the same gitlab group foot_15 , with each language having its own repository. now, all new updates on treebanks and new data are stored in this unique place. a rich collection of wiki pages, available from the same gitlab space, gathers rich documentation of the parseme corpora and shared task initiatives, the corresponding tools and procedures, etc. the grew-match (guillaume, 2021) tool has a new instance foot_16 which gives access to the parseme corpora. with this tool, it is possible to make graph-based queries to observe the annotated data; both parseme annotations and the underlying ud annotations can be used in queries. data from each release is available. moreover, thanks to a continuous integration system, data synchronized with the current development state of each of the 26 corpora (i.e. the data available on the master branch of each gitlab repository) can be accessed in grew-match and the corresponding consistency checks web page is updated automatically when data changes foot_17 . as an example of grew-match usage, a simple request foot_18 can be used to observe what verb lemmas are used in lvc.full annotation in a given corpus (the english one in the example). ## system results we began training two state-of-the-art systems, namely seen2seen (pasquer et al., 2020) and mtlb-struct (taslimipoor et al., 2020), on each corpus of release 1.3. ranked first in the parseme shared task edition 1.2 closed track (as far as the global mwe-based f-measure is concerned), seen2seen reads all annotated vmwes in the train and then extracts from the test all candidate occurrences of the same multi-sets of lemmas. the system subsequently runs these candidates through a sequence of morpho-syntactic filters. in total, 8 filters are defined, and seen2seen chooses which filter to activate for each language during the training phase based on its performance on the dev corpus. mtlb-struct is a semi for mtlb-struct, the evaluation of the release 1.3 models for irish, croatian, hungarian and romanian could not be performed for technical reasons. among the other 10 languages covered both in release 1.2 and 1.3, the increase of the global f-measure is the most significant in swedish. also basque, french, portuguese and turkish benefit from the data enhancements. for other languages, the f-measure is lower than in version 1.2, likely due to switching to the multi-task version of the model. the primary focus of figure 3 is to showcase how the f-score changes as the number of vmwe tokens in the training corpus varies between releases 1.2 and 1.3. by analyzing the f-scores of different languages, we can observe the effect of the number of vmwe tokens in the training corpus on the performance of the seen2seen and mtlb-struct systems. for instance, the increase of the swedish (sv) and basque (eu) datasets brought about a higher f-score. conversely, the f-score for chinese (zh) significantly decreased despite the increase in the number of vmwe annotations. this might be attributed to the increased number of unseen vmwes in the larger corpus. interestingly, the turkish dataset decreased in edition 1.3 but the global f-score for both systems increase, which might stem from the higher quality of the 1.3 release data. for seen2seen, a large increase of the dataset brings a significant decrease of the f-score, which might indicate a biased nature of the 1.2 release, balanced in version 1.3. note that we restrict our comparison to edition 1.2. it would be less meaningful to compare the scores of editions 1.0 and 1.1 with the current version since the splitting methods used in those editions did not prioritize unseen vmwes. but, even restricted to releases 1.2 and 1.3, the comparison may not be fully reliable, since: (i) each corpus was re-split into train, dev and test sets, i.e. the systems are not trained and evaluated with the same data partitions, (ii) only teh multi-task version of mtlb-struct is examined for release 1.3. ## future work this paper summarises the first release of the parseme corpora out of the context of a shared task. this fourth release (v.1.3) is the first one to cover the union of all the languages included in the previous three releases. moreover, 2 new languages were included, a significant amount of additional data was added for 3 languages, and annotations for many languages were enhanced in various ways. the future of the parseme corpus collection relies on the interests and availability of its volunteer contributors for each language. from the infrastructure perspective, we would like to consolidate the release methodology so that future yearly releases can smoothly integrate and make available the upgrades performed throughout the year by language teams. this includes further automation of procedures, in the spirit of ci/cd foot_19 , including updates of the ud morpho-syntactic annotations, validating the file formats and mwe annotations, and checking the readme.md documentation. another important goal of parseme is the extension of its guidelines to (a) non-verbal mwes, (b) verbal mwes not covered in the current guidelines, and (c) improved cross-lingual account of phenomena that are currently biased by the set of languages covered in the corpora. finally, we envisage synergies with the ud community so that the mwe layer and the morphosyntactic annotations become gradually even more compatible. the challenges to achieving this goal include reaching compatible tokenisation decisions, unified terminology, reduction of redundancy (e.g. mwes annotated as subrelations of syntactic dependencies), and syntactic connectiveness of annotated mwes. two other initiatives which contributed to the outcomes presented here are the ca21167 cost action unidive (universality, diversity and idiosyncrasy in language technology) and the dagstuhl seminar 21351 (universals of linguistic idiosyncrasy in multilingual computational linguistics).