R omanian Multiword Expression Detection Using Multilingual Adversarial Training and Lateral Inhibition (2023)

Abstract: Multiword expressions are a key ingredient for developing large-scale and linguistically sound natural language processing technology. This paper describes our improvements in automatically identifying Romanian multiword expressions on the corpus released for the PARSEME v1.2 shared task. Our approach assumes a multilingual perspective based on the recently introduced lateral inhibition layer and adversarial training to boost the performance of the employed multilingual language models. With the help of these two methods, we improve the F1-score of XLM-RoBERTa by approximately 2.7% on unseen multiword expressions, the main task of the PARSEME 1.2 edition. In addition, our results can be considered SOTA performance, as they outperform the previous results on Romanian obtained by the participants in this competition.

## introduction the correct identification and handling of multiword expressions (mwes) are important for various natural language processing (nlp) applications, such as machine translation, text classification, or information retrieval. for example, in machine translation, if an mwe is not recognized as such and is literally translated rather than as an expression, the resulting translation either is confusing or has the wrong meaning @xcite . in text classification, mwes recognition can provide important information about the topic or sentiment of a text @xcite , while in information retrieval, mwes can clarify the meaning of a query and improve the accuracy of search results @xcite . the parseme cost action 1 organized three editions @xcite ; @xcite of a shared task that aimed at improving the identification of verbal mwes (vmwes) in 1 https://typo.uni-konstanz.de/parseme/ . our experiments show that by employing these two algorithms, the results of the crosslingual robustly optimized bert approach (xlm-roberta) @xcite improve by 2.7% on unseen mwes when trained on the combined dataset. additionally, we report state-of-the-art (sota) results with the monolingual training of romanian bidirectional encoder representations from transformer (robert) @xcite in comparison with the results obtained at the parseme 1.2 edition, achieving an f1-score of 60.46%, an improvement of over 20%. ## dataset the parseme multilingual corpus was annotated with several types of vmwes, to serve as training and testing material for the shared task. the quality of the manual annotation was further enhanced by a semi-automatic way of ensuring annotation consistency. for edition 1.2, the corpus contained 14 languages: basque, chinese, french, german, hebrew, hindi, irish, italian, modern greek, polish, portuguese, romanian, swedish, and turkish. the types of vmwes (i.e., universal, quasiuniversal, and language-specific types) annotated therein are described in the annotation guidelines foot_0 . the types of vmwes annotated for romanian are as follows: vid (verbal idiom) like "fura somnul" (eng., "steal sleep-the", "fall asleep"), lvc.full (light verb construction with a semantically bleached verb) like "da citire" (eng., "give reading", "read"), lvc.cause (light verb construction in which the verb has a causative meaning) like "da foc" (eng., "give fire", "put on fire"), and irv (inherently reflexive verb) like "se gândi" (eng., "refl.cl. think", "think"). the whole corpus version 1.2 contains 5.5 million tokens with 68k vmwes annotations, split into train, dev, and test sets, on the one hand for controlling the distribution of unseen vmwes both in dev with respect to test and in test with respect to train+dev, and on the other hand in ensuring a sufficient number of unseen vmwes in the test set for each language. the romanian training corpus contains 195k tokens in which 1,218 vmwes are annotated. the romanian dev set contains 134,340 tokens and 818 annotated vmwes; the romanian test set includes 685,566 tokens and 4,135 annotated vmwes. the frequency of occurrence of vmwes in romanian ranges from 8% (for lvc.full) to 22% (for lvc.cause), with an average of 12%, thus being quite redundant @xcite . ## conclusions the detection and processing of mwes play an important role in various areas of nlp. this paper made notable improvements in unseen romanian mwe identification by employing a lateral inhibition layer and adversarial training to multilingual large language models like xlm-roberta. this way, we were able to improve the results of xlm-roberta. in addition, we achieved sota results on this task with a simple fine-tuning of robert that involved a better hyperparameter search and text preprocessing pipeline, respectively. future work considers an analysis of the language-independent embeddings produced in the multilingual training, together with more experiments on other languages, to validate the generalization of this approach. in addition, we intend to add these results in liro -the public benchmark for romanian nlp models @xcite .