Token-level Identification of Multiword Expressions using Pre-trained Multilingual Language Models (2023)

Abstract: In this paper, we consider novel cross-lingual settings for multiword expression (MWE) identification (Ramisch et al., 2020) and idiomaticity prediction (Tayyar Madabushi et al., 2022) in which systems are tested on languages that are unseen during training. Our findings indicate that pre-trained multilingual language models are able to learn knowledge about MWEs and idiomaticity that is not languagespecific. Moreover, we find that training data from other languages can be leveraged to give improvements over monolingual models.

## introduction multiword expressions (mwes) are combinations of lexical items that exhibit some degree of idiomaticity @xcite . for example, ivory tower exhibits semantic idiomaticity because its meaning of a place where people are isolated from real-world problems is not transparent from the literal meanings of its component words. multiword expressions can be ambiguous in context with similar-on-the-surface literal combinations. for example, red flag is ambiguous between an mwe meaning a warning sign and a literal combination. knowledge of mwes can enhance the performance of natural language processing systems for downstream tasks such as machine translation @xcite and opinion mining @xcite . much work has therefore focused on recognizing mwes in context, by identifying which tokens in a text correspond to mwes (e.g., @xcite @xcite and by distinguishing idiomatic and literal usages of potentially-idiomatic expressions (e.g., @xcite @xcite @xcite . one interesting line of investigation in such work is the ability of models to generalize to expressions that were not observed during training. for example, this was a focus in the evaluation of @xcite . @xcite further explore the ability of language models to encode information about idiomaticity that is not specific to a particular language by considering cross-lingual idiomaticity prediction, in which the idiomaticity of expressions in a language that was not observed during training is predicted. in this paper we further consider cross-lingual idiomaticity prediction. semeval 2022 task 2 subtask a (tayyar madabushi et al., 2022) is a binary sentence-level classification task of whether a sentence containing a potentially-idiomatic expression includes an idiomatic or literal usage of that expression. in this subtask, the training data consists of english and portuguese, while the model is evaluated on english, portuguese, and galician. as such, the shared task considered evaluation on galician, which was not observed during training. in this paper, we examine cross-lingual settings further, conducting experiments which limit the training data to one of english or portuguese, to further assess the cross-lingual capabilities of models for idiomaticity prediction. parseme 1.2 is a sequence labelling task in which tokens which occur in verbal mwes, and the corresponding categories of those mwes (e.g., light-verb construction, verb-particle construction), are identified @xcite . this shared task considered a monolingual experimental setup for fourteen languages; separate models were trained and tested on each language. in this work, we consider two different experimental setups: a multilingual setting in which a model is trained on the concatenation of all languages, and a cross-lingual setting in which, for each language, a model is trained on training data from all other languages, and is then tested on that language that was held out during training. for each task considered, we use models based on multilingual language models (e.g., mbert). our findings in cross-lingual experimental setups indicate that language models are able to capture information about mwes that is not restricted to a specific language. moreover, we find that knowledge from other languages can be leveraged to improve over monolingual models for mwe identification and idiomaticity prediction. ## models for semeval 2022 task 2 subtask a we apply bert @xcite models for sequence classification. in the initial shared task, a multilingual bert (mbert) model is used for the baseline. we consider this, and also more-powerful models, including xlm-roberta @xcite and mdeberta @xcite . for parseme 1.2, we use the mtlb-struct system @xcite , which performed best overall in the shared task. mtlb-struct simultaneously learns mwes and dependency trees by creating a dependency tree crf network @xcite using the same bert weights for both tasks. ## materials and methods in this section, we describe our datasets and experimental setup (section 3.1), implementation and parameter settings (section 3.2), and evaluation metrics (section 3.3). ## results here we present results on the semeval (section 4.1) and then parseme (section 4.2) tasks. ## conclusions in this paper, we considered new cross-lingual settings for the semeval 2022 task 2 subtask a and parseme 1.2 shared tasks, in which models are evaluated on languages that are not seen during training. our findings indicate that language models are able to learn information about mwes and idiomaticity that is not language-specific. our findings further show that additional training data from other languages can be leveraged to give improve-ments over monolingual models for identifying mwes and predicting idiomaticity. in future work, we intend to further explore the influence of language families and categories of multiword expressions on the ability of idiomaticity prediction and mwe identification models to generalize to unseen languages. we further plan to explore the ability of these models to generalize to languages that were unseen during language model pre-training @xcite .