OCHADAI at S em E val-2022 Task 2: Adversarial Training for Multilingual Idiomaticity Detection (2022)

Abstract: We propose a multilingual adversarial training model for determining whether a sentence contains an idiomatic expression. Given that a key challenge with this task is the limited size of annotated data, our model relies on pre-trained contextual representations from different multi-lingual state-of-the-art transformer-based language models (i.e., multilingual BERT and XLM-RoBERTa), and on adversarial training, a training method for further enhancing model generalization and robustness. Without relying on any human-crafted features, knowledgebase, or additional datasets other than the target datasets, our model achieved competitive results and ranked 6thplace in SubTask A (zero-shot) setting and 15thplace in SubTask A (one-shot) setting

## introduction large-scale pre-trained language models such as bert @xcite have achieved great success in a wide range of natural language processing (nlp) tasks. however, more recent studies show that even such contextual models have a limited ability to capture idiomaticity @xcite . idiomatic expressions denote a group of words that behave as single words to some extent. their linguistic behavior cannot be inferred from the characteristics of their components, and still pose a challenge to natural language processing (nlp) systems. this paper describes the system developed by the ochadai team for semeval-2022 task 2 -multilingual idiomaticity detection and sentence embedding @xcite . given that a key challenge in this task is the limited size of annotated data, we follow best practices from recent work on enhancing model generalization and robustness and propose a model ensemble that leverages multilingual pre-trained representations and adversarial training. our model ranked 6th 2 task description semeval-2021 task 2 subtask a consists of a binary classification task that requires classifying sentences with a target multiword expression (mwe) into either "idiomatic" or "literal" across english, portuguese and galician @xcite . further, it is subdivided into two settings to better test models' ability to generalize: zero-shot and one-shot. in the zero-shot" setting, multiword expressions (potentially idiomatic phrases), in the training set are completely disjoint from those in the test and development sets. in the "one-shot" setting, one positive and one negative training examples are included for each mwe in the test and development sets. note that the actual examples in the training data are different from those in the test and development sets in both settings. only the datasets provided by the organizers are allowed to train the models. participants can use only the data provided for the zero-shot setting to train the zeroshot model. however, participants were allowed to use data provided for both settings to train models in the one-shot setting. the statistics of the corpus are presented in ## system overview we focus on exploring different training techniques using bert and roberta, given their superior performance on a wide range of nlp tasks. each text encoder and training method used in our model are detailed below. ## conclusion we proposed a simple and efficient model for multilingual idiomaticity detection. our experiments demonstrated that it achieves competitive results on both zero-shot and one-shot settings, without relying on any additional resource other than the target task dataset. although in this paper we focused on the multilingual idiomaticity detection task, our model can be generalized to solve other