Z hichun R oad at S em E val-2022 Task 2: Adversarial Training and Contrastive Learning for Multiword Representations (2022)

Abstract: This paper presents our contribution to the SemEval-2022 Task 2: Multilingual Idiomaticity Detection and Sentence Embedding.We explore the impact of three different pre-trained multilingual language models in the SubTaskA.By enhancing the model generalization and robustness, we use the exponential moving average (EMA) method and the adversarial attack strategy. In SubTaskB, we add an effective cross-attention module for modeling the relationships of two sentences. We jointly train the model with a contrastive learning objective and employ a momentum contrast to enlarge the number of negative pairs. Additionally, we use the alignment and uniformity properties to measure the quality of sentence embeddings.Our approach obtained competitive results in both subtasks.

## introduction in recent years, the pre-trained models have been widely used and play a vital role in the natural language processing tasks. the success of language models relies on huge amounts of unlabeled data and the useful representation layers that are designed to draw on information from the surrounding context @xcite . however, more recent studies show that even state-of-the-art pre-trained contextual models (e.g. bert) can't accurately represent idiomatic expressions @xcite . one reason for this is that many expressions can be used both literally and idiomatically. specifically, the size of vocabulary can't increase indefinitely, which makes representing idiomatic phrases particularly challenging @xcite . idioms occur in almost all languages, to distinguish whether an expression has an idiomatic sense would leverage both cross-lingual models and multiword expresour contributions can be summarized as follows: 1) we choose three transformer-based language models from the xtreme leaderboard @xcite , and compare the effectiveness of mbert base @xcite , xlm-r base @xcite , and infoxlm base @xcite . 2) we adopt the exponential moving average method (ema) and adversarial training strategy to improve the model's generalization and robustness. we achieved considerably performance gain of 2.91%, 3.87% over the baseline solution, and ranked 12th in the zero-shot settings and 4th in the one-shot settings. 3) with finetuning on the supervised target datasets, we use cross-attention module and jointly train the model with an extra contrastive loss layer on top of the bert encoder. our approach achieved an 8.22%, 4.5% improvement compared to the baseline solution, and ranked top-4 in the pre-train and fine-tune settings. we release the source code and pre-trained models associated with this work. 1 moreover, we find that adversarial training can achieve good performance by setting the appropriate batch size. in the subtaskb, we show that joint training regularizes the sentence embeddings' anisotropic space to be more uniform but also suffers a degeneration in alignment slightly. the tradeoff between the alignment and uniformity @xcite indicates that perfect alignment and perfect uniformity are likely hard to simultaneously achieve in practice. ## background semeval-2022 task 2(tayyar madabushi et al., 2022) provides two subtasks. subtask a consists of a binary classification task aimed at determining whether a sentence contains an idiomatic expression. the sample of the dataset consists of the previous sentence, target sentence, next sentence, and mwe. the target sentence contains the potentially idiomatic mwe, and the label of 0 indicates "idiomatic" and the label of 1 indicates "non-idiomatic". our model receives the context sentences as input in the zero-shot setting, and receives the target sentence by adding the mwe as an additional feature in the one-shot setting. this is based on the results presented in the dataset paper @xcite . subtaskb consists of a novel task which requires the model to output the correct semantic text similarity (sts) scores. the task is designed to test a model's ability to generate sentence embeddings that accurately represent sentences regardless of whether or not they contain idiomatic expressions. when evaluating the trained model, we first obtain the sentence embeddings, then we calculate the spearman correlation between the cosine similarity scores of sentence embeddings and the gold labels. the statistics of the corpus are shown in ## system overview we focus on comparing the impact of different training techniques adopted in our system. in this section, we first present the bert-like text encoder, then we introduce several strategies for improving models' robustness. finally, we talk about the design of the cross-attention module and the jointly training way of incorporating supervised signals and unsupervised signals. ## conclusion and future work in this work, we provide an overview of the combined approach to detect and represent multiword expressions. we use infoxlm base model as the text encoder and enhance the model generalization and robustness with exponential moving average (ema) method and the adversarial attack strategy in the subtaska. in the subtaskb, experimental results show that the cross-attention module and the contrastive learning task can considerably improve the performance. finally, we analyze the alignment and uniformity properties to measure the quality of sentence embeddings. future work of our system includes: 1) using the larger pre-trained language models, such as mbert large , infoxlm large . 2) adopting other data augmentation, including token-shuffle, token-cutoff and mix-up.