YNU - HPCC at S em E val-2022 Task 2: Representing Multilingual Idiomaticity based on Contrastive Learning (2022)

Abstract: This paper will present the methods we use as the YNU-HPCC team in the SemEval-2022 Task 2, Multilingual Idiomaticity Detection and Sentence Embedding. We are involved in two subtasks, including four settings. In subtask B of sentence representation, we used novel approaches with ideas of contrastive learning to optimize model, where method of CoSENT was used in the pre-train setting, and triplet loss and multiple negatives ranking loss functions in fine-tune setting. We had achieved very competitive results on the final released test datasets. However, for subtask A of idiomaticity detection, we simply did a few explorations and experiments based on the xlm-RoBERTa model. Sentence concatenated with additional MWE as inputs did well in a one-shot setting. Sentences containing context had a poor performance on final released test data in zero-shot setting even if we attempted to extract effective information from CLS tokens of hidden layers.

## introduction meaning of sentence could be captured by compositionality of word representations. however, there widely exists potentially idiomatic phrases in different languages, which are multiword expressions (mwes) with idiomatic and literal meanings. therefore, representation of idiomatic phrases is not directly compositional. a previous study has shown that the representation of idiomatic phrases by contextual models, such as elmo @xcite , bert @xcite and some of its variants, are not accurate @xcite . it will be challenging to represent the mwes correctly in the downstream tasks @xcite . semeval-2022 task 2, multilingual idiomaticity detection and sentence embedding @xcite , involves english (en), portuguese (pt), and galician (gl). this task includes two subtasks and each subtask contains two settings: the remainder of this paper is organized as follows. in section 2, we describe the structures of model and system. the details about data and implementation and comparative results are presented in section 3. finally, a conclusion is drawn in section 4. ## experiments 3.1 dataset there includes en, pt, and gl in datasets. gl is only provided in test dataset and one-shot training data, aiming to test models' ability to transfer learning across languages. besides, modern galician (gl) is part of the west iberian languages group, which belongs to a family of romance languages including the portuguese. a brief intro- x sentence_embeddings1 sentence_embeddings2 cosine_sim cosine_sim cross-entropy loss ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... s1 s2 sn-1 sn s1 s2 sn-1 sn ... ... ... ... ... s1 s2 sn-1 sn ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... duction about datasets describes as follows, and more details are in the task description paper (tayyar madabushi et al., 2022). in the subtask a, training data provided includes zero-shot and one-shot data. a label of 0 indicates idiomatic and a label of 1 indicates non-idiomatic. zero-shot training data combined with the context, sentences preceding and succeeding the one containing the idioms, are used as training sentences in the zero-shot setting. in the one-shot setting, both the zero-shot and one-shot data are used to train, which exclude the context but add the mwes as an additional feature. sentence was concatenated to mwe and separated them by sep. train split of stsbenchmark datasets in english and assin2 datasets in portuguese are as training dataset in the subtask b pre-train setting. development and test datasets consist of sentence pairs, some of whom contains idiomaticity and other has no idiomaticity, which replaced by non-idiomatic paraphrases. they will be computed a score (cosine similarity) after model's outputing. in the subtask b fine-tune setting, development and test datasets are the same as the pre-train setting. train datasets provided contain en and pt training examples with type of one-shot and zeroshot totally, and also contain some gl training one-shot examples. for integrity of idiom tokens, mwes in datasets were added into the tokenizer of model. training data have positive and negative types. the positive examples mean that sentence with an idiom (s m w e ) is the same as the sentence in which the idiom has been replaced by a phrase that correctly represents the meaning of the idiom in context (s c ). so their sts score are equal to 1 (sim(s m w e , s c ) = 1). for the negative examples, a sentence with an idiom and the same sentence in which the idiom has been replaced by a phrase that incorrectly represents the meaning of the idiom in context (s i ) should have a low sts score. the score is approximately equal to the sts score between a sentence where the idiom has been replaced by a phrase that correctly represents its meaning and one wherein it incorrectly represents the meaning (sim(s m w e , s i ) = sim(s c , s i )). ## conclusions herein, we discuss and experiment the methods we used in semeval-2022 task 2. we participated in two subtasks, idiomaticity detection and representation of idiomaticity. each of subtasks including two settings, achieved the 19th, 6th, 5th, and 1th places in the final test sets, respectively. our results showed that the idea introduced contrastive learning in representation of idiomaticity achieved good results, but methods of zero-shot setting in idiomaticity detection did not well and were lack of ability of generalization. we intended to explore and improve the performance of zero-shot and few-shot learning further in future work.