YNU - HPCC at S em E val-2022 Task 8: Transformer-based Ensemble Model for Multilingual News Article Similarity (2022)

Abstract: This paper describes the system submitted by our team (YNU-HPCC) to SemEval-2022 Task 8: Multilingual news article similarity. This task requires participants to develop a system which could evaluate the similarity between multilingual news article pairs. We propose an approach that relies on Transformers to compute the similarity between pairs of news. We tried different models namely BERT, ALBERT, ELECTRA, RoBERTa, M-BERT and Compared their results. At last, we chose M-BERT as our System, which has achieved the best Pearson Correlation Coefficient score of 0.738.

## introduction in the field of natural language processing (nlp), how to measure the similarity of two texts has been a topic of interest among researchers for decades. text similarity measures play an increasingly important role in text related research and applications in tasks such as information retrieval, text classification, document clustering, topic detection, topic tracking, questions generation, question answering, essay scoring, short answer scoring, machine translation, text summarization and others @xcite . it is widely known that text is a high-dimensional semantic space, hence how to abstractly decompose it so that we can quantify its similarity from a mathematical point of view has become the focus for many researchers. there are three methods to measure text similarity: one is the traditional method based on keyword matching, such as n-gram similarity; the second is to map the text to the vector space, and then use the cosine similarity and other methods; the third is the method of deep learning, such as the deep learning semantic matching model dssm based on user click data, convnet based on convolutional neural network, and the current state-of-art siamese lstm and other methods. however, since the introduction of bidirectional encoder representations from transformers (bert) @xcite , the accuracy and training efficiency in both text classification and sequence labeling have reached new heights. semeval 2022 task 8 is a multilingual news article similarity task(x. chen, zeynali, camargo, flöck, gaffney, grabowicz, hale, jurgens, and samory 2022). there are mainly 3 difficulties in the task compared with regular text similarity task: (1) the task is interested in the real worldhappenings covered in the news articles, not their style of writing, political spin, tone, or any other more subjective design. therefore, the system built by participant should neglect the subjective part of the text and focus on objective part only;(2) there were over six different languages in both training and test dataset, and some of test data set were composed of multilingual text pair to test the multilingual ability of participating system; (3) the test dataset contains new languages which have never appeared in training data. this situation disguisedly reduces the training data required to train the model. in this paper, we primarily present a deep learning system for the semeval-2021 task 8: multilingual news article similarity. since there are not any subtasks in the semeval 2022 task 8, our system will focus on calculating the overall similarity only. our approach is based on transformers, which is a classic nlp model proposed by google's team in 2017 @xcite . we fine-tuned pre-trained masked language models namely bert, albert, electra, roberta and m-bert, which are all based on transformers. we compared their performance at the task, then picked best of them as our system. experimental results show that most of the transformers based model are valid in the field of text similarity @xcite . however, when it comes to multilingual text, there is a clear drop figure 1 : the example of the given training data in results for all models. among these models, m-bert has achieved the best score with the pearson correlation coefficient of 0.738. that is why we chose m-bert as our system at last. the implementation for our system is made available via github foot_0 . the rest of the paper is organized as follow. section 2 describes the specific requirements of the task and dataset. section 3 describes the details of the transformers model used in our system. section 4 presents the experimental results. finally, the results and conclusions are presented in section 4 and 5. ## task overview the task organizers have provided training and test dataset for the task. the details of this task are given below, and some examples are shown in the figure 1 . ## system description we use the transformer based pre-trained model as solution to accomplish the task. as shown in figure 2 , the system we built contains a tokenizer, a model layer, fully connected layer and mean squared error function. the mean squared error is loss function of our system. the model layer represent a transfomers based pre-trained model, it will be replaced by bert or any model mentioned above to compare their effect on task 8. the rest of the section will describe the details of every part of the system and their mechanics. ## conclusion in this paper, we described our deep learning models for the multilingual text similarity task semeval-2022 shared task 8. the best pearson's correlation score we got was 0.738. we showed that the transformer based approaches is valid in the field of multilingual text similarity. however, our system is far from perfect, lots of possible improvement can be implemented in the current model. we would like to further explore how to improve it, and employ more interesting methods in the task.