TCU at S em E val-2022 Task 8: A Stacking Ensemble Transformer Model for Multilingual News Article Similarity (2022)

Abstract: Previous studies focus on measuring the degree of similarity of textsby using traditional machine learning methods, such as Support Vector Regression (SVR). Based on Transformers, this paper describes our contribution to SemEval-2022 Task 8 Multilingual News Article Similarity. The similarity of multilingual news articles requires a regression prediction on the similarity of multilingual articles, rather than a classification for judging text similarity. This paper mainly describes the architecture of the model and how to adjust the parameters in the experiment and strengthen the generalization ability. In this paper, we implement and construct different models through transformer-based models. We applied different transformer-based models, as well as ensemble them together by using ensemble learning. To avoid the overfit, we focus on the adjustment of parameters and the increase of generalization ability in our experiments. In the last submitted contest, we achieve a score of 0.715 and rank the 21st place.

## introduction providing computer the ability to understand the abstract meaning of real world is a fundamental tasks. given a pair of news articles, this task seek to evaluate the semantic similarity between them, which focuses on the real world-happenings covered in the news articles. it's a regression problem for measuring similarity of multilingual texts. previous studies measured the similarity between texts by using traditional machine learning methods, such as using support vector regression (svr) @xcite . recently many deep learning methods came out, such as pre-trained model. it had attracted the interest of researchers and had shown good result. for example, exploring bidirectional encoder representations from transformers (bert), xlnet and robustly optimized bert approach (roberta) and finally got a good ranking @xcite . and a new hybridized approach using weighted fine-tuned bert feature extraction with siamese bi-lstm model has been implemented. it is employed for determining question pair sets using semantic-textsimilarity from quora dataset @xcite . these novel deep learning methods have performed well. in this study, we explored some transformerbased models. we had employed bert @xcite , roberta @xcite , a lite bert for self-supervised learning of language representations (albert) @xcite , dis-tilbert @xcite , besides this, we used them as base models, merged them together with ensemble learning, and the prediction result is used as a new training set, and then svr is used as the meta model. the training set generated by the base model is put into the meta-model. afterwards, the final result is predicted by the meta-model. the advantage of the pre-trained model is that the upstream corpus has already trained the parameters of the model well. we only need to fine-tune it, and we don't need a huge training set for training @xcite . the rest of this paper is organized as follows. section 2 describes all the models which are used for measuring similarity between sentence pairs. experimental results are summarized in section 3. conclusion is drawn in section 4. ## model description this section will describe what models we have used, and how they organized. because of the attention mechanism, pre-trained model had been made a huge success in nature language processing (nlp). we use transformer-based models such as bert, roberta, albert, distilbert to produce hidden representations. then, a stacking ensemble strategy was used to ensemble the results. the details are presented as follow. sentence pair sentence x sentence y ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... stacking ensemble bert roberta albert distilbert [cls] … x1 xn y1 e [cls] h [cls] [sep] ym … e[sep] h[sep] … … … … 1 e x e n x 1 e y e m y 1 x h n x h 1 y h m y h classification sentence pair sentence x sentence y ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... stacking ensemble bert roberta albert distilbert ## experimental results in this section, we will describe how the whole experimental part is done, and the main focus will be on the part that implements the model. the experimental part will be divided in to 5 parts as follows. ## conclusions in this paper, we are participating in semeval-2022 task 8 @xcite . in this task, we perform regression prediction on the similarity of multilingual news articles, and we use various methods such as bert, albert, roberta, dis-tilbert, and the stacking method built with them as the base model. the model we proposed can effectively predict this task. among the multiple models we submitted, the bert model we finally submitted achieved the best score with a score of 0.715, ranking 21st in the leaderboard. at present, in terms of deep learning, the processing methods of multilingual texts have not been widely popularized. so in the future, we hope to go further in the processing of multilingual texts.