N ikkei at S em E val-2022 Task 8: Exploring BERT -based Bi-Encoder Approach for Pairwise Multilingual News Article Similarity (2022)

Abstract: This paper describes our system in SemEval-2022 Task 8, where participants were required to predict the similarity of two multilingual news articles. In the task of pairwise sentence and document scoring, there are two main approaches: Cross-Encoder, which inputs pairs of texts into a single encoder, and Bi-Encoder, which encodes each input independently. The former method often achieves higher performance, but the latter gave us a better result in SemEval-2022 Task 8. This paper presents our exploration of BERT-based Bi-Encoder approach for this task, and there are several findings such as pretrained models, pooling methods, translation, data separation, and the number of tokens. The weighted average ensemble of the four models achieved the competitive result and ranked in the top 12.

## introduction measuring sentence and document similarity is a task that has been studied for many years in the field of natural language processing. one of the applications is to identify whether news articles address the same subject. if news articles can be properly clustered, they can be used for a wide range of purposes, such as recommendation and displaying related articles. semeval-2022 task 8 attempts to tackle this task with multilingual news articles @xcite . nowadays, it is common for this kind of tasks to use transformer-based models like bert @xcite . there are several research directions, including post-processing @xcite , unsupervised learning @xcite , and supervised learning @xcite @xcite . here, since a labeled dataset was provided, we decided to use a supervised learning approach. it is standard practice to combine two sentences as input when dealing with pairwise similarity of sentences in a supervised learning approach with bert @xcite . in contrast, we chose an approach that embeds each sentence separately. figure 1 shows these two approaches, named cross-encoder and bi-encoder to follow the previous research @xcite . this paper describes our system in semeval-2022 task 8. first, we explain the experimental results by adopting the bi-encoder architecture rather than cross-encoder. we also present the following research questions: 1) which pretrained model works well when dealing with multilingual news articles, 2) what kind of pooling method is proper for this task, 3) is it useful for translating the other language into english, and 4) is there some effect of data splitting and max length? our code is available at https://github.com/upura/semeval2022task8-multilingual-news-article-similarity . ## task description semeval-2022 task 8 provides a dataset that contains pairs of news articles. the dataset contains the information like the language and url of each arti- ## system overview the developed system is outlined in figure 3 . the final prediction is calculated by a weighted average of the output of the four neural networks. figure 4 illustrates the base architecture of each neural network. we use a bi-encoder approach where the two texts are entered into separate bert. each input is the combined title and body text of the article. as a pooling method, the representations of the last four cls tokens are concatenated. cls is the token to be attached to the beginning of an input of bert. at the end, the score is given based on the interaction of the two sentence vectors. the rest of this section describes the key points of the system. first, we consider whether cross-encoder or bi-encoder should be used. next, we address the perspectives of the research questions listed in section 1: pretrained models, pooling methods, translation, data splitting, and maximum length. finally, we explain the ensemble of models. ## results this section reports the experimental results that facilitated the design of the system described in the previous section. the validation dataset is extracted from the training dataset. the column named "folds" shows the number of folds in cross validation. that is, in case of folds set 5, 20 % of the training dataset is removed for the validation. each model is trained for seven epochs, and the scores with the best performance on the validation dataset are reported. in our experimental setting, the performance for the validation dataset converged after about five training epochs. the scatter plots of the performance for the vali-dation and evaluation datasets are shown in figure 6 , and it can be observed that they are correlated. it is suggested that the improved performance on the validation dataset is also useful on the evaluation dataset, and that the validation framework works well. the rest of this section describes the result in detail from the same perspective as in the previous section. the models in ## conclusion this paper presented our exploration of bertbased bi-encoder approach for semeval-2022 task 8. the experiment showed that bi-encoder architecture worked better than cross-encoder. there are several findings, such as pretrained models, pooling methods, translation, data separation, and the number of tokens. the exploration of these different variants led to the creation of several diverse models. finally, a weighted average ensemble of the four models achieved the competitive result.