HFL at S em E val-2022 Task 8: A Linguistics-inspired Regression Model with Data Augmentation for Multilingual News Similarity (2022)

Abstract: This paper describes our system designed for SemEval-2022 Task 8: Multilingual News Article Similarity. We proposed a linguistics-inspired model trained with a few task-specific strategies. The main techniques of our system are: 1) data augmentation, 2) multi-label loss, 3) adapted R-Drop, 4) samples reconstruction with the head-tail combination. We also present a brief analysis of some negative methods like two-tower architecture. Our system ranked 1st on the leaderboard while achieving a Pearson’s Correlation Coefficient of 0.818 on the official evaluation set.

## introduction in task 8 @xcite , we are expected to assess the similarity of pairs of multilingual news articles as shown in the foundation model @xcite we choose is xlm-roberta (xlm-r) @xcite which has been proved to be a powerful multilingual pre-trained language model compared with other models like mbert @xcite and it can process all the languages existing in task 8. based on that, a great variety of strategies have been tested along with our exploration like data augmentation (da), head-tail combination, multilabel loss, adapted r-drop, etc. through this task, we realized the importance of data quality and efficient training schemes in such a cross-lingual setting. by struggling to improve the richness of the data and find out what methods are effective when training such a similarity assessment model, our system foot_0 ranked 1st in this competition. ## system overview our baseline system is simply providing a pair of articles to xlm-r and regressing its output from @xcite token to the manually annotated similarity score by training with mean squared error (mse). all the optimization methods discussed below are applied based on this architecture and the overall framework of our final system is illustrated in figure 1. after training with all the positive strategies, we then made an ensemble of the best models on each fold for the final prediction. ## conclusion by deploying various optimization methods, including data augmentation, head-tail combination, multi-label loss, adapted r-drop and adding extra linear layers, we built a relatively strong system for assessing the similarity between a pair of news articles in multilingual and cross-lingual settings and ranked 1st in the competition with a pearson's cc of 0.818 on the official evaluation set. in the future, apart from enriching the training data, we are also supposed to analyze the languages individually and try to leverage the exclusive rules or features of each language rather than relying too heavily on general pre-trained models to further boost the model performance, especially on those minority languages.