H ua AMS at S em E val-2022 Task 8: Combining Translation and Domain Pre-training for Cross-lingual News Article Similarity (2022)

Abstract: This paper describes our submission to SemEval-2022 Multilingual News Article Similarity task. We experiment with different approaches that utilize a pre-trained language model fitted with a regression head to predict similarity scores for a given pair of news articles. Our best performing systems include 2 key steps: 1) pre-training with in-domain data 2) training data enrichment through machine translation. Our final submission is an ensemble of predictions from our top systems. While we show the significance of pre-training and augmentation, we believe the issue of language coverage calls for more attention.

## introduction in the recent few years, there has been a growing interest towards automating news understanding tasks thanks to the continuous demand by downstream applications. one of the most important aspects of news understanding is identifying news articles that cover the same stories. grouping such similar articles can be useful in multiple application scenarios including news recommendation, news stories analysis, news retrieval and ranking amongst others. task 8 in semeval 2022 @xcite provides an experimental setup to address and identify the challenges of assessing the similarity between two news articles. unlike standard document similarity tasks @xcite , this task is focused on a more challenging multilingual setting where the systems are not only expected to evaluate pairs of long text articles in the same language but in different languages as well. moreover, the task assumes that accurately identifying articles that share the same story cannot be solely captured by textual similarity since the underlying similarity function is hypothetically a combination of a set of other features like time, geo-location, mentions of named entities and narratives. participating teams are required to provide a similarity score for each pair of news articles. the scores should range between 1 and 4 where 1 indicates that two stories are almost identical and 4 indicates no similarity at all. the systems are evaluated based on the pearson's correlation with ground truth scores which are provided in a test set annotated by human evaluators. in our submission, we hypothesize that the subdimensions (that are assumed to be story similarity predictors by the task organizers) do not have to be independently modeled and present an approach that assumes that such sub-dimensions can be represented in the model's latent space while directly optimizing to learn article similarity scores. formally speaking, we model the similarity task as a supervised regression problem to optimize the similarity score between two news articles. to test our hypothesis, we experiment with different pre-trained language models and evaluate multiple methods of boosting the model's performance via domain pre-training and data augmentation. our final submission ranked 8 out of a total of 32 teams in the official rankings with a pearson correlation of 0.771 -a score difference of 0.047 compared to the top ranked submission. this paper is organized as follows: in section 2 we describe the official datasets as well as the external datastes that were used. in section 3 we present our baselines and systems' setups. section 4 presents our data splits and model training specifics. we show and analyse our results in section 5, then we conclude and discuss future work in section 6. ## datasets the used datasets are categorized into two categories: 1) task datasets which are provided by the task organizers and 2) external datasets that are used as additional assets for model training. ## system overview all of our systems are solely trained on the text descriptions of input articles as features and their similarity score as the target variable. all the presented systems except for the baseline system 3.1, are based on xlm-roberta (xlm-r). this choice is made based on the fact that it's a multilingual model that is pre-trained on large amounts of data spanning 100 languages and performs competitively on several cross-lingual transfer tasks @xcite . the general architecture of our systems itself is kept relatively simple i.e. a language model with a regression head on top. ## experimental setup 4.1 data setup to be able to run our model selection experiments and validate our hyper-parameter settings, a standard split of 80:10:10 is applied to the task dataset 2.1 to split it into train, validation and held-out sets. this setting resulted in a training set size of 3719 samples, validation set size of 464 and a held-out set size of 460 samples. the validation set was used for hyper-parameter tuning and the held-out set was used for system comparisons since there were no datasets provided for such purposes by the task organizers. our experiments that leveraged data augmentation techniques made use of a total of 12707 training samples. to avoid any data leakage, we only used samples from the train set to augment the data. ## results and analysis during the development phase, we used the fixed held-out set to compare the performance of different systems. when the evaluation phase ended, the gold labels were made available for the evaluation set and thus we re-evaluated all our systems using that set as well. the results on the held-out set are shown in our submitted system is es 2 s 3 ( our per language evaluation ( ## discussion in this paper we described our submissions to the news similarity task in semeval 2022. our models showed competitive performance by leveraging pretrained language models and showed that further improvements can be gained by the use of domain pre-training and data augmentation using machine translation. due to the competition time limits such domain pre-training and translation experiments were conducted on relatively small datasets and we did not manage to experiment with a model that combines both additions. we believe that scaling these approaches by using huge amounts of monolingual data across different languages is potentially a direction that is worth exploring. we see improvement posibilities when it comes to modeling as well. in the early stages of our experimentation we tried a contrastive learning based approach similar to the works done by @xcite though, initial results were not promising and we decided to discard this direction, we believe that further efforts can be fruitful. we've also experimented with explicit modeling of named entities within our models without a positive outcome however this could be due to the fact that we used a very simple string matching approach for named entities identification. another modeling aspect is the train/evaluation language distribution modeling. given that the distribution of evaluation language pairs are available, one could leverage this to improve the model optimization process. finally, in this exploratory work we haven't made use of any available article related meta-data which can have strong predictive power of article similarity. examples include url normalization to identify parallel articles in different languages, domain and country information among other features. we leave out these territories to be explored in future works.