Team Innovators at S em E val-2022 for Task 8: Multi-Task Training with Hyperpartisan and Semantic Relation for Multi-Lingual News Article Similarity (2022)

Abstract: This work represents the system proposed by team Innovators for SemEval 2022 Task 8: Multilingual News Article Similarity. Similar multilingual news articles should match irrespective of the style of writing, the language of conveyance, and subjective decisions and biases induced by medium/outlet. The proposed architecture includes a machine translation system that translates multilingual news articles into English and presents a multitask learning model trained simultaneously on three distinct datasets. The system leverages the PageRank algorithm for Long-form text alignment. Multitask learning approach allows simultaneous training of multiple tasks while sharing the same encoder during training, facilitating knowledge transfer between tasks. Our best model is ranked 16 with a Pearson score of 0.733.

## introduction over the last decade, english has been one of the most dominant languages on the internet. however, the number of non-english websites is rapidly increasing. from 2001-to 11, online use of english increased at a slower rate than that of spanish, chinese, etc. approximately 4 billion people connect to the internet every day, but only half of them access web pages written in english @xcite . this creates a severe problem regarding verifying the integrity of the documentation because most systems in use are enhanced with english as the medium of information delivery. semeval has conducted similar tasks on sentence-level semantic textual similarity in the past. the semeval 2017 task 1 @xcite dealt with finding the similarity between sentence pairs of both monolingual and cross-lingual nature. ecnu @xcite was the best model that used feature engineering and deep averaging network (dan) @xcite . furthermore, language-agnostic representation for sentence similarity described in @xcite uses meaning embedding to estimate the cross-lingual sentence similarity without using human annotations. it improves the performance of any pre-trained multilingual sentence encoder, even in low-resource languages, with a few thousand parallel sentence pairs. inspired by @xcite , we explored the concept of semantically aligned multilingual sentence embedding. it covers the biases induced by monolingual similarity evaluation and multilingual sentence retrieval to generate language-aware embeddings. this method aligns semantic structures across different languages and uses a teacher network to distill the knowledge of pivot languages, thus achieving state-of-the-art sts 2017 multilingual corpora. our model is inspired by a multitask training approach using a bert-based encoder. we use multiple subtasks to improve the overall accuracy score. our model uses a machine translation model to cope with the articles' linguistic diversity. additionally, since the average size of each article in the training data is close to 512 tokens, we use a text ranking algorithm that can capture the long-range sentence-level similarity between two documents. the next sections provide a more in-depth exposition. ## task description the shared task emphasizes finding the similarity of multi-lingual news articles irrespective of the style of writing, political spin, tone, or any other more subjective "design decision" imposed by a medium/outlet. it gives participants access to a cross/multi-lingual dataset that spans over ten languages, including english, german, french, italian, polish, russian, chinese, turkish, arabic, and spanish. the dataset includes an overall matching score between two different news articles. additionally, the dataset consists of other dimensionality scores, such as geo-location, time, shared entities, and shared narratives (see ## system overview our system pipeline can be decomposed into four modules i.e., extraction, translation, text ranking, and multitask training. ## experimental setup this section describes various hyper-parameters we use in data preprocessing and training. after scraping all the valid urls, we could access 3651 pairs of articles for training, 408 for validation, and 4902 for a test. for training, we implement simple transformer models. we use deberta @xcite as the pre-trained language model with a batch size of 4. we found that the model performs better when the initial learning rate is 10 -6 to 10 -5 . we use the adamw optimizer. we use google colab with a tesla v100 gpu for various experiments. we apply various python libraries like pytorch, transformers, numpy, and pandas to implement our multitask learning model. ## performance analysis in this section, we analyze the performance of several components of our system and compare different schemes for representing the document and the techniques used. since the commencement of this task, our team leaned toward using a multitask learning strategy because it seemed to be the ideal fit for this problem. to do so, we ideated with several subtasks and experimented with our base encoder architecture. after experimentation with various combinations of subtasks, as mentioned in ## results we use the pearson score to evaluate semeval 2022 task 8, which measures the linear relationship between the predicted and ground truth values. like other correlation coefficients, the pearson score varies between -1 and +1, with 0 implying no correlation. correlations of -1 or +1 imply an exact linear relationship. the dataset available in the evaluation phase of the task lacked features like geography, entity, etc., which were present in the training phase. furthermore, the test dataset has an additional set of languages not present in the training dataset. this test dataset ensured that the model trained by the participants was an accurate multilingual model. as stated in the second row of next, model type model subtasks pearson score (on test set) roberta semantic textual similarity + stance detection 0.724 phrase detection + stance detection 0.730 deberta semantic textual similarity + hyperpartisan detection 0.733 ## conclusion & future work our model performs well on english, spanish, and french data while falling short on german. even though the german data is the second largest, the biggest problem is that there is a lot of data to consider. the model could not correctly address each attribute of the article, resulting in underperformance. some of the future work we anticipate to do which can increase the performance of our model are stated below: