tmn at S em E val-2023 Task 9: Multilingual Tweet Intimacy Detection Using XLM - T , G oogle T ranslate, and Ensemble Learning (2023)

Abstract: The paper describes a transformer-based system designed for SemEval-2023 Task 9: Multilingual Tweet Intimacy Analysis. The purpose of the task was to predict the intimacy of tweets in a range from 1 (not intimate at all) to 5 (very intimate). The official training set for the competition consisted of tweets in six languages (English, Spanish, Italian, Portuguese, French, and Chinese). The test set included the given six languages as well as external data with four languages not presented in the training set (Hindi, Arabic, Dutch, and Korean). We presented a solution based on an ensemble of XLM-T, a multilingual RoBERTa model adapted to the Twitter domain. To improve the performance on unseen languages, each tweet was supplemented by its English translation. We explored the effectiveness of translated data for the languages seen in fine-tuning compared to unseen languages and estimated strategies for using translated data in transformer-based models. Our solution ranked 4th on the leaderboard while achieving an overall Pearson’s r of 0.5989 over the test set. The proposed system improves up to 0.088 Pearson’s r over a score averaged across all 45 submissions.

## introduction intimacy is a significant social aspect of language, which helps to explore existing social norms in various contexts @xcite . the concept of intimacy describes how an individual relates to his addressee in their perceived interdependence, warmth, and willingness to personally share @xcite ). an automatic evaluation of intimacy in language provides us with a clearer picture of social interactions and different linguistic strategies. this paper describes a system developed for the semeval-2023 task 9: multilingual tweet intimacy analysis @xcite . the purpose of the task was to predict the intimacy of tweets. the test set contained tweets in the languages presented (seen) and not presented (unseen) in the training set. the main challenge of the task was the zeroshot intimacy prediction performance for unseen languages. inspired by the recent success of transformerbased models in multilingual tasks @xcite , we evaluate the performance of bert (bidirectional encoder representations from transformers) @xcite and its modifications. in our experiments, we investigated the impact of translation to improve the zero-shot performance. we have found that the joint use of original and translated texts increases the model performance for unseen languages in comparison with the separate use of original or translated data. the paper is organized as follows. section 2 presents the task and the dataset provided by the task organizers. our methods are presented in section 3. in section 4, we describe the experimental setups that we used during the development phase of the competition. we discuss our results in section 5. finally, section 6 concludes this paper. ## task description the task aims to predict the intimacy of tweets in ten languages. the organizers presented a training set containing tweets in six languages (english, spanish, italian, portuguese, french, and chinese) annotated with intimacy scores ranging between 1 and 5 where 1 indicates "not intimate at all" and 5 indicates "very intimate". the model performance was evaluated on the test set consisting of tweets in the given six languages and the four unseen languages that were not included in the training data (hindi, arabic, dutch, and korean). the total amount of entries in the training set is 9,491. the size of the test set is 3,881. pearson's r was utilized as an evaluation metric. the dataset statistics are presented in ## experimental setup in this section, we describe the experimental setup utilized during the development phase of the competition. in this phase, the organizers provided participants with a training set containing 9,491 entries. we shuffled the provided data with a fixed random seed and divided it into training and validation subsets in a ratio of 70:30. therefore, the training and validation subsets consisted of 6,643 and 2,848 entries respectively. to estimate the performance for unseen languages, we consistently excluded one language from the training set. therefore, the model was fine-tuned on the five remaining languages and evaluated on the unseen language. to evaluate the results during the development phase, we utilized two metrics. the first one was pearson's r which was an official evaluation metric. the second metric we used was the mean squared error (mse) which estimates the mean squared distances between actual and predicted values.