OPI at S em E val-2023 Task 9: A Simple But Effective Approach to Multilingual Tweet Intimacy Analysis (2023)

Abstract: This paper describes our submission to the SemEval 2023 multilingual tweet intimacy analysis shared task. The goal of the task was to assess the level of intimacy of Twitter posts in ten languages. The proposed approach consists of several steps. First, we perform in-domain pre-training to create a language model adapted to Twitter data. In the next step, we train an ensemble of regression models to expand the training set with pseudo-labeled examples. The extended dataset is used to train the final solution. Our method was ranked first in five out of ten language subtasks, obtaining the highest average score across all languages.

## introduction intimacy can be expressed in language in a variety of ways. the degree of intimacy in an utterance is indicated by both thematic and stylistic features, often subtle and difficult to quantify automatically. one of the most apparent aspects of intimacy is selfdisclosure. sharing personal details about oneself or one's life can create a sense of intimacy. this information can relate to both factual as well as emotional spheres, addressing matters such as feelings, goals, dreams, or fears. other features which may indicate intimacy involve the use of certain types of terms or phrases, especially those creating a sense of closeness and connection between the author and the reader. automatic identification and quantification of intimacy in natural language is a challenging problem, with a difficulty similar to automatic emotion recognition. both tasks involve measuring inherently subjective and ambiguous aspects. intimacy can be influenced by a variety of factors such as the context, culture, and personal experiences of the individual. intimacy analysis in multilingual text presents additional challenges due to differences in language structure, cultural norms, and expression of emotions across different languages. so far, this topic has not received much attention. @xcite conducted an intimacy analysis of questions from social media, books, and movies. in the study, they created a question dataset and examined the performance of automatic intimacy prediction using methods such as logistic regression and transformer-based language models @xcite . the multilingual intimacy analysis task was a part of semeval-2023. the goal of the task was to measure the intimacy of twitter posts in ten languages @xcite . the organizers provided a training set of 9,491 tweets and a test set of 13,797 tweets, in which each sample was annotated with an intimacy score ranging from 1 to 5. the training data included texts in only six of the ten languages, while the evaluation was performed on all ten. the task thus tested the performance of the submitted solutions for both standard fine-tuning and zeroshot prediction. the metric selected to evaluate the solutions was the pearson correlation coefficient. the systems were ranked according to the correlation value for the entire test set, as well as for subsets in each language. this paper presents our solution to the multilingual tweet intimacy analysis shared task. the proposed approach is a combination of domain adaptation and semi-supervised learning. we first train a transformer language model on a large corpus of multilingual tweets and then create an ensemble of regression models to expand the training set with pseudo-labeled examples. our method achieved the best score in five out of ten language subtasks, the highest number among all participants. according to the pearson correlation calculated for the entire dataset, the proposed method was ranked third. ## system description our solution for the multilingual tweet intimacy analysis task can be summarized in the following train valid train valid expanded train [mask] are how today you step 1: in-domain pre-training step 2: training ensemble for dataset expansion step 3: training ensemble on the expanded dataset first, we further pre-train an existing language model on a large corpus of tweets. next, we train an ensemble, which is used to label additional data. the expanded dataset is utilized to create a new set of models, which are the final solution to the task. figure 1 shows our approach on a diagram. in the following sections, we explain the steps of this process in detail. ## experiments and results this section contains a discussion of the official results of the multilingual tweet intimacy analysis task. we also conducted post-evaluation experiments using the gold labels provided by the organizers to analyze the results obtained by models different from the submitted solution. ## conclusion in this paper, we described our solution for the multilingual tweet intimacy analysis shared task. our system placed first in five out of the ten languages. the paper demonstrated a method for combining domain adaptation with semi-supervised learning. as part of our research, we trained and published a multilingual language model using a corpus of 156 million tweets. building on this model, we fine-tuned an ensemble of regressors to extend the training dataset with pseudo-labeled examples. we also performed additional experiments, comparing the model to other publicly available multilingual models, in which our method proved to be more effective in predicting intimacy scores.