Lazybob at S em E val-2023 Task 9: Quantifying Intimacy of Multilingual Tweets with Multi-Task Learning (2023)

Abstract: This study presents a systematic method for analyzing the level of intimacy in tweets across ten different languages, using multi-task learning for SemEval 2023 Task 9: Multilingual Tweet Intimacy Analysis. The system begins with the utilization of the official training data, and then we experiment with different fine-tuning tricks and effective strategies, such as data augmentation, multi-task learning, etc. Through additional experiments, the approach is shown to be effective for the task. To enhance the model’s robustness, different transformer-based language models and some widely-used plug-and-play priors are incorporated into our system. Our final submission achieved a Pearson R of 0.6160 for the intimacy score on the official test set, placing us at the top of the leader board among 45 teams.

## introduction intimacy is a fundamental social aspect of language. understanding the dynamics of intimacy in conversation is important for a variety of practical applications. research on intimacy in conversations between people, particularly in the context of chance encounters, such as the phenomenon of the "stranger on the train," can provide valuable insights into human communication and relationships @xcite @xcite . studying the content on different social platform can help researchers to better understand how people establish and develop relationships through conversation. social workers can also benefit from an understanding of how intimacy is established and maintained in conversation to help clients build stronger relationships with others @xcite . the multilingual tweet intimacy task was hosted by the university of michigan and snap inc. this task focuses on predicting the intimacy of tweets in 10 languages @xcite , where 6 lan-guages with training data (english, spanish, portuguese, italian, french, chinese), and 4 zero-shot languages (korean, arabic, hindi, dutch). our submission achieved the first place overall score in these 10 languages. in this paper, we demonstrate the following contributions: (1) we designed a novel multi-language and multi-task training framework that can simultaneously train data from all languages. (2) based on 10 million crawled twitter, book, movie, and reddit data in various languages, we conducted domain pre-training of multiple pre-training models to enhance the model's prediction ability in that domain. (3) during the training process, we also adopted contrastive learning strategies such as group-layerwise learning rate decay (gllrd), adversarial weight perturbation (awp) to enhance the semantic understanding and robustness of language models. (4) for zero-shot language prediction, we tried bidirectional translation data augmentation to enhance the model's prediction ability for small languages in this task. ## background sociologists and linguists have conducted many studies on how people use language to measure the relationships between individuals. by analyzing conversations, researchers can estimate the relationship between people, such as friendly, neutral, or adversarial relationship @xcite @xcite . recently, some scholars have proposed computational models to evaluate the intimacy of a conversation, which greatly reduces the workload of linguistic research @xcite . by training on a large amount of annotated data, the proposed model can learn to better understand the semantics of language and accurately predict the intimacy level in multiple languages. this task is significant for comprehending how language is utilized to convey emotions and relationships, and has practical applications in fields such as psychology, social work, and marketing. ## system overview to better understand the content and semantics of the training and testing sets, which are drawn from twitter posts and comments, we use large language models trained specifically on tweets, for language encoding. we also use additional datasets from twitter and reddit, as well as some from books and movies to pretrain language models. at the same time, we used multiple fine-tuning tricks to improve the model's performance. to make the distribution of the predicted intimacy scores are close to the distribution of the true annotation scores, we designed a multi-task training mode to adjust the distribution of the predicted labels. we also used model fusion methods to generate more robust results. the following section introduce the detail of language model, pretraining method, fine tuning tricks, multi-task setting and ensemble method, etc. were used in this paper. ## experimental setup in this competition, we applied 3 strategy with different languages models and datasets to predict the intimacy score of tweets in 10 languages. based on the organizers' description, this task is more like a classification task since the initial task of the annotators was to label the data into one of the five levels of intimacy on a scale of 1-5. however, the true training data provided is the average value of multiple types of annotation data, which is not a standard multi-level classification. considering the final evaluation metric for this competition is pearson correlation coefficient (pearson r) @xcite , we consider this task as a regression problem for prediction. the goal of the task is to predict a continuous value of intimacy score rather than a discrete class label, and the pearson r evaluates the correlation between the predicted values and the true values. the three strategies are described as below. strategy 1 (single task on official data): the first strategy we used is to treat this task as a pure regression problem, using the raw data provided by the organizer for training. since the data was already cleaned, we only performed some basic text processing on features unique to tweets. we then added an attention layer and fine-tuned the model using various language models. strategy 2 (single task on augmented data): due to limited size of our training dataset, we perform data augmentation through google translation cloud api foot_6 to translate all data into english. we then fine-tuned a well-performing english pretrained language model using the translated data. this approach allowed the model to better handle the linguistic features, particularly in zero-shot tasks. strategy 3 (multi task with pseudo labels): the label distribution from 1 to 5 is unbalanced and has very few labels greater than 3 (see to address this issue, we firstly tried removing the weighted sampler in the batch sampling process. however, this strategy did not work and actually had a negative impact. therefore, we ultimately adopted a multi-task learning approach to train the model. we converted the official labels into pseudo-labels based on their specific numerical values. specifically, we divided the labels into four categories based on four intervals: [1-2), [2, 3), [3, 4), and [4,5]. in multi-task learning, we trained the regression with mse loss and classification with cross entropy loss at a same time. in validation stage, we still only applied the pearson r as the metric. figure 1 shows the label prediction using multi task learning is closer to the annotated distribution rather than single task distribution. 5 result and discussion additionally, we also tried other approaches which not demonstrated in details. for instance, we tried training each language separately, but due to the small size of the training data, the validation score was relatively low. we also tried another multi-task learning approach, where each language was treated as a separate task and trained separately as a regression problem. however, the performance of this approach was even worse than that of the single task, probably because of the lack of training data. finally, to ensure the prediction performance for zero-shot languages, we also tried translating all of the training data into korean and hindi to improve the performance of these two languages. however, the results were not as good as using the multi-language strategy directly, possibly due to the noise introduced by translation. ## conclusion and future work overall, research on intimacy in conversations between people can provide valuable insights into social relationships. this work developed a pretrained and multilingual language model to determine the intimacy level of tweets. the model perform well on both trained language and zero-shot languages. in the future, research can consider adding feature engineering and tuning more deep learning tricks to improve the performance of the model prediction. additionally, this study demonstrated that pretraining a model with domain knowledge can improve its robustness, so it is important to consider pretraining the model with relevant content before using it to predict intimacy scores for specific types of conversations.