S ea_and_ W ine at S em E val-2023 Task 9: A Regression Model with Data Augmentation for Multilingual Intimacy Analysis (2023)

Abstract: In Task 9, we are required to analyze the textual intimacy of tweets in 10 languages. We fine-tune XLM-RoBERTa (XLM-R) pre-trained model to adapt to this multilingual regression task. After tentative experiments, severe class imbalance is observed in the official released dataset, which may compromise the convergence and weaken the model effect. To tackle such challenge, we take measures in two aspects. On the one hand, we implement data augmentation through machine translation to enlarge the scale of classes with fewer samples. On the other hand, we introduce focal mean square error (MSE) loss to emphasize the contributions of hard samples to total loss, thus further mitigating the impact of class imbalance on model effect. Extensive experiments demonstrate remarkable effectiveness of our strategies, and our model achieves high performance on the Pearson’s correlation coefficient (CC) almost above 0.85 on validation dataset.

## introduction the concept of intimacy dates from social psychology, which is of great value to indicate degree of closeness in the relationships between people. and analysis of social intimacy is beneficial to unveil complicated mechanisms of social interaction. for another, as verbal and textual communications are universally a powerful means to express the affection and nurture social relationships, language plays an indispensable role in unraveling the intimacy between individuals. thus analyzing textual contents of social media has a potential to unearth the intimacy information in the field of natural language processing @xcite . although feasible the task looks like, it faces two core challenges, how to quantify the intimacy of language and how to gauge it according to textual information by computational modelling. to tackle such challenges, @xcite first proposed a computational framework to measure the intimacy of language. however, despite demonstrating the construction of textual intimacy, this work was only based on the form of questions. and there are still gaps with the real situations that contain other forms of language, such as statements and dialogues between people. to enlarge the span of applications and improve the generalization performance of language models, htt proposed a multilingual textual intimacy dataset covering tweets in 10 languages, including english, spanish, italian, portuguese, french, chinese, as well as hindi, arabic, dutch and korean, which has been set as the official dataset in semeval-2023: task 9 @xcite . overall, works on textual intimacy remain scarce. in task 9, we are required to predict the intimacy of textual tweets on the official released dataset. so far, there are several multilingual pre-trained language models, including m-bert @xcite , xlm-r @xcite , erine-m @xcite , which can be utilized to handle multilingual regression tasks. in our work, we adopt xlm-r @xcite model to deal with the task. and its contributions can be summarized as follows: 1) aiming at a significant challenge of class imbalance in the raw training dataset, we utilize a data augmentation technique fulfilled by machine translation to augment samples of the minority. 2) we introduce a new loss function, focal mse loss, to the optimization process to further weaken the impact of the majority samples. 3) experiments demonstrate the effectiveness of our strategies, and our model achieves high performance on pearson's cc almost above 0.85 on validation dataset. on the official test set, our model gets an average score of 0.65 on 6 languages included in the training set, which ranking 31st overall on the leaderboard, and gets an average score of 0.38 on 4 new languages that are not appearing in the training set, which ranking 27th overall. in the official released training dataset @xcite , there are 9491 textual tweets in 6 languages, including english, spanish, italian, portuguese, french, and chinese. and they are all sampled from dialogues on social media tweet from 2018 to 2021 and annotated by scores ranging from 1 to 5, where a higher score implicates more intimacy between users. then we are required to predict the intimacy scores according to series of given unlabeled texts. details of the dataset are shown in ## experimental setup to validate the effectiveness of our model and strategies, we conduct extensive experiments both on the raw data and the dataset with data augmentation. during the training process, we make attempts at 5-fold and 10-fold cross validations, as well as training without cross validation, respectively. after k-fold training, the model with the best score of the k models will be trained again with dataset including training set and validation set, and examined by the test dataset split in advance to get a final score. when it comes to the loss function, we also compare the performances of the original mse loss and our focal mse loss. more details are delineated as follows. ## results as shown in ## conclusion in task 9, we fine-tune xlm-r pre-trained model to figure out the multilingual regression task of predicting the textual intimacy. we recognize severe class imbalance in the raw dataset as a primary obstacle that impede the attempts of the model to achieve high performance. to tackle this challenge, we conduct data augmentaion through machine translation provided by google translator to enlarge the scale of the minority classes. meanwhile, we introduce a new loss function, focal mse loss, to further punch above the weight of hard samples on the total loss. and extensive experiments demonstrate the effectiveness of our model and strategies by high performance on pearson's cc. however, there is no denying that overfitting on the augmented dataset still exist, and we intend to explore more solutions to such limitation in the future.