Arizonans at S em E val-2023 Task 9: Multilingual Tweet Intimacy Analysis with XLM - T (2023)

Abstract: This paper presents the systems and approaches of the Arizonans team for the SemEval 2023 Task 9: Multilingual Tweet Intimacy Analysis. We finetune the Multilingual RoBERTa model trained with about 200M tweets, XLM-T. Our final model ranked 9th out of 45 overall, 13th in seen languages, and 8th in unseen languages.

## introduction intimacy is an integral part of human relationships and an essential component of language @xcite . pei and jurgens also propose that intimacy can be modeled with computational methods and that analyzing textual intimacy can potentially reveal information about social norms in different contexts. with social media taking an increasingly large part in people's lives, intimacy has found itself a new platform for display. in these digital social contexts', twitter is an important source for analyzing textual intimacy. to further promote computational modeling of textual intimacy, @xcite make use of publicly available twitter data to create and annotate a textual intimacy dataset called mint from the following 6 languages: english, spanish, french, portuguese, italian, and chinese. they also annotate a smaller set of tweets in dutch, hindi, korean, and arabic to allow for zero-shot prediction experiments. semeval 2023 task 9: multilingual tweet intimacy analysis @xcite , asks participants to predict the intimacy of tweets in the aforementioned 10 languages. in this paper, we explain our approach to intimacy analysis: finetuning xlm-t, the multilingual roberta model trained with about 200m tweets. to improve performance we experimented with different hyperparameters and changed the training data label distribution by duplicating tweets with certain labels. we discuss the issues we ran into while working on our model, such as the imbalanced range of intimacy scores in the provided training dataset. our final submission ranked 9 th out of 45 overall, 13 th in seen languages, and 8 th in unseen languages. we release our code at https: //github.com/beyzabozdag/tweet-intimacy . ## background intimacy is an essential part of language, however resources on textual intimacy analysis remain rare @xcite . the first textual intimacy dataset was annotated by pei and jurgens (2020) with 2,397 english questions collected mostly from social media posts and fictional dialogues. however, @xcite claim that models trained over phrases in the question structure might not generalize well to text in other forms of languages. to work on a more generalizable dataset and promote computational modeling of textual intimacy, @xcite proposed their new dataset mint, which is a multilingual intimacy analysis dataset covering 13,384 tweets in 10 languages including english, french, spanish, italian, portuguese, korean, dutch, chinese, hindi, and arabic. each tweet in the dataset has a label between 1 and 5 as an intimacy score; 1 indicating "not intimate at all' and 5 indicating "very intimate". these scores were generated by taking the mean of all labels given by different annotators. semeval 2023 task 9: multilingual tweet intimacy analysis asks participants to predict the intimacy of tweets in the 10 languages in mint. they benchmark some of the large multilingual pretrained language models such as xlm-t @xcite , xlm-r @xcite , bert @xcite , distillbert @xcite and minilm @xcite , and report that distilled models generally perform worse than other normal models @xcite . the xlm-t model which is a multilingual roberta model trained with about 200m tweets, is the bestperforming model in the task organizers' reports, and the model we experiment with. ## methodology we decided that using the best-performing model in the baseline models listed in @xcite should be the starting point and therefore commenced our experiments with xlm-t. we used the https: //github.com/cardiffnlp/xlm-t quick prototyping google colab notebook for fine-tuning. our initial goal was to first reach the baseline performance for each language as reported in the task paper @xcite , and then improve from that point. since there were no data available in the zero-shot languages (hindi, korean, dutch, arabic), we shifted our focus to training individually on the 6 languages; english, spanish, portuguese, italian, french, and chinese. once we confirmed that we could attain close results to the baseline xlm-t scores for the abovementioned 6 languages by training the model on each language and testing on that language, we started training the model on all available languages making use of xlm-t's multilingual capacity. after exploring hyperparameters on the validation data to decide which hyperparameter settings performed the best, we trained a model on the combined training and validation data to make use of all of the provided data. ## experimental setup the data provided by the task organizers include 9491 annotated tweets in english, spanish, portuguese, italian, french, and chinese, with almost an equal number of samples from each language, as shown in to test each individual language in the dataset, we split a given language into 60% train, 20% ## results according to the released evaluation results our submitted model had an accuracy of 0.7129227988 for seen languages, and 0.4441965257 for unseen languages, placing our model 13 th and 8 th respectively out of 45 submissions, and 9 th overall. for seen languages, our model performed as expected with some improvements for spanish, italian, and french relative to the baselines stated by @xcite . ## conclusion in our work, we have shown that intimacy prediction on multilingual tweets with the xlm-t model performs well in the languages available in the training dataset, and also the zero-shot languages tweet actual predicted difference @user @user i hope they never google aral sea 3.33 1.119051 2.2142823 @user @user lmfao. we stood a better chance with the women. that one go cut off your ‚äútwins‚äù.