UMUT eam and SINAI at S em E val-2023 Task 9: Multilingual Tweet Intimacy Analysis using Multilingual Large Language Models and Data Augmentation (2023)

Abstract: This work presents the participation of the UMUTeam and the SINAI research groups in the SemEval-2023 Task 9: Multilingual Tweet Intimacy Analysis. The goal of this task is to predict the intimacy of a set of tweets in 10 languages: English, Spanish, Italian, Portuguese, French, Chinese, Hindi, Arabic, Dutch and Korean, of which, the last 4 are not in the training data. Our approach to address this task is based on data augmentation and the use of three multilingual Large Language Models (multilingual BERT, XLM and mDeBERTA) by ensemble learning. Our team ranked 30th out of 45 participants. Our best results were achieved with two unseen languages: Korean (16th) and Hindi (19th).

## introduction in natural language processing (nlp), intimacy can be described as how people communicate their perception and willingness to share personal data and emotions to their audience @xcite . the semeval 2023 task 9, entitled multilingual tweet intimacy analysis (mtia) @xcite , consists of a regression task in which the participants should rate in a score from 1 to 5 the intimacy of short documents written in 10 languages: english, spanish, italian, portuguese, french, chinese, hindi, arabic, dutch and korean. this task was co-organized by university of michigan and snap inc. there are two main challenges concerning this task. on the one hand, the training dataset provided to the participants does not cover all the evaluated languages, but only six of them: english, spanish, italian, portuguese, french, and chinese. however, the evaluation is conducted in those six languages plus hindi, arabic, dutch and korean. on the other hand, participants were only allowed to submit a unique run, which hinders the shared task. our strategy to solve the mtia challenge consists of an ensemble learning composed of three multilingual large language models (llm): multilingual bert @xcite , xlm @xcite , and mdeberta @xcite . besides, we use data augmentation incorporating to the training the dataset suggested by the organizers and provided in the work of @xcite , with more than two thousand english questions from reddit and other sources and annotated with intimacy scores in the range @xcite . our participation achieved modest results in the task, reaching the 30th position in the leader-board, with a pearson's r of 0.53. the best result is achieved by lazybob, with a pearson's r of 0.62. as commented above, as the participants were only allowed to submit a unique run, the analysis of our proposal is mainly based on a custom validation split. additional resources concerning our participation can be found at https://github.com/ nlp-umuteam/semeval-2023-mtia . ## background the organisers of the task provided the participants with the novel mint dataset @xcite , whose original training split consists of 9491 tweets rated with an intimacy score. the tweets were compiled between 2018 and 2022. to obtain tweets in different languages, the authors combined language filters in twitter with language detectors models such as fasttext @xcite . next, the authors created clusters of tweets of each language and several annotators rated the tweets in a scale from 1 (not intimate at all) to 5 (very intimate). as it can be observed, in the histogram plotted in figure 1 , most of the samples are rated with low scores. regarding the six languages involved during the training, these are almost balanced, with 1596 documents written in portuguese and chinese, 1592 in spanish, 1588 in french, 1587 in english and 1532 in italian. an example of the dataset is the spanish text "necesito paz mental" 1 , rated with an intimacy score of 2.8. in figure 2 the rounding label distribution is shown. the majority of labels are between 2 and 3 and with fewer instances of labels near to 0 or 5. the participants of the task were encouraged to use the dataset provided in pei and jurgens (2020); which contains english sentences with an intimacy score between -1 and 1. ## system overview our pipeline for solving the mtia 2023 shared task is depicted in figure 3 . in a nutshell, it can be described as follows. first, we clean and preprocess the mtia dataset and keep a small portion of the training split to create a custom validation 1 in english: i need peace of mind split. second, we perform a data augmentation stage applying google translate to the dataset of @xcite . third, we evaluate three multi-lingual llms and one model based on linguistic features. forth, we build an ensemble learning model that averages the predictions of the three llms to send our final predictions to the organizers of the task. concerning the data cleaning stage, we strip hyperlinks, hashtags, mentions and white space characters. regarding the dataset splitter step, we reserve a 20% of the tweets from the training split for custom validation purposes. next, we enlarge the training dataset proposed by incorporating the dataset provided in @xcite . this dataset contains sentences written in english. we use google translate to translate these sentences to spanish, italian, portuguese, french, hindi, arabic, dutch and korean. this way, we could incorporate 21573 new sentences to the training. as this dataset is rated in rank from -1 to 1, we translate the ratings to a scale from 1 to 5, maintaining the ratio. besides, it is worth noting that none of these new instances are used for custom validation. ## experimental setup during the evaluation phase, apart from the multilingual llms, we evaluate the usage of linguistic features from umutextstats @xcite . the linguistic features from umu-textstats have been evaluated in several nlp tasks, such as author profiling @xcite , satire identification (garcía-díaz and valencia-garcía, 2022), and hate-speech detection @xcite . umutextstats is designed for the spanish language, but it has a subset of language-independent features. these features are stylometric features and features related to named entity recognition (ner) and part-of-speech (pos). to extract these features, umutextstats relies on stanza @xcite to extract some features related to pos and ner. however, not all the languages involved in the mtia shared task have models per stanza, so the linguistic features were not useful for some of the languages involved on this shared task. accordingly, we decided not to include the linguistic features (lf) in the final submission. however, we use the linguistic features to make an analysis of the spanish split of the dataset and we observe a correlation with misspelled words with intimacy followed by morphological features related to proper and common nouns, personal pronouns in first, second person, and third person. we also identify a correlation with stylometric clues concerning the length of the tweets and with the usage of hyperboles, proper from figurative language. these results are depicted in figure 4 . next, the regression neural network architecture is described. for each llm we conduct an hyperparameter optimization stage consisting of training 10 models for each llm evaluating different parameters, including the learning rate, the number of epochs for traning, the warm up steps and the weight decay. the results of the best model for each llm are depicted in finally, we conduct another hyperparameter op-timization stage using keras. we follow this step to be consistent with the lfs and the llms. the results of this experiment are reported in for each feature set, we evaluate 55 models changing the neural network architecture (its number of neurons and hidden layers), the dropout, the batch size and the activation function. we can observe that the best models for the lf and for mbert are complex neural networks with 5 and 8 hidden layers respectively. the lf neural network has brick size (all layers have the same number of neural networks) but mbert has a diamond shape (the inner layers have much more neurons). all models benefit for a strong dropout mechanism and most of them also benefit from large batch sizes. ## conclusion despite the fact that our results are limited, we are very pleased with our participation. first, because this is the first time we participated in a shared task concerning intimacy. second, because the mtia shared task was challenging as we could only send one result and because there are four unseen languages during testing. our proposal based on ensemble learning on three multilingual llm reached position 30th in the official leaderboard from a total of 45 participants. our best results are achieved with two unseen languages: korean (16th) and hindi(19th). after the evaluation of our results, we consider that there are several ways in which we could have improved our results. first, we should have conducted an in-deep analysis of the dataset. however, this was not easy for us because we are not fluent speakers of many of these languages, so we can miss important aspects related to the context. second, it is possible that the data augmentation process was not beneficial for the performance of our model, as the translations could be less accurate in some languages or it is possible that cultural and background differences are not well represented in the dataset. however, we consider that we could have translated all sentences into a common language (spanish or english, for instance) and could include features related to topics to our model. we will explore this path in future multilingual shared tasks. three, our models could be biased to our custom validation split. in this sense, we will incorporate to our pipeline a nested-cross validation evaluation. fourth, our ablation analysis is limited, as we only consider the data augmentation step. however, we need to conduct more experiments in order to gain understanding of other modules such as the preprocessing module.