HULAT at S em E val-2023 Task 9: Data Augmentation for Pre-trained Transformers Applied to Multilingual Tweet Intimacy Analysis (2023)

Abstract: This paper describes our participation in SemEval-2023 Task 9, Intimacy Analysis of Multilingual Tweets. We fine-tune some of the most popular transformer models with the training dataset and synthetic data generated by different data augmentation techniques. During the development phase, our best results were obtained by using XLM-T. Data augmentation techniques provide a very slight improvement in the results. Our system ranked in the 27th position out of the 45 participating systems. Despite its modest results, our system shows promising results in languages such as Portuguese, English, and Dutch. All our code is available in the repository https://github.com/isegura/hulat_intimacy .

## introduction the intimacy analysis of multilingual tweets @xcite is a very novel task whose goal is to estimate the level of intimacy of a text. if we can detect the degree of intimacy of a text, this could help us better understand the social norms that exist within a culture or society. furthermore, from a computational point of view, this task will allow us to assess whether current computational models are capable of identifying the intimacy level of a text, distinguishing between texts with very intimate content and texts without any intimate information. although the automatic analysis of textual intimacy in language could be very beneficial for politicians, governments, sociologists, anthropologists, companies, and so on, very few research efforts have been devoted to this challenging task to now @xcite . another important aspect of this task is its multilingual setting. the dataset used in the task is the mint dataset @xcite with a total of 13,384 tweets in 10 languages including english, french, spanish, italian, portuguese, korean, dutch, chinese, hindi, and arabic. however, the training subset does not include tweets written in arabic, dutch, korean, or hindi. the task can be modeled as a regression task. we explore some of the most successful transformers such as bert @xcite , distilbert @xcite , minilm @xcite , xlm-roberta @xcite , and xlm-t @xcite . in addition, we also study some data augmentation techniques to increase the training dataset. based on our experiments during the development phase of the competition, we decided to combine the xlm-t transformer model and the data augmentation techniques to estimate our predictions for the test set during the test phase. our team, hulat, obtained an overall correlation (pearson's r) of 0.55. the highest correlation was 0.616, while the lowest correlation was 0.03. our system ranked in the 27th position out of the 45 participating systems. although our results are modest, our system performs very well for some languages such as portuguese and english, ranking third and ninth in the competition, respectively. as expected, our system obtains worse results for the unseen languages (arabic, korean, or hindi), but it ranks seventh for dutch. regarding the data augmentation techniques, our experiments show that the use of synthetic data does not appear to provide a significant improvement in the performance of the transformers. all our code is available in the repository https://github.com/isegura/ hulat_intimacy . ## background this task is a regression task whose goal is to predict a numerical value ranging from 1-5 that represents the intimacy level of a tweet, where 1 is the lowest level, while 5 is the highest level of intimacy. an example is "here is a nice equation: 0+0-0-0+0=0", which was annotated with an intimacy score of 1.0. another example is "hes eating cake and im planning to eat him", which was anno-tated with an intimacy score of 4.40. the training dataset contains 9,491 tweets from six different languages: portuguese, chinese, spanish, french, english, italian. the test dataset has 3,881 tweets written in the previous languages, but also in four unseen languages: korean, dutch, arabic, and hindi. both training and test datasets have balanced representations of their languages (see fig. 4 ). in the training dataset, each language is represented by a sample of approximately 1,500 tweets. in the test dataset, the average size is around 400 tweets per language, except for hindu, which only has 280 tweets. we have studied the distribution of the intimacy scores in the training and test datasets (see fig. 7 ). the mean intimacy score is around two tokens in both datasets. 95% of the tweets have an intimacy score lower than 3.8 in the training dataset, and lower than four in the test dataset. since our dataset is multilingual, we are also interested in studying the distribution of the intimacy scores for each language. in the training dataset, the tweets in all languages tend to have low intimacy scores, with their highest densities at 1.5 score (see fig. 8 ). english, italian, and french are the languages that have a larger number of tweets with intimacy scores lower than 2.0. chinese, portuguese, and spanish show more uniform distribution than the others. chinese and spanish tweets appear to have the tweets with higher intimacy scores. we also observe the same distribution of the intimacy scores for the six languages in the test dataset (see fig. 9 ). regarding the distribution of the intimacy scores for the new languages in the test dataset (that is, hindi, korean, dutch, arabic), we observe that dutch and arabic appear to have a large number of tweets with lower intimacy scores than korean and hindi (see fig. 1 ). indeed, 75% of the tweets written in dutch or arabic have intimacy scores lower than 1.8. in both languages, the mean score is around 1.5. on the contrary, 75% of the tweets written in hindi have intimacy scores greater than 1.5, with a mean score of 2.5. korean tweets appear to have even greater intimacy scores, with a mean score of 3.0, and 75% of them show intimacy scores greater than 2.5. therefore, textual intimacy may be strongly related to language (see fig10). we also studied the length of the tweets. the mean size is around 10 tokens in both training and test datasets. moreover, 95% of the tweets have less than 22 tokens (see fig. 5 ). figure 6 shows the distribution of tweet length for each language in the training dataset. as expected, chinese tweets are considerably shorter than the other languages, with an average length of between three and four tokens. this is due to the fact that the chinese language uses ideographs to represent words and ideas, and therefore, these symbols have more information than individual characters. for the romance languages, tweets are larger with an average length of around 10 tokens. all languages show a similar distribution of their lengths. moreover, the same distribution was seen in the test dataset (see fig. 2 ). for the unseen languages, arabic, dutch, hindi, and korean, (which are only present in the test dataset), we can see the tweets written in korean are shorter than the tweets written in other languages (see fig. 3 ). this is because korean is also based on ideographs. in fact, this language is derived from chinese (goodman et al., 2012). tweets written in hindi are larger with an average length of 15 tokens approximately. arabic and dutch show a similar distribution, with an average length of around 10 tokens. ## experimental setup during the development phase, we divided the training dataset into three splits with a ratio of 70:10:20. that is, 6,643 texts for training, 940 for validation, and 1,908 for testing. these splits allowed us to train and validate our models, and choose the best model for the test phase. the three splits have the same distribution of languages and intimacy scores. we used the eda foot_3 and nlpaug (version 1.11.1) foot_4 libraries for data augmentation on the training split. before tokenization, we used some filters to clean texts. for example, we removed all mentions of '@user' and urls. we also removed punctuation. based on the length distribution of texts (see language pearson top pearson ranking portuguese .699 .702 3 english .721 .758 9 chinese .709 .762 21 italian .693 .742 22 french .670 726 25 spanish .698 .784 29 overall .707 .707 18 fig. 5 ), we consider 50 tokens as the maximum length. we used the trainer class from the transformers library (version 4.27.4) to train our models, using pytorch as the backend framework. all models were trained with batch size 64 for three epochs, while the batch size was 20 for validation. we train all models using pearson's r as the metric to choose the best model. pearson correlation coefficient measures the linear relationship between two datasets, in our case, the actual and predicted values. it can range from -1 to +1, where 0 means that there is no correlation. a positive score tells us that there is a positive association between the two datasets. ## results our team, hulat, participated in the task by using the xlm-t model @xcite and data augmentation. our approach obtained an overall correlation (pearsons'r) of 0.55. the highest correlation was 0.616, while the lowest correlation was 0.03. our system ranked in the 27th position out of the 45 participating systems. regarding the results for the seen languages (that is, the languages that are in the training dataset), our system ranked third for portuguese with a correlation of 0.699, very close to the highest correlation (0.702). for english, our system ranked ninth for english with a correlation of 0.721, being the highest correlation of 0.758. regarding the results for the unseen languages (that is, the languages that were not in the training dataset), our system achieved a correlation of 0.355, while the highest correlation was 0.449. however, our system ranked eighth for dutch with a corre during the development phase, we explored different models described in section 3. ## conclusion in this paper, we fine-tuned some of the most successful transformers by using an augmented training dataset created with some data augmentation techniques. based on the results during the development phase, we chose the xlm-t model with data augmentation for the test phase. our system obtained an overall correlation of 0.55, six points below the highest correlation in the ranking. for portuguese and english, it ranked fourth and ninth, respectively. although our results for unseen languages are modest, our system ranked eight for dutch. as future work, we plan to use zero-shot text classification methods to automatically annotate a small sample of tweets written in the unseen languages of the task, which later will be manually reviewed by experts. we also plan to extend the dataset by using texts from novels (which should contain more intimacy) and from academic books (whose texts, in general, should have a less degree of intimacy). fortunately, novels and academic books are usually published in different languages. a group of experts will annotate a small sample of these texts. then, we will explore some data augmentation techniques, especially based on back translation, to create new synthetic data for the task and for all the languages.