C ha P at at S em E val-2023 Task 9: Text Intimacy Analysis using Ensembles of Multilingual Transformers (2023)

Abstract: Intimacy estimation of a given text has recently gained importance due to the increase in direct interaction of NLP systems with humans. Intimacy is an important aspect of natural language and has a substantial impact on our everyday communication. Thus the level of intimacy can provide us with deeper insights and richer semantics of conversations. In this paper, we present our work on the SemEval shared task 9 on predicting the level of intimacy for the given text. The dataset consists of tweets in ten languages, out of which only six are available in the training dataset. We conduct several experiments and show that an ensemble of multilingual models along with a language-specific monolingual model has the best performance. We also evaluate other data augmentation methods such as translation and present the results. Lastly, we study the results thoroughly and present some noteworthy insights into this problem.

## introduction intimacy is a crucial aspect of human nature and communication. the feeling of intimacy implies a certain degree of mutual closeness, openness, and trust between the people involved. it intends to convey information beyond the general semantic meaning, to increase the effectiveness of communication. capturing this information is a pivotal part of human-like language interaction. although the intimacy of text is largely determined by its content, several subtle semantic cues can be identified across the text to gauge its intimacy. the shared task @xcite ) @xcite was to predict the intimacy of tweets in ten languages. the primary dataset involved a set of tweets in six languages (english, spanish, italian, portuguese, french, and chinese) annotated with intimacy scores ranging from one to five. in addition to this data, we used the question intimacy * equal contribution dataset @xcite which contains 2247 english questions from reddit as well as another 150 questions from books, movies, and twitter with intimacy scores in this dataset ranging from -1 to 1. the model performance has been evaluated on the test set in the given six languages as well as an external test set with four languages absent in the training data (hindi, arabic, dutch, and korean) based on pearson's r as the evaluation metric. large language models provide a scalable and optimizable framework for performing a large range of text-classification language tasks. transformers have emerged as the de-facto standard for language modeling due to their ease of parallelization and expressive power as a general-purpose computer. owing to the democratization with pretrained transformers, we opted for an ensemble to capture distinct sets of features and capabilities. we used bert-based mono-lingual pre-trained models for all six languages as well as multi-lingual pre-trained models to capture inter-language dependencies in the ensemble-based method for this task. ## related work the importance of intimacy in our lives is wellstudied @xcite . intimacy, like other interpersonal characteristics, plays an important role in shaping public opinions. recent efforts have been made to computationally identify such information @xcite . the task of quantifying intimacy in written language was taken up initially by @xcite , where a dataset and model suited to the task were proposed. data was manually collected from subreddits and annotated using a best-worst scaling (bws) @xcite scheme. the model was built on a roberta base and finetuned in a self-supervised setting using masked language modelling. mse and pearson's r corre-data preprocessing m-bert xlm-r xlm-t english french chinese spanish italian portuguese ((1 * 2.1) + (1.1 * 2.0) + (1.5 * 2.3) + (2 * 2.2)) / (1+ 1.1 + 1.5 + 2) 2.16 2.1 2.0 2.3 2.2 language specific models multilingual models model predictions ensembling final prediction check language lation were the metrics used for evaluation against human judgment. their research also suggests interesting conclusions such as the most intimate questions are generally asked between close friends or total strangers. ## data the training dataset contains 9491 examples. the testing dataset provided during the submission phase of the task contains 13697 examples. however only 3881 samples are considered in the test set, and the rest of the samples are unannotated and are added to dissuade manual labeling of the data. the unannotated samples have an intimacy score of zero. thus the actual test dataset only contains 3881 examples. the data consists of tweets obtained from the social networking website twitter. all of the training samples have intimacy level scores ranging from one to five, where one denotes least intimate while five denotes most intimate. the test dataset also has the same range of intimacy scores. the task involves predicting the intimacy level of tweets from ten languages. the training dataset consists of tweets from six languages: english, italian, portuguese, french, spanish, and chinese. the testing dataset, in addition to these six languages, consists of tweets from four more languages: hindi, dutch, arabic, and korean. the additional languages are absent in the training dataset. they are added to test the capacity of language agnostic performance of models in the prediction of intimacy. all of the tweets are present in scripts native to their languages, although some tweets contain some text from a different script, used for purposes such as acronyms and names. the data is anonymized by replacing username mentions with a generic '@user' handle. the dataset contains hashtags and emojis. in addition to the given dataset, we used the reddit questions dataset, containing questions in english from various sources such as tweets, books, and movies in addition to reddit posts. we used 2247 examples from this dataset. the examples in this dataset originally have intimacy levels ranging from -1 to 1. we mapped these scores to a new range of one to five to be compatible with the main training dataset. we split the training dataset to obtain the training and validation datasets and added the reddit questions to the training dataset. the training, testing, and validation dataset sizes are summarized in ## system overview the presence of multiple languages in the training dataset and additional unseen languages in the testing dataset present unique challenges. we have explored several avenues to solve the problem efficiently. our approaches utilize pre-trained large language models. we elaborate on our successful as well as unsuccessful experiments in this and the following sections. ## results we present and discuss the results of our experiments in this section. the evaluation metric for the shared task is pearson's r. we have computed the scores using the scorer presented by the organizers. the language-wise, as well as aggregate performances of all our methods, are listed in detail in our experiments conclude that using an ensemble of various multilingual models along with a language-specific model yields the best overall performance. it has the highest score in the seen languages as well. the ensemble of only multilingual models has the second-best overall score. note that the multilingual models used in both of these approaches are the same, but with different weights during the ensembling procedure. the data augmentation approach with translation performs worse than the other methods. this approach has inferior performance in all languages, except for hindi, where the translation seems to provide better results. it can be observed that our approaches perform similarly well on all of the seen languages, with scores averaging around 0.7. among the unseen languages, multilingual models perform considerably poorly in hindi and korean texts. however, the models perform substantially better in dutch and arabic, despite both of the languages being absent in the training dataset. we can see that the multilingual models do not perform equally well on all the languages. this is expected as the pre-training data size varies per language for the models. it is ostensible that the predicted intimacy scores on unseen languages are bound to be less accurate than on seen languages. however, it is interesting to note that certain unseen languages perform considerably better than others. this disparity might suggest two things: the multilingual models perform better on certain languages because they are pre-trained on larger and higher quality corpora of these languages, or the language-agnostic understanding of the models is better for some languages than others. further research is necessary to disprove the former idea as it is difficult to ascertain the impact of the size of the pre-training data due to the current ensembling process. also, some models show a disparity in performance on unseen languages despite being pre-trained on similar-sized corpora for these languages. the inclusion of a language-specific model in the ensemble provides superior results in every seen language. the increase in performance over the multilingual models also varies per language. this suggests that additional training or fine-tuning of models on specific languages is an effective way of boosting performance for tasks where a sample contains only one language, irrespective of the total number of samples across the dataset. translating the dataset yields poor performance than the other methods. this is rather obvious, as translating text often leads to loss of important semantic information. this loss is further exacerbated in tasks where the meaning of the text is important, such as quantifying intimacy in our task. for example, the english word "baby" might be translated into french as "nourrisson", which is more similar to "infant". although the translation is valid, the original word is far more intimate than its translated counterpart. however, translation yields better performance in the hindi language as compared to relying on the multilingual ensemble. this suggests that although translation generally leads to worse results, it might be useful if the model performs very poorly on certain languages. we submitted the data augmentation-based approach as our final submission for the task. we scored 35th rank as per the overall score on all languages. we finished 18th based on the aggregate score of the seen languages. we also finished 7th in the chinese language. ## conclusion we present our approaches to the semeval task 9 of multilingual tweet intimacy analysis in this paper. we conducted several experiments involving language models and compared them. we also tried data augmentation methods such as translation of texts from languages absent in the training dataset. we show that an ensemble of multilingual models along with a language-specific model performs well. we show that translation does not yield good results in general. however, other avenues in data augmentation might improve the performance.