S em E val-2023 Task 9: Multilingual Tweet Intimacy Analysis (2023)

Abstract: Intimacy is an important social aspect of language. Computational modeling of intimacy in language could help many downstream applications like dialogue systems and offensiveness detection. Despite its importance, resources and approaches on modeling textual intimacy remain rare. To address this gap, we introduce MINT, a new Multilingual intimacy analysis dataset covering 13,372 tweets in 10 languages including English, French, Spanish, Italian, Portuguese, Korean, Dutch, Chinese, Hindi, and Arabic along with SemEval 2023 Task 9: Multilingual Tweet Intimacy Analysis. Our task attracted 45 participants from around the world. While the participants are able to achieve overall good performance on languages in the training set, zero-shot prediction of intimacy in unseen languages remains challenging. Here we provide an overview of the task, summaries of the common approaches, and potential future directions on modeling intimacy across languages. All the relevant resources are available at https: //sites.google.com/umich.edu/ semeval-2023-tweet-intimacy.

## introduction intimacy has long been viewed as a primary dimension of human relationships and interpersonal interactions @xcite @xcite . existing studies suggest that intimacy is an essential social component of language and can be modeled with computational methods @xcite . recognizing intimacy can also serve as an important benchmark to test the ability of computational models to understand social information @xcite . despite the importance of intimacy in language, resources on textual intimacy analysis remain rare. @xcite annotated the first textual intimacy dataset containing 2,397 english questions, collected mostly from social media posts and fictional dialogues. however, such question phrases are often used primarily for interrogative situations, and, as such, models trained over the dataset may not generalize well to other types of text. to further promote computational modeling of textual intimacy, we introduce a new multilingual textual intimacy dataset (mint). the training data in mint covers tweets in 6 languages, including english, spanish, french, portuguese, italian, and chinese, which are languages used by over 3 billion people on earth, in the americas, europe, and asia. a total of 12,000 tweets are annotated for the six languages. to test the model generalizability under zero-shot settings, we also include small test sets for dutch, korean, hindi, and arabic (500 tweets for each), which are spoken by over 0.8 billion people around the world. we benchmarked a series of large multilingual pre-trained language models including xlm-t @xcite , xlm-r @xcite , bert @xcite , distillbert @xcite and minilm @xcite . we found that distilled models generally perform worse than other normal models, while an xlm-r model trained over the twitter dataset (xlm-t) performs the best on seven of the ten languages. while the pre-trained language models are able to achieve promising performance, zero-shot prediction of unseen languages remains challenging, especially for korean and hindi. based on mint, we organized semeval 2023 task 9: multilingual tweet intimacy analysis, which attracted 45 teams from more than 20 countries. participants primarily built their systems based on multilingual pre-trained language models like xlm-t @xcite , mbert @xcite , and xlm-r @xcite due to the multilingual nature of this task. however, participants also used a variety of modelbuilding techniques to potentially improve performance, with data augmentation methods like translation, external label generation, and word-level substitution being among the most heavily used. overall 24 out of 45 teams are able to @xmath0.7 pearson's r on the training languages and 22 teams are able to @xmath1.4 pearson's r on the unseen languages, which beat the strong xlm-t baseline. ## task description intimacy is one of the most fundamental dimensions of human relationships @xcite . intimacy has long been used as the index for not only relationships but also the interactions between people @xcite . language plays a central role in interpersonal interactions as it allows people to communicate various types of information, share emotions and build connections @xcite . one of the most prominent forms of interaction involving language is self-disclosure, which refers to the process of sharing personal experiences or emotions about themselves @xcite . due to the importance of self-disclosure in interpersonal relationships, existing studies on intimacy in language generally focus on self-disclosure. those studies usually consider intimacy as the primary dimension of self-disclosure, where deep disclosures of personal emotions or relationships are considered as intimate @xcite . datasets and nlp models are also built to automatically analyze self-disclosure in communications, especially on social media @xcite . however, self-disclosure is only part of the language that people use in daily communications. solely studying intimacy in the context of self-disclosure overlooks many types of language interactions which do not involve self-disclosure. for example, pei and jurgens (2020) build the first dataset of english questions and found that questions can also have various levels of intimacy. while the question intimacy dataset provides resources to study intimacy in language, it only focuses on questions, missing the variation of intimacy in other types of texts. moreover, the expressions of intimacy depend on the specific types of language. different languages may have different expressions of intimacy. therefore, to support further studies on modeling intimacy in different languages, we present semeval 2023 task 9: multilingual tweet intimacy analysis. multilingual tweet intimacy analysis is a task to predict the intimacy of tweets in different languages. defining intimacy in language is challenging as intimacy is a natural concept @xcite . in our task, we focus on perceived intimacy by asking annotators to give their subjective judgment of tweet intimacy. we draw a diverse pool of native speakers for each language from prolific.co to participate in our annotation task. each annotator is asked to answer "how intimate do you think the given tweet is?" using a 1-5 likert scale, which finally leads to 13,372 tweets in 10 languages annotated with an intimacy label. the training data contains labeled intimacy for six languages: english, french, spanish, italian, portuguese, and chinese. pre-trained language models (plms) have achieved huge advances in recent years and have shown promising zero-shot abilities for downstream tasks in different languages @xcite . to encourage new studies on understanding intimacy in language as well as understanding zeroshot ability of plms, we also include four other languages without training data (dutch, hindi, korean, and arabic). the participants are asked to build models using the six training languages that can predict tweet intimacy from 1 (not intimate at all) to 5 (very intimate). the final model performance is evaluated on the test set in all ten languages and pearson's r is used as the final evaluation metric. foot_0 ## data we choose twitter as the source of our dataset because twitter is a public social media platform that naturally includes multilingual text data and, from our analysis, a fair amount of intimate texts. in this section, we introduce the data collection and annotation process for mint. ## baseline models we benchmark several baseline models on the tweet intimacy prediction task. we compare the following multilingual pre-trained language models: 1. bert @xcite all the models are trained with 10 epochs and the best-performing model is selected based on the dev set 8 . we train each model with 5 different random seeds and report the mean score. the learning rate is set as 0.001 and the batch size is 64. we use adamw as the optimizer @xcite . for zero-shot tasks, while xlm-t still performs the best on hindi and arabic, xlm-r and minilm achieved the best result on dutch and korean, respectively. moreover, the zero-shot performance is generally lower compared with the tasks with ## results this task received final submissions from 45 teams. each team was allowed to submit multiple times but only the final submission was scored on the test set. of the 45 teams, 22 submitted a system description paper. here, we summarize the submissions' approaches by the following aspects: base pretrained language model, data augmentation, and other heuristic techniques. ## discussion intimacy is an important dimension of human relationships and language. predicting intimacy across languages is a difficult task, because language is embedded in different cultures with different perceptions of intimacy. to address this issue, we built mint, the multilingual intimacy dataset. forty-five participating teams have submitted their solutions to our multilingual tweet intimacy prediction task. overall the participants are able to achieve relatively high performance (pearson's @xmath2.7) on languages that are in the training set. while the performance on the unseen languages is relatively lower, many participants are able to improve their performance over the already-strong xlm-t baseline. we observe a clear trend that most of the participants focused on the data augmentation and ensemble methods, with less effort spent on traditional feature engineering. moreover, despite the fact that in-domain lm fine-tuning has been found to be helpful for downstream tasks @xcite , only a very limited number of participants considered in-domain lm fine-tuning as part of their solution. this is potentially due to the computational resources required for large-scale fine-tuning. despite recent advances in prompting, only one participant explored soft prompts, suggesting that the regression task remains challenging for prompting large language models. our dataset provides a valuable multilingual resource for studying prompting regression tasks in the context of understanding social information. we encourage future researchers to explore this direction. as a first step towards understanding textual intimacy in language, in this task, we only released aggregated labels from all the annotators. however, the perception of intimacy may vary with factors like gender, age, and culture. we are planning to release detailed background information of all the annotators. this will allow a potential follow task to predict intimacy ratings that incorporate annotators' background information. ## conclusion in this paper, we present semeval 2023 task 9: multilingual tweet intimacy analysis along with mint, the first multilingual textual intimacy analysis containing 13,372 tweets in ten languages (english, french, spanish, italian, portuguese, korean, dutch, chinese, hindi, and arabic). we benchmarked a series of multilingual pre-trained language models. our task attracted participation from 45 teams and 18 teams are able to beat the already strong xlm-t baseline. our overview indicates that data augmentation, external labels, and ensem-ble methods are commonly used by the participants, and led to good performance. overall, participants generally achieved good performance on training languages with an average pearson's r above 0.7, showing that the current pretrained language models, once fine-tuned, are able to accurately predict intimacy in various languages. however, the overall performance remains low on unseen languages, suggesting that zero-shot intimacy analysis remains a challenging task. further research is needed to better analyze textual intimacy across languages.