jelenasteam at S em E val-2023 Task 9: Quantification of Intimacy in Multilingual Tweets using Machine Learning Algorithms: A Comparative Study on the MINT Dataset (2023)

Abstract: Intimacy is one of the fundamental aspects of our social life. It relates to intimate interactions with others, often including verbal self-disclosure. In this paper, we researched machine learning algorithms for quantification of the intimacy in the tweets. A new multilingual textual intimacy dataset named MINT was used. It contains tweets in 10 languages, including English, Spanish, French, Portuguese, Italian, and Chinese in both training and test datasets, and Dutch, Korean, Hindi, and Arabic in test data only. In the first experiment, linear regression models combine with the features and word embedding, and XLM-T deep learning model were compared. In the second experiment, cross-lingual learning between languanges was tested. In the third experiments, data was clustered using K-means. The results indicate that XLM-T pre-trained embedding might be a good choice for an unsupervised learning algorithm for intimacy detection.

## introduction natural language processing (nlp) models have access to more information than any human speaker during their life. still, it would be hard for them to reach human-level competence and performance @xcite . nlp community needs to address social factors to get closer to the humanlike language understanding. performances of the nlp models for detection of the social-components in the language like patronizing and condescending language detection, irony and sarcasm understanding, persuasion, humor and emphasis detection, and many others, should be improved. the aim of our work was analysis of intimacy in tweets, as intimacy is one of the fundamental social aspects of language. in this paper, a new multilingual textual intimacy dataset named mint annotated by @xcite was used. the dataset is released along with the semeval 2023 task 9: multilingual tweet intimacy analysis, co-organized by university of michigan and snap inc. it contains a total of 13, 384 tweets in 10 languages, including english, spanish, french, portuguese, italian, and chinese in both training and test data, and dutch, korean, hindi, and arabic in test data only. during work on this task, a three experiments were conducted: ## background intimacy is a fundamental aspect of how we relate to others in social settings @xcite . we express it by offering emotional support, reassurance of love and care, or a deeper understanding of self and others. intimate contact has a high impact on personal growth, feeling secure, achievement of our potential in friendships and romantic relationships @xcite . there are multiple conceptions of intimacy. some define intimacy as a kind of intimate interactions, which are most frequently associated with verbal self-disclosure. in their work @xcite proposed a method for detection and analysis of self-disclosure in online news commentaries. their model used detection of sentence form, active or passive, based on the subject-verb-object relation in the sentence. they tested model over different categories based on the type of personal info disclosed: age, race, sexuality, location or affiliation, relationship status, money, interests, opinions and feelings. their model was used by @xcite in a study of self-disclosure during a coronavirus pandemic. in their work on automatic classification of sexism in social networks, (rodríguez-sánchez et al., 2020) compared bert model with bilstm and traditional classifiers based on combined features with word embedding and tweet and user meta-data. the results show that a combination of additional information about the tweet can increase classification accuracy. authors pointed out that some tweets could sound lovely or friendly, but also could be considered sexist. the authors (shen and rudzicz, 2017) investigated binary anxiety detection on a small english reddit dataset. they used posts without meta-data and preform classification with word2vec, doc2vec, the latent dirichlet allocation (lda), n-grams combined with logistic regression (lr), support vector machine (svm) and artificial neural networks (ann), optimized over features and classifier. the results indicated the differences in the lexics of these two classes, in the most frequent unigrams, bigrams, and trigrams. in their research (pei and jurgens, 2020) conducted several studies of expressions of intimacy in language, and showed, using a fine-tuned roberta, that the intimacy of language is not only a personal choice, where people may use different linguistic strategies for the expressions of intimacy, but reflects constraints of social norms, including gender and social distance. they reproofed stranger on a train phenomenon. the data set used in this research was multilingual, which rises questions about cultural differences expressed through different language groups on twitter. with the language chosen, a social identity is created. speakers may make use of specific words or stylish elements to represent themselves in a certain way @xcite . people from different countries have their own ways of redrawing the boundaries of what may be said, written and done within a given discourse community @xcite . some expressions of intimacy that are acceptable in one culture, could be rude and impolite or even absurd in another one. the authors @xcite researched how multilingual language models maintain a shared multilingual representation space, while encoding language sensitive information in each language. they used xlm-r model and compared space between 88 languages. after performing dimension reduction on embedding from different layers in the network, the authors concluded that there are a language-sensitive axis, and a language-neutral axis. the limitation of their work was that the most of languages were european. the multilingual pre-trained bert embedding and lda topic model were used by @xcite , for analysis of topic evolution in monolingual and multilingual topic similarity settings. for each topic, they multiply its lda probability value by the averaged tensor similarity of bert embedding to explore the evolution of the topic in scientific publications. the explosion of digital trace data with the availability of computational methods that are faster, cheaper and easier to use, has ushered in a new scientific approach to measuring culture, bringing the new challenges to overcome. biased nlp algorithms, caused by inappropriate data collection, can cause instant negative effects on society by discriminating against certain social groups and shaping the biased associations of individuals through the media they are exposed to. in the end, it is important to mention that social media languages have their own specifications, like more colloquial expressions and more linguistic variation, such as the use of slang and dialects, platform-specific factors such as misspellings, vulgarisms, emoji and multi-modality. ## system overview the task of intimacy analysis was conducted through three experiments, where each one concentrates on specific aspect of this problem. in the first experiment, comparative study of intimacy detection with different models was conducted. we have noticed that in the last few years on semevals, most participants researched deep learning models, which was expected considering they are state-ofthe-art in many nlp tasks. however, we think that traditional models could be useful, not just because of their low complexity, but also because of their non-black-box nature. two linear regression (lr) models with l2 regularization separately trained on either bag of words (bow) features and term frequency-inverse document frequency (tf-idf) features were included. features were extracted using countvectorizer and tfidfvectorizer from the sklearn library @xcite , with unigrams, bigrams and trigrams, and jieba tokenizer. regularization parameter was determined using cross-validation. tweets from the training dataset, for each seen language, were randomly split into 5 folds, stratified by the intimacy score. these folds were used for training and validation. based on the cross-validation results, the regularization parameter was determined. the model trained with the optimal regularization parameter was tested on train and test data. apart from features, word embedding-based models were compared. embedding were generated using word2vec from gensim library @xcite and the pre-trained xlm-t model introduced by @xcite . the word2vec based model was optimized over the vector size, starting from 100 and increasing the size for 50, until model performance stopped improving. the optimal size was 200. for xlm-t embedding, a pre-defined size of 768 was used. both embedding were used with the lr models, with parameters set in the same way as for a feature-based model. in the end, xlm-t model was used, as it represents one of the dl models that achieved the best performance in the task proposal paper @xcite . the data was split randomly stratified to language, into a train and validation set, with a ratio of 80 : 20. batch size was 64, and the model was optimized over 0.01, 0.001 and 0.0001 learning rate. better results were gained using 0.001 and 0.0001 learning rates and 0.001 was chosen. the model was trained with 15 epochs. we evaluated the model after each 100 steps, and loaded the best model in the end. in the second experiment, cross-lingual analysis was conducted. models were trained on data in one language from train dataset and tested on all languages in test dataset. xlm-t model fine-tuned in the same way as in the first experiment was used. the aim was to research what is the best choice for zero-shot for each language, and to compare similarities for intimacy expressions of languages got through cross-lingual training, with the known cultural similarities. in the third experiment we clustered the data, in an attempt to see whether the tweets would group by language or intimacy score, thus verifying the validity of the extracted features. based on the results of the first experiment, we decided to use xlm-t embedding as a feature for clustering. kmeans was implemented using sklearn library @xcite . during experiment, we were changing the number of clusters and observing the statistics of clusters. the inspection involved examining the number of unique languages in each cluster and the average intimacy score for a fixed number of clusters. all clustering were performed with the random initialization. after clustering training data, the test data clusters were predicted. ## conclusion this paper represents an analysis of intimacy detection in a multilingual data collected and proposed for semeval 2023 task 9: multilingual tweet intimacy analysis. the task was to train model to predict the level of intimacy in tweets, for languages present and non-present in train data. in three experiments, we compared feature-based, embeddingbased and dl models, cross-lingual learning and clustering. our study indicates that xlm-t pretrained embedding might be a good choice for an unsupervised learning algorithm for intimacy detection. an increase in the training data even in different languages could improve performance of the model, and the best choice for training language might be made using prior knowledge about cultural or historical connections of languages. the clustering of data was more closely related to intimacy score than to the language of the tweet, indicating that embedding generated using pre-trained language model can capture an hidden information meaningful for intimacy analysis. this finding indicates that properly collected and pre-processed data can be utilized for unsupervised learning, reducing the need for extensive data labeling efforts. intimacy is one of the core components of our relationships and interpersonal interactions. considering our everyday exposure to social networking platforms, it is important to continue researching on models for social factor detection in these media.