ROZAM at S em E val 2023 Task 9: Multilingual Tweet Intimacy Analysis (2023)

Abstract: We build a model using large multilingual pretrained language model XLM-T for regression task and fine-tune it on the MINT (Multilingual INTmacy) analysis dataset which covers 6 languages for training and 4 languages for testing zero-shot performance of the model. The dataset was annotated and the annotations are intimacy scores. We experiment with several deep learning architectures to predict intimacy score. To achieve optimal performance we modify several model settings including loss function, number and type of layers. In total, we ran 16 end-to-end experiments. Our best system achieved a Pearson Correlation score of 0.52.

## introduction intimacy has long been considered as the main dimension of human relationships @xcite @xcite . although intimacy has a significant role in language, there are few datasets in this field. the first textual intimacy dataset @xcite ) contains 2,397 english questions collected through social media. however, models trained on this dataset may not generalize well to other situations. mint dataset @xcite is a textual intimacy dataset covering 10 languages which is collected from people's tweets and annotated with an intimacy score of 1 to 5. the task is to predict intimacy scores of tweets for 10 languages. six languages are seen during training including english, spanish, portuguese, italian, french, chinese and 4 other languages @xcite are not included in training data. these four languages are used for evaluating zero-shot performance of proposed models. by doing this task, we can extract some social information inside the sentence and hence calculate how intimate it is. xlm-roberta @xcite is a multilingual pre-trained language model, pre-trained on 2.5tb of filtered commoncrawl data containing 100 languages (it is a multilingual version of roberta) and it was pre-trained with the masked language modeling (mlm) objective. the main strategy is to use xlm-t @xcite which is a xlm-roberta-base model trained on about 198m multilingual tweets. our model contains xlm-t for extracting embeddings of sentences and also some dense and dropout layers for predicting (approximating) the intimacy score of a sentence. we fine-tune the last few layers of xlm-t and train the regressor part of our model using the training dataset. since our dataset was relatively small, we also used some data augmentation methods. eventually, teams are ranked based on pearson correlation metric calculated on overall score for seen and unseen languages. the proposed method was ranked 33 out of 45. the details of our implementations are available through our github repository. ## background the inputs to our model were tweets collected from twitter and we expect our model to produce a number between 1 to 5 as an intimacy score as final output. a schematic of our model is shown below: output of xlm-t which is an embedding of the input sentence is a tensor of shape (768*1) is fed to the regressor model to predict a number as an intimacy score. furthermore, some augmentation methods were added for zero-shot languages in order to improve the model performance. ## experimental setup the data is splitted into three groups with the portion of 0.9, 0.05, 0.05 for training, testing and validation. learning rate of 0.0001 is chosen after some trial and error. choosing a large learning rate causes for the number of epochs for training, 10 is chosen. choosing a small number of epochs is because of the large language model that has been used, and was pre-trained on tweets which are the same data as our dataset. batch size of 32 is chosen. layers of 0 to 9 of xlm-t were freezed. in the next step, we used the same model that achieved the best results in the first set of experiments. finally, we decided to use mse-pearson(10,50) loss as it got the best pearson correlation in the previous step and run a new set of experiments with different number of hidden layers and neurons. as we can see in ## conclusion in this paper we present some models with different hyper-parameters for mint dataset for semeval 2023 task 9: multilingual tweet intimacy analysis and try to optimize the model with different loss. also we augment the dataset using translating existing tweets to other languages which affects the results. choosing the right learning rate also has effects on converging faster and the results. training the last few layers of xlm-t improves results.