YNU - HPCC at S em E val-2023 Task 9: Pretrained Language Model for Multilingual Tweet Intimacy Analysis (2023)

Abstract: This paper describes our fine-tuned pretrained language model for task 9 (Multilingual Tweet Intimacy Analysis, MTIA) of the SemEval 2023 competition. MTIA aims to quantitatively analyze tweets in 6 languages for intimacy, giving a score from 1 to 5. The challenge of MTIA is in semantically extracting information from code-mixed texts. To alleviate this difficulty, we suggested a solution that combines attention and memory mechanisms. The preprocessed tweets are input to the XLM-T layer to get sentence embeddings and subsequently to the bidirectional GRU layer to obtain intimacy ratings. Experimental results show an improvement in the overall performance of our model in both seen and unseen languages.

## introduction intimacy involves lots of social information (e.g., social norms). @xcite ) by evaluating the intimacy score of tweets crawled on social media sites like twitter quantitatively, it is possible to mine more meaningful social information. mtia aims to build a sentiment analysis model (or system) to predict the intimacy scores, ranging from one to five, of tweets in a twitter dataset containing ten languages, including english, spanish, portuguese, italian, french, chinese, hindi, dutch, korean and arabic. among those ten languages, the first six are observable (i.e., appear during training), and the rest are unobservable (used for zero-shot learning). tweets abound with online slang, emojis and face characters, which usually contain extra intimate thoughts. thus, it is critical to extract semantic information from those unconventional texts. however, many accessible pre-trained models (e.g., xlm) lack consideration of non-traditional text in the training process. the non-traditional text needs elimination to de-noise when we perform transfer learning on these models. another feasible approach is to convert these non-traditional symbols to conventional words. however, the extraction of information from face characters remains a challenge. fortunately, we can perform transfer training using the xlm-t model @xcite trained on the twitter dataset, thus avoiding information loss in the symbolic text. additionally, we suggest a better way to link bidirectional gru @xcite layers at the xlm-t. the approach, which combines attention and memory mechanisms, is inspired by the analogy to humans. we will demonstrate experimentally that this measure enhances the model's overall performance, both seen and unseen text. our model achieved 13/45 places in terms of overall performance in the leaderboard, with both dutch and arabic ranking fourth. the code will be available at https://github . com/cskujou/tweet-intimacy. the roadmap for this paper is as follows. section 2 describes the work related to sentiment analysis. section 3 describes the architecture of our model. section 5 reports the experimental results. section 6 concludes the paper. ## related works in the past, researchers have used lexicon-based methods and machine-learning approaches for sentiment analysis. @xcite since then, the performance of sentiment analysis tasks has significantly increased because of the development of deep learning algorithms. @xcite , lstm @xcite , and gru @xcite are examples of deep learning models. @xcite proposed a contextual sentiment embedding model that can distinguish the meaning of the same word in different contexts and improve the performance of the sentiment task. recently, by resolving the issue of long-distance dependency, bert-based large-scale pre-training models achieved a significant advancement in sentiment analysis (zheng @xmath0... ... r[cls] r1 r2 r3 @xmath1... ... xlm-t cell cell cell cell cell cell linear layer intimacy score cat cat cell cell ... ... cell cell sentencepiece tweet tokenizer xlm-t model bigru figure 1: model architecture of xlmt-gru et al. @xcite . early multilingual nlp models included mbert @xcite , xlm @xcite , and mbart @xcite . goyal et al. merged the xlm and the roberta model to create the more potent xlm-r @xcite . @xcite obtained the xlm-t using an identical architecture to the xlm-r, trained on the twitter dataset. the xlm-t has a significant improvement in its ability to extract symbolic information. ## model description the xlm-t model @xcite was applied as the baseline model. moreover, we connect two more bidirectional gru layers on top of xlm-t to get the improved model called xlmt-gru. figure 1 illustrates its architecture. we'll go into more detail about this model in the following paragraphs. ## experimental setup in this section, we first introduce the dataset and the evaluation metric and then go over the model implementation detail. next, we demonstrate the improvement of our model over the baseline model through experiments. further, we will include two models for comparison purposes. the experiments are divided into two parts. the first part reveals the performance of our model on the observable language, while the second part shows the performance on the unobservable language. ## conclusion in this paper, we study the multilingual intimacy analysis task. we attempted to combine xlm-t and bigru and found that the overall performance of our model outperformed the baseline model. our model ranked 13 th on the leaderboard and 11 th on unobservable languages. combining at-tentional and memory mechanisms can boost the performance of our model for sentiment analysis on small datasets. in the future, we will also try to integrate other models (e.g., treelstm), mainly to complement the shortcomings of our current model on hindi.