UZH _ CL yp at S em E val-2023 Task 9: Head-First Fine-Tuning and C hat GPT Data Generation for Cross-Lingual Learning in Tweet Intimacy Prediction (2023)

Abstract: This paper describes the submission of UZH_CLyp for the SemEval 2023 Task 9 “Multilingual Tweet Intimacy Analysis. We achieved second-best results in all 10 languages according to the official Pearson’s correlation regression evaluation measure. Our cross-lingual transfer learning approach explores the benefits of using a Head-First Fine-Tuning method (HeFiT) that first updates only the regression head parameters and then also updates the pre-trained transformer encoder parameters at a reduced learning rate. Additionally, we study the impact of using a small set of automatically generated examples (in our case, from ChatGPT) for low-resource settings where no human-labeled data is available. Our study shows that HeFiT stabilizes training and consistently improves results for pre-trained models that lack domain adaptation to tweets. Our study also shows a noticeable performance increase in cross-lingual learning when synthetic data is used, confirming the usefulness of current text generation systems to improve zeroshot baseline results. Finally, we examine how possible inconsistencies in the annotated data contribute to cross-lingual interference issues.

## introduction social media texts are a rich source for studying social behavior, but manual analysis of them is prohibitively expensive. predictive models that accurately estimate regression scores, such as intimacy, emotion, and valence, enable large-scale studies of social behavior through social media text. sparse training data in numerous languages presents another challenge to globalize such studies of social behaviors. to overcome such limitations, this shared task focuses on developing models capable of accurately regressing intimacy in languages without training data. in our shared task submission, we explore a novel approach to fine-tuning multilingual language models to maximize cross-lingual learning abilities. the key contributions and findings of this study are the following: our results suggest that this approach yields slightly better performance in challenging cross-lingual learning of unseen languages. ## related work cross-lingual learning involves the transfer of knowledge between natural languages, which can be achieved through the use of multilingual language models (mlms). mlms are pre-trained in multiple languages and then fine-tuned using available training data, with the aim of achieving positive transfer to languages with limited or no training data. one such mlm is xlm-r @xcite , an xlm model @xcite pre-trained with roberta objectives @xcite , which has shown good multilingual performance. another model, xlm-t, extends the xlm-r model with additional pre-training on tweets, enhancing its performance on multilingual and monolingual tasks @xcite . recent advances in multilingual models aim to enhance their adaptation to new languages by using of lightweight language adapter parameters @xcite . subsequent research has aimed to improve the performance of these models in cross-lingual transfer tasks by implementing language adapters whose weights are generated by contextual parameter generation networks @xcite . this approach has also been extended to handle multilingual multitask learning scenarios @xcite . our hypothesis is that the adaptor-free architecture of mlms is adequate to support crosslingual learning when accompanied by our proposed staged fine-tuning and synthetic data generation pipeline that allows the models to improve performance in an unseen language with minimal compromise of performance for the training languages. ## conclusion this system description paper presents and discusses our submission, which was ranked second and achieved a pearson's r of 0.614 in the multilingual twitter intimacy regression task of se-meval 2023. we propose a fine-tuning approach that improves xlm-r's adaptation to twitter and demonstrates minor improvements in cross-lingual learning. additionally, we generate a mini-dataset of synthetic tweets in all 10 task languages using chatgpt, which suffers only minor performance loss in a few-shot scenario against manually annotated training data. finally, we observe indications of self and cross-lingual interference for romance languages in both models, and we would like to investigate if this phenomenon is replicable in other tasks and datasets.