Dolomites@# SMM 4 H 2024: Helping LLM s “Know The Drill” in Low-Resource Settings - A Study on Social Media Posts (2024)

Abstract: The amount of data to fine-tune LLMs plays a crucial role in the performance of these models in downstream tasks. Consequently, it is not straightforward to deploy these models in low-resource settings. In this work, we investigate two new multi-task learning data augmentation approaches for fine-tuning LLMs when little data is available: “In-domain Augmentation” of the training data and extracting “Drills” as smaller tasks from the target dataset. We evaluate the proposed approaches in three natural language processing settings in the context of SMM4H 2024 competition tasks: multi-class classification, entity recognition, and information extraction. The results show that both techniques improve the performance of the models in all three settings, suggesting a positive impact from the knowledge learned in multi-task training to perform the target task.

## introduction collecting an adequate amount of data to fine-tune large language models (llm) in low-resource settings is an expensive practice. to address the lack of enough data in such settings, data augmentation techniques have been commonly used in different natural language processing tasks @xcite as a low-cost solution. such techniques focus on generating new examples close to the original samples and data distribution, thus increasing the number of training samples. more recently, data augmentation has been studied from the perspective of multi-task learning. in multi-task learning data augmentation (mtl-da), the dataset of the target task is augmented with the data of other auxiliary tasks that can improve the model performance, despite having different characteristics @xcite . the model is then jointly optimized in a multi-task manner on the augmented data, resulting in more robust performance compared to traditional augmentation techniques @xcite . in this work, we investigate two new mtl-da techniques to fine-tune llms for low-resource tasks: a) in-domain augmentation where we augment the original target dataset with the data (and tasks) collected from the same source (e.g. reddit social forum); and b) drills where we extract from the target task a set of smaller tasks and replicate the dataset to cover the corresponding samples for the extracted drills. we study the proposed techniques on three language processing tasks in the context of smm4h 2024 challenge foot_0 : i) task 3: four-class classification of reddit social forum posts; ii) task 4: an entity recognition task on reddit posts to identify two entity types @xcite ; iii) task 6: information extraction from tweets and reddit posts. a complete description of the tasks is presented in §a. the results indicate the effectiveness of both augmentation techniques by achieving higher f 1 scores on the validation sets, compared to target task fine-tuning (note that the competition test sets are not publicly available), as well as scoring two to five points higher than overall scores in smm4h competition. ## evaluation since the test sets of the competition tasks are not publicly available, we analyze the performance of the model and the impact of the mtl-da techniques on the validation set. we consider the vanilla fine-tuning of the model for the target downstream task as the baseline in our analysis. furthermore, we compare the performance of the model with gpt-4 foot_2 via in-context learning (icl) in {0,5}-shot settings in each task as a low-cost alternative. we then present the scores obtained by the best-performing models and mtl-da techniques in smm4h competition, compared to the overall mean and median scores obtained by all participants. ## conclusion we studied two new multi-task learning data augmentation techniques to address the lack of adequate data to fine-tune llms in low-resource settings. we evaluated the proposed techniques in three tasks in the context of the smm4h 2024 competition, and we observed a) the positive impact of augmentation techniques in improving model performance; and b) the potential of in-context learning as a low-cost surrogate to fine-tuning. nevertheless, besides further ablation studies, there are a few unanswered questions: a) "what is the best set of drills given a task?"; b) "when should a dataset be considered in-domain?"; c) "would the same task on different domains also contribute positively?"; d) "how does the introduced approach compare to existing data augmentation techniques?"; and e) "does the replication of the dataset potentially lead to overfitting?". we aim to explore these questions in future work.