UTR ad- NLP at # SMM 4 H 2024: Why LLM -Generated Texts Fail to Improve Text Classification Models (2024)

Abstract: In this paper, we present our approach to addressing the binary classification tasks, Tasks 5 and 6, as part of the Social Media Mining for Health (SMM4H) text classification challenge. Both tasks involved working with imbalanced datasets that featured a scarcity of positive examples. To mitigate this imbalance, we employed a Large Language Model to generate synthetic texts with positive labels, aiming to augment the training data for our text classification models. Unfortunately, this method did not significantly improve model performance. Through clustering analysis using text embeddings, we discovered that the generated texts significantly lacked diversity compared to the raw data. This finding highlights the challenges of using synthetic text generation for enhancing model efficacy in real-world applications, specifically in the context of health-related social media data.

## introduction in recent years, the burgeoning field of social media mining has opened new avenues for health-related research @xcite , providing rich data sources for public health surveillance, including understanding public health trends and individual health behaviors. the social media mining for health (smm4h) initiative, through its various tasks, aims to leverage these data sources to address pertinent health questions @xcite . this paper focuses on our approaches to tasks 5 and 6, both of which present unique challenges and opportunities in the realm of text classification for health-related social media mining. task 5 targets the binary classification of tweets related to children's medical disorders, differentiating between tweets that report a genuine diagnosis and those that only mention these disorders without a diagnosis. task 6 involves identifying the exact ages from social media posts, crucial for health research applications and enabling more accurate analysis of age-related health outcomes and behaviors in observational studies. for both tasks, the challenge of imbalanced datasets is prominent. to address this, we employed the large language model (llm), gpt-4, aiming to augment our training data with synthetic positive examples to balance the dataset and enhance the performance of our binary classification models. despite these efforts, our initial results were underwhelming, as the synthetic texts generated by gpt-4 lacked the diversity found in the raw data. this finding raises important questions about the practical challenges and limitations of using synthetic data augmentation in real-world applications, particularly in the nuanced field of health-related social media mining. ## discussion & conclusion the data provided for the tasks 5 and 6 were both imbalanced datasets with less positive examples than negative ones. to address such imbalances, data augmentation can be an effective approach. previous methods using deep learning have been proposed as techniques applicable to named entity recoginition and text classification, such as labelwise token replacement, synonym replacement, and entity replacement within sequences @xcite @xcite . there have also been proposals for approaches combining context and entity levels using llms @xcite . in this study, we proposed a method that employs few-shot learning techniques to generate new text using llms @xcite . by leveraging llms, a vast amount of completely new data can be generated, and by producing high-quality data, improvements in the performance of classification models can be expected. however, in this study, the addition of synthetic data did not significantly improve the performance compared to the baseline model. as evident from the visualization by clustering, the synthetic data generated by gpt-4 for the tasks 5 and 6 exhibits a localized distribution in the embeddings extracted from the pre-trained language model, compared to the original data. this suggests that the synthetic data lacks diversity in comparison to the original data, and as a result, the addition of the synthetic data did not improve the overall diversity of the training data, resulting in no explicit improvement in the model's performance. this study has several limitations. since we used only gpt-4 for generating synthetic data from sources like reddit and twitter, future work should explore new methods to increase diversity. prior research suggests that diversity can be enhanced by specifying attributes @xcite . for instance, identifying the characteristics of posters or the types of children's diseases could lead to greater diversity. future efforts could include using multiple language models, presenting more original examples, or adjusting hyperparameters like temperature to improve data diversity. the tasks 5 and 6 involved relatively few positive examples, resulting in imbalanced data, which is often the case in real-world settings. while the use of synthetic data generated by language models can be an effective solution for augmenting training data, this study suggests that various efforts are necessary to create diverse data that is effective for training language models. 6. output6 7. output7 8. output8 9. output9 10. output10