Enhancing Social Media Health Prediction Certainty by Integrating Large Language Models with Transformer Classifiers (2024)

Abstract: This paper presents our approach for SMM4H 2024 Task 5, focusing on identifying tweets where users discuss their child’s health conditions of ADHD, ASD, delayed speech, or asthma. Our approach uses a pipeline that combines transformer-based classifiers and GPT-4 large language models (LLMs). We first address data imbalance in the training set using topic modelling and under-sampling. Next, we train RoBERTa-based classifiers on the adjusted data. Finally, GPT-4 refines the classifier’s predictions for uncertain cases (confidence below 0.9). This strategy achieved significant improvement over the baseline RoBERTa models. Our work demonstrates the effectiveness of combining transformer classifiers and LLMs for extracting health insights from social media conversations.

## introduction the smm4h shared tasks focus on using machine learning and natural language processing to solve the challenges associated with extracting health insight from social media @xcite . this paper presents our work in smm4h 2024 task 5. the focus of task 5 is to identify tweets where users talk about their own child having attention-deficit/hyperactivity disorder (adhd), autism spectrum disorders (asd), delayed speech, or asthma, and differentiate them from tweets that simply discuss these conditions in general. for a description of this dataset, see @xcite . one of the main challenges of processing social media text is dealing with the lay language that often lacks proper grammar and structure, containing misspellings, abbreviations, and unconventional phrasing @xcite . additionally, texts from advertising, news, social bots, and other non-personal posting modalities need to be excluded when identifying personal conversations @xcite . this requires robust algorithms capable of understanding colloquial language while maintaining accuracy in analysis and interpretation @xcite . in this paper, we present a pipeline that combines transformer-based classifiers with the gpt-4 llm to classify the tweets containing the health conditions of interest. ## method our approach consisted of three main steps. first, for data preparation we used a topic-modelling based under-sampling method to address data imbalance issues. secondly, we trained a robertabased classifier on this adjusted training data. finally, we employed gpt-based llms to refine the classifiers' predictions where the classifier's confidence in its predictions dropped below a threshold of 0.9. ## results the twitter-roberta classifier performed well independently, while the llm showed comparatively lower performance when assessed against the validation dataset. however, by using the llm for those texts where the classifier was less certain, we gained 2 percentage points over using the classifier only -this model is depicted as llm+clf in ## conclusions while standard transformer models excel at the classification task, using llms as arbiters on less interpretable texts can be beneficial. with good prompt design an llm can be tuned to discern subtleties in texts that are difficult to teach a classifier without extensive training data. however, llms on their own may not be ideal for all classification tasks due to limitations in overall accuracy, efficiency, and resource requirements.