LT 4 SG @ SMM 4 H ’24: Tweets Classification for Digital Epidemiology of Childhood Health Outcomes Using Pre-Trained Language Models (2024)

Abstract: This paper presents our approaches for the SMM4H’24 Shared Task 5 on the binary classification of English tweets reporting children’s medical disorders. Our first approach involves fine-tuning a single RoBERTa-large model, while the second approach entails ensembling the results of three fine-tuned BERTweet-large models. We demonstrate that although both approaches exhibit identical performance on validation data, the BERTweet-large ensemble excels on test data. Our best-performing system achieves an F1-score of 0.938 on test data, outperforming the benchmark classifier by 1.18%.

## introduction & motivation chronic childhood disorders like attentiondeficit/hyperactivity disorder (adhd), autism spectrum disorders (asd), delayed speech, and asthma significantly impact a child's development and well-being, often extending into adulthood. approximately 1 in 6 (17%) children aged 3-17 years in the united states experience a developmental disability, with adhd, asd, and others contributing to this statistic @xcite . in previous studies @xcite @xcite , twitter data have been utilized to identify self-reports of the aforementioned disorders; however, the identification of reports concerning these disorders in users' children has not been explored. it may be of interest to explore twitter's potential in continuing to collect users' tweets postpartum, enabling the detection of outcomes in childhood. ## preliminary experiments each selected model was trained for 3 iterations, with 10 epochs per iteration. at the end of each epoch, its f1-score was recorded. the f1-score for each model was determined based on its performance with the validation dataset. we saved the best-performing epoch (i.e., the best f1-score for the positive class) for each model in each iteration. the results are shown in as shown in ## results and conclusion since roberta-large best-run and bertweetlarge ensemble are performing equally well on the validation data, we tested the performance of both classifiers on unseen, unlabeled test data. as shown in previously, authors @xcite have achieved an f1-score of 0.92 using a bertweetlarge classifier on the same dataset. we believe that our approach achieved better results for several fundamental reasons. first, we performed more thorough hyperparameter tuning compared to the previ-ous authors. better-optimized hyperparameters can significantly improve model performance. second, by creating an ensemble of bertweet-large models from the three best epochs of three runs, we captured more robust and generalized patterns in the data. ensemble methods typically outperform individual models because they reduce variance and mitigate the risk of overfitting to specific subsets of data. third, a likely reason is the diversity in model runs. each run of our bertweet-large model may have encountered slightly different initialization and training dynamics (e.g., random seed), resulting in diverse decision boundaries. combining these diverse models helps in making more accurate predictions by leveraging the strengths of each individual model. when fine-tuning complex pre-trained language models, one issue on small datasets is the instability of the classification performance. to overcome this, we combined the predictions of multiple bertweet-large models in an ensemble. by doing so, we achieved significantly better results in terms of f1-score for smm4h'24 task 5. for future work, it's interesting to investigate how the system performance varies when adding more bertweetlarge iterations (i.e., runs) to the ensemble.