C og AI @ SMM 4 H 2024: Leveraging BERT -based Ensemble Models for Classifying Tweets on Developmental Disorders (2024)

Abstract: This paper presents our work for the Task 5 of the Social Media Mining for Health Applications 2024 Shared Task - Binary classification of English tweets reporting children’s medical disorders. In this paper, we present and compare multiple approaches for automatically classifying tweets from parents based on whether they mention having a child with attention-deficit/hyperactivity disorder (ADHD), autism spectrum disorders (ASD), delayed speech, or asthma. We use ensemble of various BERT-based models trained on provided dataset that yields an F1 score of 0.901 on the test data.

## experiments and results during our model exploration, we fine-tuned multiple pre-trained models including mentalbert @xcite , psychbert @xcite , twitterbert @xcite , and distilbert @xcite . we trained each of them for 200 epochs with a learning rate of 1e-5 and a batch size of 32. each of these models offers unique advantages that could be beneficial for our purpose. mentalbert and psychbert are tailored for mental health and biomedical text datasets which could potentially capture features relevant to our task of identifying tweets related to "developmental disorders". twitterbert's training on twitter data makes it adept at handling the informal language of tweets which can improve classification accuracy while distilbert's computational efficiency can make it suitable for scalable deployment, an advantage in handling large volumes of twitter data. we systematically evaluated the performance of each of these models by adopting an incremental approach. initially, we trained the models using the raw data without any pre-processing (m1). subsequently, we performed training after augmenting the data (m2). finally, we conducted training using cleaned data with data augmentation (m3). this comprehensive approach allowed us to assess the impact of classification performance of each model, providing valuable insights into their robustness and effectiveness for our specific task. our final model then was an ensemble of all these methods assigned weights according to their "f1-scores" on validation dataset. we hardcoded weights of 0.6 score to mentalbert, 0.2 score to twitterbert, 0.1 score to each psychbert and distilbert. this was decided based on their individual performance as can be seen in ## conclusion in this task, we experimented with various bert models and presented the results in