H ale L ab_ NITK @ SMM 4 H ’24: Binary classification of E nglish tweets reporting children’s medical disorders (2024)

Abstract: This paper describes the work undertaken as part of the SMM4H-2024 shared task, specifically Task 5, which involves the binary classification of English tweets reporting children’s medical disorders. The primary objective is to develop a system capable of automatically identifying tweets from users who report their pregnancy and mention children with specific medical conditions, such as attention-deficit/hyperactivity disorder (ADHD), autism spectrum disorders (ASD), delayed speech, or asthma, while distinguishing them from tweets that merely reference a disorder without much context. Our approach leverages advanced natural language processing techniques and machine learning algorithms to accurately classify the tweets. The system achieved an overall F1-score of 0.87, highlighting its robustness and effectiveness in addressing the classification challenge posed by this task.

## introduction the proliferation of social media platforms such as twitter (now known as x), reddit, and facebook has led to an unprecedented surge in user-generated content. millions of individuals publicly share their thoughts, experiences, and health-related information online, which presents a unique opportunity to analyze and investigate public health trends and issues. among these platforms, twitter stands out as a particularly valuable source of rich information for both the general public and researchers. by analyzing tweets, researchers can gain insights into various health-related phenomena, track the spread of diseases, monitor public sentiment toward health policies, and identify emerging health concerns. the real-time, streaming nature of twitter data makes it an indispensable tool for public health surveillance and research, facilitating a deeper understanding of health behaviors and outcomes on a global scale @xcite . the reporting of children's medical disorders stands out as a crucial area of study, given the importance of early detection, diagnosis, and treatment in pediatric healthcare. many children are diagnosed with disorders that can profoundly impact their daily lives and may persist throughout their lifetimes. conditions such as attention-deficit/hyperactivity disorder (adhd), autism spectrum disorders (asd), delayed speech, and asthma are frequently mentioned by parents and caregivers on social media. studying these discussions on twitter provides valuable insights into the prevalence and public perception of these disorders @xcite . it also helps in understanding the challenges faced by families and the effectiveness of various interventions, through analysis of patterns and trends in symptom reporting, treatment experiences, and support mechanisms, which are essential for improving healthcare strategies and policies @xcite . furthermore, by monitoring these conversations, researchers can identify gaps in awareness and education, potentially guiding more targeted and effective public health campaigns. ## smm4h'24 task 5 -description task 5 is a binary classification task that involves automatically distinguishing tweets posted by users who have reported their pregnancy on twitter and specifically mention children with adhd, asd, delayed speech, or asthma (annotated as "1"), from tweets that merely mention a disorder (annotated as "0"). this task enables the large-scale utilization of twitter, not only for epidemiologic studies but also to explore parents' experiences and directly target support interventions. the dataset (klein et al., 2023) consists of 7,398 english language tweets for training, 389 tweets for validation, and 10,000 tweets for testing purposes. tweets in which parents explicitly mention that their child is suffering from adhd, asd, delayed speech, or asthma are annotated as '1'. in contrast, other tweets are annotated as '0', which may or may not have mention of a disorder. by applying natural language processing (nlp) techniques and machine learning algorithms to this dataset, we aim to develop a robust model capable of accurately identifying and categorizing tweets related to children's medical disorders. ## methodology this binary classification task involves automatically distinguishing tweets posted by users who had reported their pregnancy on twitter and mentioned that their children had adhd, asd, delayed speech, or asthma in other tweets. various preprocessing techniques were employed on the tweets during the data processing phase to ensure they were standardized and prepared for analysis the names of disorders and digits were standardized. additionally, terms referring to a child, such as, son, child, daughter, etc., were unified to the common term "child" since the focus is on identifying tweets about child disorders posted by their parents. furthermore, urls, usernames, hashtags, emojis, and smileys were eliminated using the tweet-preprocessor library in python @xcite . common abbreviations and contractions found in tweets, such as "lol", "thx", "btw", and "we're" were expanded to their full forms, while non-alphanumeric characters and extra white spaces were pruned. the text was converted to lowercase, elongated words were corrected by keeping the occurrence of repeated characters to two, and lemmatization was performed using the spacy lemmatizer @xcite . these steps standardized the text format and improved its suitability for subsequent analysis and modeling. experiments were carried out using various transformer-based models to model the tweets. roberta-base @xcite was chosen for its ability to capture contextual information effectively and was implemented using the huggingface toolkit @xcite for classifying tweets mentioning children's medical disorders. after preprocessing the textual data as elaborated earlier, the text sequences were subjected to tokenization using the roberta tokenizer, with the maximum text length set to 128. model optimization was achieved using the adam optimizer with a batch size of 8 and a learning rate of 1e-5. the training was carried out for up to 10 epochs, with early stopping triggered by validation set performance and a patience value set at 4 epochs. moreover, a dropout rate of 0.3, determined iteratively, was applied to regularize the model. the model architecture is built on roberta-base, incorporating an additional hidden dense layer with a rectified linear unit (relu) activation function. a sigmoid activation function is used in the output layer. the experiments were conducted on google colab using a t4 gpu as the hardware accelerator. ## results and discussion the system's performance was evaluated on both the validation and test sets. initially, the performance on the validation set was compared to assess the model's effectiveness before evaluating it on the test set for submission. efforts towards hyperparameter tuning helped achieve optimal performance on the validation set. key hyperparameters, such as dropout rates and the number of hidden layers, were varied systematically to enhance the model's performance. this method enabled systematic exploration of hyperparameter configurations to determine the most effective settings based on validation set performance. the evaluation of the model's performance was based on the f1-score metric. the f1-score is a critical metric for this binary classification task because it balances precision and recall, offering a unified measure that considers both false positives and false negatives. this is particularly vital for distinguishing tweets about children with specific medical disorders from general mentions of disorders, ensuring that the model accurately identifies relevant tweets and minimizes the misclassification of non-relevant ones. precision and recall scores are reported alongside the f1-score to comprehensively evaluate the model's performance in terms of the accuracy of positive predictions and the model's ability to capture all relevant instances. the results obtained from evaluating the system on the validation set are reported in ## concluding remarks in this article, an approach to accurately distinguishing between tweets that mention specific disorders in the context of parenting and those that merely reference a disorder, using advanced nlp techniques and transformed-based models is presented. evaluation on both the validation and test sets demonstrates the system's reliability, with consistent f1-scores indicating its effectiveness in generalizing to unseen data. moving forward, we aim to explore further refinements to the model architecture, incorporating additional features, and expanding the training dataset to enhance the system's performance.