SMM 4 H 2024: 5 Fold Cross Validation for Classification of tweets reporting children’s disorders (2024)

Abstract: This paper presents our system developed for the Social Media Mining for Health (SMM4H) 2024 Task 05. The task objective was binary classification of tweets provided in the dataset, distinguishing between those reporting medical disorders and those merely mentioning diseases. We address this challenge through the utilization of a 5-fold cross-validation approach, employing the RoBERTa-Large model. Evaluation results demonstrate an F1-score of 0.886 on the validation dataset and 0.823 on the test dataset.

## introduction recent advancements in natural language processing have been groundbreaking, simplifying classification systems while making them more resilient to complex challenges. integrating real-world data with nlp techniques has enhanced system efficiency, and for practical tasks such as intent recognition, sentiment analysis, or sentence classification, the roberta model @xcite and its variants like roberta-small and robertalarge have demonstrated outstanding performance. these models' comprehension mechanisms have consistently been the preferred choice for these applications. social media platforms, such as twitter, consistently produce vast quantities of data in various formats, including text. extensively utilized for sharing and disseminating users' opinions on numerous topics, twitter data has led to initiatives like the social media mining for health application (smm4h) shared task in 2024 @xcite . this paper outlines our approach for shared task 05: binary classification of tweets that report children's medical disorders (in english). ## results after applying the model parameters and conducting 5-fold cross-validation on both the validation and test datasets, we achieved an f1 score of 0.886 with precision of 0.831 and recall of 0.948 on the validation dataset. on the test dataset, the model yielded an f1 score of 0.823, with precision and recall values of 0.721 and 0.959, respectively. ## discussion despite achieving an average f1 score of 0.823, our methodology falls short of matching the baseline model performance. several factors may contribute to this disparity. one significant factor is the suboptimal choice of the learning rate. in our methodology, the learning rate was not effectively optimized, potentially impeding the model's ability to converge to the global minimum of the loss function efficiently. consequently, the model may have struggled to achieve optimal parameter settings.furthermore, the decision not to implement text conversion to lowercase could have limited the standardization of the model's vocabulary and text, potentially hindering the training process. additionally, not setting the tokenization length to the maximum of 512 tokens, as recommended for roberta, may have reduced the model's ability to capture the complete contextual meaning of tweets, thus possibly leading to suboptimal performance.our methodology consistently achieves an impressive recall score across the training, validation, and test datasets. this can be attributed to the roberta model's dynamic masking, which adeptly captures intricate linguistic nuances. consequently, while our model excels in identifying relevant cases, it also tends to exhibit higher false positive rates, thereby impacting precision.