KUL @ SMM 4 H 2024: Optimizing Text Classification with Quality-Assured Augmentation Strategies (2024)

Abstract: This paper presents our models for the Social Media Mining for Health 2024 shared task, specifically Task 5, which involves classifying tweets reporting a child with childhood disorders (annotated as “1”) versus those merely mentioning a disorder (annotated as “0”). We utilized a classification model enhanced with diverse textual and language model-based augmentations. To ensure quality, we used semantic similarity, perplexity, and lexical diversity as evaluation metrics. Combining supervised contrastive learning and cross-entropy-based learning, our best model, incorporating R-drop and various LM generation-based augmentations, achieved an impressive F1 score of 0.9230 on the test set, surpassing the task mean and median scores.

## introduction the social media mining for health (smm4h-24) @xcite shared task 5 aims to explore data sources for assessing the link between pregnancy exposures and childhood disorders in realtime and at scale. many children face lifelong disorders, with 17% in the u.s. diagnosed with developmental disabilities and 8% with asthma. this task involves a binary classification to automatically distinguish tweets from users who reported their pregnancy and have a child with childhood developmental disorders (annotated as "1"), from tweets that merely mention a disorder without evidence of a diagnosis (annotated as "0"). smm4h task 5 includes datasets of english tweets with annotations on the presence or absence of childhood disorders. recent studies @xcite @xcite have shown that data augmentation enhances training data diversity and model robustness. it's crucial to ensure the quality of these augmentations to maintain original meanings and diversify training data. augmented data using model-based augmentations together with regularised dropout (r-drop) @xcite serve as regularization methods, mitigating overfitting in machine learning models. model-based augmentations involve generating new data samples using pre-trained language models (lms) that have rich semantic and syntactic knowledge stored in their parameters. this can create variations of the original data while preserving its semantic content. ## methodology this section outlines our methodology consisting of 3 steps: 1. enhancing textual data with lm-based augmentation techniques, 2. assessing augmentation quality, and, 3. finetuning a transformer-based @xcite model bertweet @xcite integrating supervised contrastive loss, cross-entropy loss, and regularised dropout (r-drop) @xcite in the training process. ## experiments and results the dataset comprises a training set (7, 398 tweets), a validation set (389 tweets), and a test set (10, 000 tweets). for pre-processing, we removed urls, retweets, mentions, extra spaces, non-ascii words, and characters. we also lower-cased the text, trimmed white spaces, and inserted spaces between punctuation marks. for classification, each model was fine-tuned over 10 epochs with a learning rate of 5e -5 using the adam optimizer. the batch size is set to 32, and the maximum sequence length to 128. we use pytorch and huggingface foot_1 library for training the bertweet large model (nguyen et al., 2020), applying both cross-entropy loss and supervised contrastive loss @xcite . model checkpoints are saved every 200 step based on the validation set's f1-score. the bertweet loss function was adapted to include kl divergence for r-drop regularization. training data was enriched with lm augmentations from section 3 (lm-aug). quality checks are integrated into the augmentation pipeline to filter out poor quality augmentations (aug-ql). threshold values for the augmentation quality checks were empirically determined based on the performance of the validation set. this involved experimenting with different thresholds and observing their impact on the model's performance metrics (precision, recall, f1 scores) on the validation data. we set the following threshold values for augmentation quality check: semantic @xmath0.7, @xmath1, and 0.4 < lexical @xmath2.75. the semantic similarity threshold (> 0.7) ensures that the augmented text retains a high degree of similarity in meaning to the original text. the perplexity threshold (< 100) ensures that the augmented text is coherent and grammatically correct. we calculated the perplexity using a pre-trained gpt-2 model. lower perplexity indicates better language model prediction quality. when experimenting with higher thresholds the generated text was less meaningful. a threshold of 100 was chosen because it balanced coherence with augmentation diversity. the lexical diversity range (0.4 -0.75) ensures a balance between the diversity and relevance of the vocabulary used in the augmented text. a range of 0.4 to 0.75 was set to avoid too much similarity (which would defeat the purpose of augmentation) and too much dif-ference (which might change the context or meaning). these thresholds were optimized iteratively, with each adjustment followed by re-evaluating the model's performance on the validation set to ensure the thresholds contributed positively to the model's overall effectiveness. incorporating lm augmentations and r-drop into bertweet yielded improved performance for detecting childhood disorder diagnoses in tweets (see tables 1 and 2 ) compared to baseline model setup. augmentations diversified the data, enhancing the model's robustness and generalization. the metrics for evaluating augmentation quality further improved model performance by ensuring high-quality training data. the diversity from augmented data reduced the risk of overfitting and the model's reliance on specific patterns. contextually generated lm-augmented examples facilitated better language understanding. the integration of r-drop with supervised contrastive loss and crossentropy loss further promoted the learning of more generalized features by capturing various aspects of the data, enhancing the precision of the model. in the post-evaluation, the lm augmentations were performed only on the positive class (pos) which improved results. the results surpassed the shared task's mean and median scores, demonstrating the effectiveness of this approach. ## conclusion in this work, we developed a classification model enhanced with r-drop and lm-based augmentations to mitigate label imbalance and avoid overfitting, thereby improving performance. we incorporated evaluation metrics like semantic similarity, perplexity, and lexical diversity to ensure the quality of the augmentations, adding only those that meet set thresholds. our approach of integrating data augmentation and rigorous filtering strategies showed superior performance compared to the baselines we set up. this can be attributed to the enriched dataset, which reduces overfitting and enhances generalization. similar performance levels were achieved by the bertweet model in @xcite , highlighting that the choice of model architecture and hyperparameter tuning is crucial. however, our lmbased augmentations provide a significant edge in data diversity and model robustness. furthermore, by augmenting only the positive class, our model demonstrated even more significant improvements in the overall f1 scores. the use of lm-based augmentations introduced meaningful variations in the training data, helping the model learn to generalize better from a more diverse set of examples. the rigorous filtering strategies ensured that only highquality augmented data were used, preserving the original meaning and enhancing the model's ability to handle diverse inputs. our approach significantly enhances model generalizability, achieving an impressive f1 score of 0.923 on the test set.