PCIC at SMM 4 H 2024: Enhancing R eddit Post Classification on Social Anxiety Using Transformer Models and Advanced Loss Functions (2024)

Abstract: We present our approach to solving the task of identifying the effect of outdoor activities on social anxiety based on reddit posts. We employed state-of-the-art transformer models enhanced with a combination of advanced loss functions. Data augmentation techniques were also used to address class imbalance within the training set. Our method achieved a macro-averaged F1-score of 0.655 on the test data, surpassing the workshop’s mean F1-Score of 0.519. These findings suggest that integrating weighted loss functions improves the performance of transformer models in classifying unbalanced text data, while data augmentation can improve the model’s ability to generalize.

## introduction this paper addresses task 3 of the 9th social media mining for health (smm4h) @xcite workshop at acl 2024, which focuses on analyzing the impact of outdoor activities on social anxiety through the lens of reddit posts. despite advances in natural language processing, accurately classifying such nuanced data poses significant challenges due to linguistic variability and data imbalances. previous studies have shown that there exist various possibilities to address data imbalance in classification tasks. for example, in @xcite , additional samples for underrepresented classes were generated. in @xcite , random under-sampling and synthetic minority oversampling techniques were employed. other authors proposed to tackle the class imbalance problem by introducing loss functions that focus on the underrepresented classes @xcite . our study builds on these works by incorporating a novel combination of weighted loss functions within a transformer model to address the challenge of imbalanced data. ## task description the aim of this task is to classify into four classes if an outdoor activity mentioned in a post had a positive, neutral, negative, or unrelated impact on the person's social anxiety symptoms. the outdoor activity mentioned in every text was given as a keyword in an extra column, so that the dataset consisted of the columns 'id', 'text', 'keyword' and 'label'. the training dataset includes 1800 posts, and the validation and test set includes 600 posts each. the keyword is highly important in the classification task since a post can be highly negative, but still, in only one sentence, the user mentions the positive effect of an outdoor activity. in this case, even though the entire text itself was negative, it should be classified as positive. so, it was important to somehow link the classification task to the keyword. another challenge was to tackle the highly imbalanced classes. the training dataset containing 1800 samples has 1131 texts for the class 'unrelated', 395 for class 'neutral', 160 for class 'positive' and only 114 for class 'negative'. ## methodology for the experiments run to solve task 3, different transformer models of the huggingface library were used. a small model (distilbert), two medium size models (roberta and xlnetbase), and a larger model (xlnet-large) were employed @xcite @xcite . these models were modified in some experiments to use a combined loss function. this combined loss function consists of the focal-loss, designed to address class imbalance by increasing the importance of hard-to-classify examples, weighted-cross-entropy loss, which assigns different weights to classes based on their representation in the training data, and weighted-smooth-cross-entropy loss. this variation adds smoothing to the class labels to improve generalization @xcite . to combine these three loss functions, the mean of the three values is computed after each batch. the preprocessing of the text data included cleaning the text of any urls, extra whitespaces and performing a conversion of the emojis used to text using the python library 'emoji' @xcite . apart from that, only the sentence(s) in each text containing the keyword and their previous and next sentence were used. this adaptation was made to reduce the text's length to prevent a substantial part of the text from being truncated in the tokenization process. the previous and next sentences were included to provide more context for the keyword phrases. the keyword was appended to the input text by using the model-specific separator token. for training and validation, the training dataset (1800 instances) was divided into 70% for training and 30% for validation with a seed of 42. the macro-averaged f1 score was evaluated on the validation dataset (600 instances) to test the model performance. as a termination criterion, the f1 score was used for early stopping with a patience of 6 epochs. for the learning rate, the value of 5e-6 was the best result of a hyperparameter optimization and was therefore chosen for the experiments. the batch size was set to either 16 or 32 depending on the max-length parameter of the tokenizer due to hardware restrictions. for some experiments, data augmentation was performed to address the class imbalance. on one hand, an augmentation by paraphrasing was used. on the other hand, augmentation was performed by punctuation insertion, random deletion, random insertion, and random swapping (from now on re-ferred to as 'traditional augmentation'). for the paraphrasing task, a finetuned model from the huggingface repository was used, which is based on the t5-base model and finetuned on paraphrases generated by chatgpt @xcite . for the punctuation insertion augmentation, the punctuation marks [',', '.', '!', '?', ';'] were inserted at a random position in the text with a frequency of 10% in relation to the number of words in the text. in the random deletion augmentation, each word in the text is deleted with a probability of 20%. in the randominsertion augmentation, four random words in the text were chosen, and then, using the wordnet of the library 'nltk', synonyms for these words were searched @xcite . one of the synonyms found was randomly inserted in the text for each of the four words. for the random swapping task, two random words were chosen in the text and then swapped. ## results and discussion first, a baseline classification (run 0) was run with the four different models. in this experiment, the original loss function of the model was used, and no data augmentation was performed. for the following experiments, we aimed to evaluate the effect of the combined loss function (run 1), the max-length parameter of the tokenizer (run 2), data augmentation using paraphrasing (run 3), and data augmentation using the aforementioned traditional augmentation (run 4). finally, an experiment was run where all these methods were combined, using the combined loss function, increased value for the max-length parameter, and both data augmentation methods (run 5). the results of these experiments are shown in changing the max-length parameter of the tokenizer from 128 to 256, yielded the best result in all experiments for xlnet-large, lifting the f1 score up from 0.55 to 0.60. the motivation of this experiment surged from an analysis that we conducted, showing that 334 texts (18.5%) of the tokenized text lengths in the training dataset surpass the previously set length of 128 for the xlnet tokenizer (see figure 1 ). for comparability, the same maxlength parameter was set for the experiments with distilbert and roberta, even though these models work with a different tokenizer and tokenized text length might differ. augmentation by paraphrasing only showed a slight improvement for one model, xlnet-large. at the same time, traditional augmentation showed a mixed effect on the f1 score across the different models. in the experiment with xlnet-large, traditional augmentation seems to have a negative impact on the f1 score, decreasing it from 0.55 to 0.53. in the following experiment, all of the previous modifications were combined. hence, the combined loss function was used, the max-length parameter of the tokenizer was set to 256, and the data were augmented using both augmentation methods. despite the slightly decreased performance on the validation data in this experiment, it is possible that due to the augmentation, the model's ability to generalize improves. therefore, this set-up with the model xlnet-large was used to obtain the predictions on the test set (0.655 f1 score) and was sent to the task organizers. ## conclusion this paper presents our approach to solving task 3 of the smm4h 2024 workshop, which consists of a unique pre-processing to avoid truncating important information and providing only the key phrases of the text to the model. furthermore, the xlnet model was adapted to use a combined loss function, and data augmentation was performed to address the class imbalance and improve generalizability. the keyword related to the outdoor activity was appended to the input text using the separator token. this setup has resulted in a macro-averaged f1 score of 0.655 on the test data, outperforming the mean of 0.519.