1024m at SMM 4 H 2024: Tasks 3, 5 & 6 - Self Reported Health Text Classification through Ensembles (2024)

Abstract: Social media is a great source of data for users reporting information and regarding their health and how various things have had an effect on them. This paper presents various approaches using Transformers and Large Language Models and their ensembles, their performance along with advantages and drawbacks for various tasks of SMM4H’24 - Classifying texts on impact of nature and outdoor spaces on the author’s mental health (Task 3), Binary classification of tweets reporting their children’s health disorders like Asthma, Autism, ADHD and Speech disorder (task 5), Binary classification of users self-reporting their age (task 6).

## introduction social media has become a key way for people to share their experiences and feelings. this has opened up new opportunities for researchers to understand how different aspects of life affect our well-being. the paper explores three tasks of smm4h 2024 @xcite -4-way classification of texts based on effect of nature, outdoor spaces and activities on author's mental health (task 3), binary classification of texts reporting health disorders in author's child including adhd, autism, asthma and speech disorder (task 5) @xcite , binary classification of texts self-reporting author's exact age directly / indirectly (task 6).the paper explores usage of transformer models like roberta @xcite , deberta @xcite , longformer @xcite and lls including both proprietary and open-source like @xcite , claude-opus @xcite , llama-3 8b @xcite , mistral 7b @xcite , gemma 7b(gemmateam, 2024), and ensembles along with advantages and drawbacks of each approach using the models. similar previous works can be found in @xcite , @xcite and @xcite . ## datasets the dataset for task 3 consists of 3000 reddit posts from r/socialanxiety belonging to four classes based on self reported impact of outdoor spaces and activities on the author's mental health -0: unrelated to the task, 1: had a positive impact, 2: is neutral or had no effect, 3: had a negative effect. the dataset for task 5 consists of 9734 tweets belonging to two classes -1: users reporting having a child having adhd, asthma, autism or speech disorder and the rest as class 0. similarly for task 6, the dataset of 21200 texts consists of both tweets and reddit posts from r/askdocs for two classes -class 1 being texts through which the author's current age in years may be determined and rest as class 0. the distribution of labels for the three tasks can be seen in for task 3, two approaches were tested. one where classification was done directly in a 4-way and the other where classification was done is two stages, this involved first classifying the text whether it is related to the task or not i.e class 0 or not and then classifying the effect on the user in the second stage. for task 5 and 6 it was done directly as a binary classification task 12 . in llm approaches, the proprietary versions were used as zero-shot and the rest of the llms were tested in a zero-shot and fine-tuned manner. additionally they were tested in a two stage classification for task 3. in the case of ensembles, it was through majority voting in a set of models, through and-rule for high precision requirement and through or-rule for high recall requirements. for task 5 and 6, while using llms, classification was done by dividing the criteria into parts and aggregating the individual results. i.e in the case of task 5, individual prompts test for each condition that needed to be satisfied to classify as positive and and-rule is used for generating final label. similarly or-rule was used for task 6. the performance of different approaches can be seen in ## error analysis the llms performed equally good on all kinds of data while transformers models performed less effectively when the kind of language used is off from rest of the data or when criteria for classification was mentioned in one sentence and referred to the conditions indirectly later on. it was observed that positively labelled samples were predicted correctly by either the llm approach or transformers, hence ensembles of both had recall over 0.99 with just 1 percent drop in f1 scores in task 5 and 6. many of the positively misclassified samples were in the format of advertisements where the title appears to match the criteria for positive classification. this is one area where llms were still able to distinguish effectively while other models did not. ## conclusion the performance of some of the models compared to others on the test set can be seen in it is likely the performance would be better that the current models in full precision. many of the positive label texts have been filtered out during the data collection process. for example, texts selfreporting age in text format instead of numerical. due to this, a higher focus on recall is necessary. a custom metric with higher importance to recall is better suited for task 5 and 6 compared to f1 scores. ensemble approaches like majority voting and filtering guaranteed positive label texts using llm predictions could improve performance without a significant drop in the f1 scores. finally, the performance improved on all the tasks while using dev set as additional training data compared to just the training data, hinting at the possibility of improving the performance by adding more training data. augmentation through paraphrasing existing data however did not improve the results.