interrupt-driven@ SMM 4 H ’24: Relevance-weighted Sentiment Analysis of R eddit Posts (2024)

Abstract: This paper describes our approach to Task 3 of the Social Media Mining for Health 2024 (SMM4H’24) shared tasks. The objective of the task was to classify the sentiment of social media posts, taken from the social anxiety subreddit, with reference to the outdoors, as positive, negative, neutral, or unrelated. We classified posts using a relevance-weighted sentiment analysis, which scored poorly, at 0.45 accuracy on the test set and 0.396 accuracy on the evaluation set. We consider what factors contributed to these low scores, and what alternatives could yield improvements, namely: improved data cleaning, a sentiment analyzer trained on a more suitable data set, improved sentiment heuristics, and a more involved relevance-weighting.

## introduction data was taken from the social anxiety subreddit where posts mentioned particular keywords relating to the outdoors. the task was to classify each text as either talking about the outdoors positively with regards to social anxiety, negatively, neutrally, or unrelated to the outdoors or anxiety @xcite . for example this post: "going for a walk in the rain can be really nice and refreshing so long as i'm wrapped up tight and dry in it." should be classified as positive; "i felt this. one thing i'm always afraid of when going outside is to meet people that i know, especially those of my age." should be classified as negative; "yeah my eyes are very sensitive when the wind hits or the sun is out." should be classified as neutral; and "..... run like da wind" would be unrelated. the approach we took was to classify sentences based on their sentiment (section 2.2), weight each sentence based on its relevance (section 2.3), and build a classification for the post based on the average score of each sentence (section 2.4). we pursued such an approach because, if successful, it has the following advantages: firstly that it is computationally light, requiring minimal resources to be able to run; secondly it is intuitive, making it easy to explain the reasoning for any given classification it makes; and finally it is decomposable, so that each part can be iterated on and improved upon. ## system description here we describe the steps we took to build our relevance-weighted sentiment analysis model. ## evaluation the resulting scores from our model were low. on the validation data, this approach scored as follows: label accuracy unrelated (0) 0.55 positive (1): 0.33 neutral (2) 0.18 negative (3) 0.55 total 0.45 and on the evaluation data, it scored as follows: score value f1 score 0.358 precision 0.365 recall 0.411 accuracy 0.396 as discussed in each section, the system and tools chosen did not fit well with the type of data provided. we consider the approach to be one that could work if the suggested improvements are made, such as improved data cleaning, a sentiment analyzer trained on a more suitable data set, improved sentiment heuristics, and a more involved relevance-weighting. as a comparison with other methods, here are the mean and median results of all the teams: mean median f1 score 0.5186 0.5795 precision 0.5649 0.63 recall 0.5379 0.5885 accuracy 0.5746 0.627 ## conclusion we use a relevance-weight sentiment analysis approach for classifying the sentiment of reddit posts with respect to social anxiety and the outdoors. even though our particular implementation produced poor results, there are opportunities for improvement in almost every stage of the approach: improved data cleaning, a sentiment analyzer trained on a more suitable data set, improved sentiment heuristics, and a more involved relevance-weighting each improve the overall performance of such an approach.