STFXNLP at S em E val-2025 Task 11 Track A: Neural Network, Schema, and Next Word Prediction-based Approaches to Perceived Emotion Detection (2025)

Abstract: In this work, we discuss our models that were applied to the SemEval-2025 Task 11: Bridging the Gap in Text-Based Emotion Detection (Muhammad et al., 2025b). We focused on the English data set of track A, which involves determining what emotions the reader of a snippet of text is feeling. We applied three different types of models that vary in their approaches and reported our findings on the task’s test set. We found that the performance of our models differed from each other, but neither of our models outperformed the task’s baseline model.

## introduction detecting emotion in text is a complex task, as it can be challenging to identify an emotion someone is trying to convey in one snippet. it is essential to accurately identify emotions in a text as it has implications in healthcare, social science, humanities, narrative analysis, and several other fields @xcite . it can be easier to detect emotion when someone explicitly states that they are mad or upset or uses adjectives such as delighted, pleased, or glad. "it broke my heart and nearly ruined me" clearly displays sadness and fear. however, a snippet such as "colorado, middle of nowhere" can make it difficult to determine how people will perceive the reader's emotions. the task we focused on was track a: multi-label emotion detection, which involves determining the perceived emotions of a speaker of a snippet of text. for the english dataset, there are five possible emotions a snippet can classify as (anger, fear, joy, sadness, and surprise). this is an any-of classification; therefore, each snippet can be labelled with more than 1 emotion. in our work, we applied three different models to task a on the dataset containing only english text. we wanted to compare three different models that varied in their approach, which included a standard feed-forward neural networkbased model, a model that leveraged the idea of schemas -a key part of how humans detect, classify, and process emotions (leahy, 2018) -and a model that relied on the next-word prediction of a large language model. furthermore, each model relied on different types or amounts of data which is discussed in section 4 and section 5. our models did not perform well, but our best model_ffnn model scored 82 out of 98 participants for the task a english dataset. the model_ffnn performed best with respect to the f1-score on the test set, achieving 64.7% foot_0 . it performed best on the fear class, with an f1-score of 79.03%. the fear class was the most frequent class in the training set, with 1,611 of 2,768 samples having the fear label. our model_ffnn's lowest f1-score was on anger, with only 49.68%. anger had the lowest number of samples, with only 333 samples of 2768 having the anger label. ## background there were three tracks for this shared task. we focused on track a: multi-label emotion detection. within this track, we considered the english dataset. the training set contained 2,768 samples of text snippets taken from reddit foot_1 , as well as personal narratives, talks, and speeches. each snippet is made up of 1 to 4 sentences. snippet length varies between 3 words to 81 words @xcite . the number of samples labelled with each emotion is (anger: 333, fear: 1611, joy: 674, sadness: 878, surprise: 839). the number of samples labelled with each emotion for both the training set and test set can be seen in ## system overview this section discusses the three different models we applied to the semeval 2025 task 11-track a english dataset. ## experimental setup for initial training to gauge the performance of model_ffnn and cosine, these two models were trained on 80% (2214 samples) of the training set, then performance was calculated on the other 20% (554 samples) of the training. pytorch foot_8 version 2.5.0+cpu was used for model_ffnn. scikit foot_9 learn version 1.4.0 was used for the svm involved with models model_ffnn and cosine. for training the feed-forward neural networks, data was batched into groups of 16 and shuffled. we tuned the dropout rate to 0.3 as it yielded the best f1score and accuracy results. we found that increasing the size of the hidden layers from 256 to 512 also increased the model's performance. initially, we used a single feed-forward neural network with an input size of 768 and an output size of 5. however, this performance was low, so we moved to five separate feed-forward neural networks binary probabilistic classifiers. we found we had the best results when using all five outputs provided by the ffnns to determine the binary classification for each emotion. both models model_ffnn and cosine were trained on the full training dataset before being submitted on the final test set. model_nwp uses a threshold k, which determines whether or not an emotion is assigned based on if their corresponding word, such as afraid for the fear class, appears in the top k candidate words for next word prediction. to tune our threshold k, we observe the performance (f1-score, precision, and recall) of the model on a set of samples that includes samples from the provided training and development set by the task prior to testing with respect to the fear class. limiting the model to tune k to one emotion removes the need to have seen other emotions that might be in the test set and allows it to handle unseen emotions. however, we did observe the performance of the model with different k values averaged across all emotions before submitting to the task's test set to check that the model was performing near its potential across all emotions in the set that combines the provided training set and development set. we found that the best k value for the fear class was 110, which is true for both the set that combined the provided training and development set and only the training set 11 . 110 was also the best k value for the performance averaged across all emotions on both sets of samples. further analysis showed that the best k value differs among different emotions. for this model, we only use a k value of 110 when applied to the test set. ## results in this section, we discuss the performance of our models. model_ffnn was our best-performing model with respect to the macro f1-score and the perclass performance. the model_cosine performed better than model_ffnn in accuracy on anger and joy and only marginally worse in accuracy on the other emotions. our models performed worse than the task's baseline model rembert @xcite by 6.22%. the model_cosine and model_ffnn performed poorly on recall except when classifying fear, with model_cosine r and model_ffnn achieving a recall of 77.78% and 81.28&, respectively. precision and recall for both models were approximately even for fear, 11 there were some k values near 110 that had the same rounded f1-score. ## conclusion our models underperformed compared to the task's baseline model, with our best model placing 82 out of 98 on track a: multi-label emotion detection. model_ffnn could have underperformed due to the small number of samples. there were only 333 samples labelled as anger out of a total of 2,768 samples, which could have biased this model towards classifying samples as not anger. there is a similar case for the other emotions. model_cosine could have underperformed for a similar reason. our models recruited the use of three different approaches, which included feed-forward neural networks, schemas, and next-word prediction. although their performance was relatively poor, it would be interesting to examine the effects of data augmentation to decrease the class imbalances with the binary classification setup in future work. additional datasets could also be recruited to provide more training data, which could assist with the supervised models model_ffnn and model_cosine. other embedding models could also be considered for our models. future experiments could also observe if datasets containing audio can be used to assist with a similar classification task by leveraging tonality and emphases to improve the performance of the schema model.