Hidetsune at S em E val-2024 Task 3: A Simple Textual Approach to Emotion Classification and Emotion Cause Analysis in Conversations Using Machine Learning and Next Sentence Prediction (2024)

Abstract: In this system paper for SemEval-2024 Task3 subtask 2, I present my simple textual approach to emotion classification and emotion cause analysis in conversations using machine learning and next sentence prediction. I train a SpaCy model for emotion classification and use next sentence prediction with BERT for emotion cause analysis. While speaker names and audio-visual clips are given in addition to text of the conversations, my approach uses textual data only to test my methodology to combine machine learning with next sentence prediction. This paper reveals both strengths and weaknesses of my trial, suggesting a direction of future studies to improve my introductory solution.

## introduction semeval 2024 task 3 @xcite calls for assigning an emotion to each utterance and extracting its emotion cause in conversations. subtask 2, which i participate in, requires emotion classification and identification of emotion cause utterances with audio-visual clips available whereas subtask 1 requires identification of specific textual span as well without audio-visual clips. i participate in subtask 2, for which a speaker name, text and an audio-visual clip are given for each utterance. instead of not identifying specific cause span in the emotion cause utterance in this subtask, i set a limitation to use textual data only while audio-visual data are also available. therefore, my methodology uses textual data of utterances only as input to classify emotions and identify cause utterance numbers as output. for this reason, training data from subtask 1, for which video names are not given, are used instead of data from the subtask i participate in. while the task @xcite prohibits use of additional annotation data, i overlooked the sentence stating the rule and mistakenly used additional data for my solution. i would like to show my appreciation for the task organizers and readers acknowledging and understanding my mistake of using additional data. for data preparation, official training data for subtask 1 (csv converted version) @xcite , training data (translated and csv converted version) from semeval2024 task10 subtask 1 (erc) by @xcite and data by nikam (n.d.) are used. they all are concatenated in that order and adjusted so that the resulting dataset has first 7001 neutral utterances (including 7000th counting from 0) and first 5001 utterances at maximum for each emotion other than neutral. then, spacy-v3 model @xcite is trained using the adjusted training data for emotion classification. in addition to that, next sentence prediction (cathrine, 2023) is used for identification of emotion cause utterances. in that step, i decided to simplify the methodology by hypothesising that the emotion cause utterance is the utterance itself or its previous utterance. with this simple assumption, my algorithm checks the relatedness of each utterance and its previous utterance using next sentence prediction (cathrine, 2023), which returns true or false. previous utterance is chosen as cause utterance if the two utterances are deemed related, and the utterance itself if not. the result shows a limited performance of my introductory solution, but it also clarifies a direction to its improvement. although my combined methodology has a large room for improvement, it does have a potential in its simplicity and limitation to use textual data only. this paper aims to share an experimental trial to test my combined methodology, guiding a direction to its future application and improvement. my code is available on github foot_0 . ## background the subtask i participate in (subtask 2) focuses on emotion classification and emotion cause analysis with text data and audio-visual clips. subtask 1, on the other hand, does not allow participants to use audio-visual clips and requires extracting specific textual cause spans as well. my participation in subtask 2 sets a limitation to use textual data only, which means that it is substantially the same as subtask 1 except that i do not extract specific textual cause spans. training data from subtask 1 are used instead of that from subtask 2 because they seem to be identical to each other except that they have no video names in the dataset. therefore, my methodology for subtask 2 uses data from subtask 1 and additional data from other sources for training. given the evaluation dataset with audio-visual clips available, my methodology, which is trained by textual data only, assigns an emotion category and its cause utterance as output using textual data of the evaluation dataset. this task is technically a mixture of two topics, which are emotion classification and emotion cause analysis. as for emotion classification, many previous studies have been conducted especially on social media including twitter and facebook. for instance, a work by @xcite classifies text on social media into six emotion categories with high accuracy. another study by @xcite investigates in people's emotions during crises using a support vector machine. in addition to its use for social media, its application to real conversations is also getting an attention. a study by @xcite , for example, applies emotion detection to social robotics, aiming to improve its ability to interpret feelings of humans from a viewpoint of nlp methods. there are many previous studies for emotion cause analysis too. a study by @xcite , for example, uses hierarchical neural network to get high accuracy. another study by @xcite adopts a complicated approach, resulting in reliable accuracy. on the other hand, this paper aims to test a simple approach to combine classical machine learning method with next sentence prediction with a certain assumption. my methodology has a strength in its simplicity, but the result shows a large room for improvement. ## system overview the main strategy of my system is a combination of classical machine learning method with next sentence prediction. machine leaning is used for emotion classification and next sentence prediction is used for identification of emotion cause utterances. audio-visual clips are available in this subtask, but only textual data of the utterances are used for my solution. a quick overview of my combined methodology is as follows. in the first step, training data are prepared from multiple sources. the official training data from subtask 1 @xcite are imported as a json file and converted into a pandas dataframe with text and an emotion for each utterance. here, data from subtask 1 are used instead of that from subtask 2 because it is likely that the data are identical to each other except that video names are not given for data of subtask 1. then, the resulting pandas dataframe, translated and converted version of training data from semeval2024 task10 @xcite and data by nikam (n.d.) are imported via csv file format. as for the two additional datasets, irrelevant columns are dropped so that they are composed of two columns, in which text and emotions are stored respectively. they are concatenated into one dataframe and adjusted to have 7001 utterances (including 7000th counting from 0) for neutral and 5001 at maximum for each one of the other emotions related to this task (surprise, anger, sadness, joy, disgust and fear). after this process of data preparation, spacy-v3 model @xcite is trained using the prepared training data. an unlabeled evaluation dataset is imported as a json file, and the trained model assigns an emotion to each utterance. at the same step, next sentence prediction with bert (cathrine, 2023) assigns an utterance number of emotion cause to each utterance if its assigned emotion is not "neutral". as stated before, it is hypothesised that the emotion cause utterance is either the utterance itself or its previous utterance. under this assumption, next sentence prediction (cathrine, 2023) checks whether or not an utterance that it is looking at is related to its previous utterance. the previous utterance is chosen as its emotion cause utterance if these are deemed related, and the utterance itself is chosen if not. after all these processes, lists that include emotions with the utterance numbers and emotion cause utterance numbers (['2_sadness', '1'] for example) are added to the original evaluation data for submission. my participation in this task using the combined methodology reveals its limitations of the simple approach to emotion cause analysis. since my methodology trains spacy-v3 model with over 33000 utterances, it is more natural to assume that the simplistic application of next sentence prediction is the main reason for the limited accuracy. my algorithm takes only the utterance itself and its previous utterance into account as possible emotion cause utterances. this premise does not allow my solution to cover cases where one utterance has an influence beyond multiple utterances, limiting the ability to deal with the entire conversation from a macroscopic point of view. on the other hand, there is no doubt that the accuracy of emotion classification is also a reason for the limited ability of my trial. emotion cause utterances are assigned to non-neutral emotion utterances only, meaning that the algorithm loses its accuracy for both emotion classification and emotion cause analysis at a time if the trained model mistakenly assigns "neutral" to non-neutral emotion utterances. ## experimental setup for the emotion classification part, multiple datasets needed to be processed to make an adjusted training dataset. first of all, the official training data for subtask 1 @xcite are imported as a json file. in the dataset, a conversation id is assigned to each conversation, and one conversation has multiple utterances. for each utterance, an utterance id, text, its speaker name and an emotion are assigned. the json file is converted into a pandas dataframe to make the data easier to deal with. conversation ids, utterance ids and speaker names are dropped from the dataframe so that it has only "text" and "emotion" as columns. data from semeval2024 task10 @xcite , which have hindi-english code-mixed utterances, are translated into english and converted into a pandas dataframe similarly. data by @xcite are also imported as a pandas dataframe, and irrelevant columns of the two additional datasets are dropped so that they have text utterances and emotions as columns only. the number of utterances for each different emotion category is as shown on table1. after these processes, the three dataframes (official training data for subtask 1, data from semeval2024 task10 @xcite and data by @xcite ) are concatenated into one dataframe in that order. only first 7001 (including 7000th counting from 0) neutral emotion utterances and first 5001 utterances for each one of the other emotions are extracted, dropping all the utterances that exceed the limitation from the concatenated dataset to compose an adjusted training data. after this data preparation step, spacy-v3 model @xcite is trained using the adjusted training data, and the trained model is used for emotion classification of the unlabeled evaluation dataset. next sentence prediction (cathrine, 2023) is also used for emotion cause analysis as stated in the previous section. ## results in the evaluation phase, my solution was tested using the unlabeled test dataset. the result shows a limited ability of my approach, which combines classical machine learning with next sentence prediction under a simplistic assumption. ## conclusions to summarize, my methodology sets a limitation to use textual data only, testing a simple algorithm with a certain premise. i use classical machine learning for emotion classification, and next sentence prediction for identification of emotion cause utterances. the next sentence prediction (cathrine, 2023) in my simple approach takes only the utterance itself and its previous utterance into account, limiting its ability to cover the entire conversation from a macroscopic viewpoint. in addition to that, the accuracy of emotion classification between neutral and non-neutral turned out to be more important than previously thought since it has a significant effect on identification of emotion cause utterances as well. although the trial of my simple approach has a large room for improvement, it clearly guides a direction to its future studies. with improvements to enhance the ability to cover conversations from a macroscopic point of view, it might open the door for the potential of my combined methodology.