L y S at S em E val-2024 Task 3: An Early Prototype for End-to-End Multimodal Emotion Linking as Graph-Based Parsing (2024)

Abstract: This paper describes our participation in SemEval 2024 Task 3, which focused on Multimodal Emotion Cause Analysis in Conversations. We developed an early prototype for an end-to-end system that uses graph-based methods from dependency parsing to identify causal emotion relations in multi-party conversations. Our model comprises a neural transformer-based encoder for contextualizing multimodal conversation data and a graph-based decoder for generating the adjacency matrix scores of the causal graph. We ranked 7th out of 15 valid and official submissions for Subtask 1, using textual inputs only. We also discuss our participation in Subtask 2 during post-evaluation using multi-modal inputs.

## introduction semeval 2024 task 3 focused on multimodal emotion cause analysis in conversations @xcite . figure 1 shows an example provided by the organizers to illustrate the task. two subtasks were proposed: subtask 1, which uses only textual inputs, and subtask 2, which allows for the consideration of video and audio processing as well. the shared task is timely given the recent success of multimodal architectures combining computer vision @xcite , natural language processing @xcite , and speech processing @xcite advancements. in the particular context of multimodal emotion analysis, the task builds on top of previous work such as recognizing the triggered emotions as a classification task @xcite or predicting complex causeeffect relations between speakers @xcite . for the particular case of the shared task, the dataset -centered in english -relies on @xcite and provides text, image and audio inputs. the goal of the task consists of predicting (i) the emotion associated to each utterance within the conversation, (ii) the cause-effect relations that trigger the emotions between utterances and (iii) the associated span in the cause utterance. we had time and resources only to build a textual model for official participation in subtask 1. we validated some multimodal baseline approaches using vision and audio inputs, but the computational resources required to fine-tune text and video data were beyond our range, so we participated in subtask 2 only during post-evaluation. in what follows, we describe our approach. the implementation of our early prototype can be found at https://github.com/anaezquerro/semeval24-task3 . ## background multimodal emotion cause analysis a number of datasets collecting multi-party conversations @xcite @xcite have been published to train and test multimodal neural architectures. simpler configurations involve recognizing the speaker emotion at each utterance -this task is commonly known as emotion recognition (er) @xcite while others require a deeper level of understanding to model interactions and causal relations between speakers -emotion-cause pair extraction (ecpe) @xcite . the most common approaches follow an encoder-decoder neural architecture where the encoder is conformed by multiple modules -one module per input modality (text, image and/or audio) -and produces an inner representation at utterance level; and the decoder accepts the encoder outputs as inputs and returns a suitable output adapted to the specifications of the targeted task. in the context of @xcite proposed a gcn-based decoder to capture temporal relations @xcite , while dutta and ganapathy (2024) used cross-attention to fusion the input modalities and a final classification layer to predict the targeted emotions. approaches in ecpe require an extra effort to represent and model causal information: wei et al. (2020) scored all possible utterance tuples to predict the most probable list of emotion-cause pairs. other authors, like @xcite and @xcite , represented the emotion-cause pairs as a labeled graph between utterances and tried to predict the set of causal edges using a gcn or a transition-based system, respectively. the semeval 2024 task 3 joins the recognition and causal extraction tasks and challenges a system able to both model speaker emotions and elicit relations. graph-based decoding for structured prediction tasks, such as dependency parsing, graphbased approaches are a standard for computing output syntactic representations @xcite . particularly, @xcite introduced a classifier that computes a head and dependent representation for each token and then uses two biaffine classifiers: one computes a score for each pair of tokens to determine the most likely head, and the other determines the label for each head-dependent token pair. we will also build upon a biaffine graph-based parser: we will frame the task as predicting a dependency graph, where utterances are the nodes and emotions are dependency labels between pairs of utterances. ## system overview our system consists of two modules: a large pretrained encoder and a graph-based decoder (see figure 2 ). it can add extra input channels into the encoder without requiring any adjustments to the decoder, so the same decoder is used for both tasks, while the encoder is adapted to incorporate textonly (subtask 1) or multimodal (subtask 2) inputs. figure 2 : high-level architecture of our system. the encoder takes as input the sequence of m utterances of a given conversation and returns a unique vector representation for each utterance. the decoder uses the utterance embedding matrix to apply the affine attention product in the decoder, obtain the scores of the adjacent matrix and return the predicted sequence of emotions and the cause relations between utterances. let @xmath0, ..., u m ) be a conversation of m utterances, where each utterance ## results for approaches (i), (ii) and (iii), only bert was fine-tuned, whereas for approach (iv), all pretrained weights were frozen. these weights solely served to contextualize input information, with the learning process confined to the decoder component. our top-performing model for the first subtask achieved a validation score of 9.75 and ranked in the evaluation set in 7th position among 15 participants with 6.77 points. we observed a slight performance improvement by increasing the hidden dimension of the decoder. thus, considering the expansion of decoder layers could improve the performance. it is worth noting the significant impact of span prediction on the model performance: the proportional results consistently outperform strict metrics. removing span prediction while retaining only text inputs results in a notable increase in f-score (20.43 points for the second subtask), indicating the crucial role of span prediction in model learning. furthermore, we noticed that there was a consistent delay in the alignment between recall and precision metrics, with precision consistently exceeding recall by more than 5 points across all approaches. this suggests that our system tends to adopt a conservative behavior, avoiding the number of false cause emotion predictions. the best validation performance for the second subtask is achieved through the integration of text and audio, yielding a score of 23.36 points in the weighted f-score. using image data also improves the text-only baseline, though unexpectedly lags behind the audio model. it is important to note that these two approaches are not directly comparable due to differences in their data inputs: the text and image model only considers a fixed number of sampled frames, suggesting that providing more image data (ideally, the full sequence of frames) could potentially yield a better performance that surpasses the audio-based approach. unfortunately, we could not fine-tune bert with the full multimodal encoder, so we were restricted to projecting the multimodal inputs to their respective contextualizations, and relying on the trainable weights of the decoder to optimize the full architecture. the results prove the importance of, at least, finetuning the text encoder: the f-score only reaches 11.25 points, whereas the text finetuned baseline nearly doubles its performance with 20.43 points, highlighting the insufficient context of the original pretrained bert embeddings to address this task. once the post-evaluation period concluded, we upload an experimental submission of our best multimodal system to the official competition. we obtained 15.32 points in the weighted f-score, positioning our baseline in the 13th place out of 18 participants. ## conclusion we proposed a graph-based prototype for the analysis emotion-cause analysis in conversations. given the limited preparation time, we only submitted official results for subtask 1 (text-only), but also report post-evaluation results for subtask 2 (multimodal). the task required predicting several aspects of the conversation: (i) the emotion associated with each utterance, (ii) the cause-effect relationships triggering these emotions between utterances, and (iii) the specific span within the cause utterance responsible for the emotion. we achieved 7th place out of 15 valid submissions for subtask 1, a promising outcome considering the time and resource constraints we had to prepare the task. yet, our results make us optimistic about exploring future research avenues to enhance our system and study lighter approaches that can perform competitively. as future work, we aim to experiment with smaller and distilled models to encode textual, visual, and audio inputs, enabling us to fine-tune the full model cheaply.