GAV x at S em E val-2024 Task 10: Emotion Flip Reasoning via Stacked Instruction Finetuning of LLM s (2024)

Abstract: The Emotion Flip Reasoning task at SemEval 2024 aims at identifying the utterance(s) that trigger a speaker to shift from an emotion to another in a multi-party conversation. The spontaneous, informal, and occasionally multilingual dynamics of conversations make the task challenging. In this paper, we propose a supervised stacked instruction-based framework to finetune large language models to tackle this task. Utilising the annotated datasets provided, we curate multiple instruction sets involving chain-of-thoughts, feedback, and self-evaluation instructions, for a multi-step finetuning pipeline. We utilise the self-consistency inference strategy to enhance prediction consistency. Experimental results reveal commendable performance, achieving mean F1 scores of 0.77 and 0.76 for triggers in the Hindi-English and English-only tracks respectively. This led to us earning the second highest ranking in both tracks.

## introduction the ediref shared task at semeval 2024 @xcite encompasses two challenges in natural language processing (nlp): emotion recognition in conversation (erc) and emotion flip reasoning (efr). our work focuses on the latter challenge-efr, which aims at identifying the utterances responsible for triggering a shift in a speaker's emotional state, hereafter referred to as an emotion flip, within a multi-party conversation. the task offers two tracks: one involving hindi-english code-mixed conversations and the other focusing on english-only conversations. the first track particularly addresses the complexities 1 https://codalab.lisn.upsaclay.fr/co mpetitions/16769 inherent in multilingual contexts. each track comes with its respective dataset annotated by the organisers, wherein each utterance is labelled to represent one of the six primary emotional statesanger, disgust, fear, joy, sadness, and surprise @xcite . additionally, the emotion neutral is assigned to utterances devoid of any expressed emotion @xcite . given these emotion labels, locating the emotion flip is straightforward. our task is to identify the triggers behind it. figure 1 shows a hindi-english code-mixed conversation conducted between two speakers, complemented by english translations. during the chat, speaker a undergoes an emotion flip from sadness to joy, while speaker b transitions from surprise to joy. utterance u4 is identified as the trigger causing both speakers' emotion flip. particularly, when speaker b delivers utterance u4, their emotional state also undergoes a change, rendering u4 as a self-trigger for speaker b's own emotion flip. it is worth noting that, for an emotion flip, there can be no trigger utterances at all, or there can be one or multiple trigger utterances originating from any involved speakers, including themselves @xcite . the efr task can be formulated as follows: given a conversation between p speakers 𝑠 𝑖 𝑖=1..𝑝 involving q utterances 𝑢 & &'(..) , each assigned an emotion 𝑒 & &' (..) , if speaker 𝑠 " changes their emotion at utterance 𝑢 * , there may exist a set of utterances 𝑢 + , wherein 1 ≤ 𝑙 ≤ 𝑘, that trigger the emotion flip. if we use 1 to denote a trigger utterance and 0 to denote a non-trigger utterance, then the array [𝑡 ( , 𝑡 , , … , 𝑡 * ], in which 𝑡 -equals either 0 or 1 and its position in the array corresponds to the position of the utterance in the conversation, can conveniently represent the task's label for an emotion flip. for instance, considering the conversation in figure 1 , the array [0, 0, 0, 1, 0] indicates that the utterance at position 4 caused speaker a to shift from sadness to joy. in this paper, we introduce an instruction-based framework designed to finetune large language models (llms) for addressing the efr task. initially, leveraging the training data, we construct multiple distinct instruction sets to guide the model in identifying triggers for emotion flips. these instructs emulate human cognitive processes, incorporating both human feedback and selfevaluation procedures as integral components of the reasoning process. subsequently, we execute a supervised stacked finetuning pipeline to refine the model using these instructions. once the model is tuned, we employ an inference strategy called selfconsistency @xcite to generate predictions for the test data. besides the system description, we made the following observations in our experiments: ## related work the efr task was first introduced by @xcite , who utilised a masked memory network and a transformer-based architecture to tackle it. in subsequent research in 2023, they delved deeper into the instigators behind emotion flips and introduced a neural architecture named tgif. this architecture integrates transformer encoders and stacked gated recurrent units (grus) to comprehensively capture the conversation context, speaker dynamics, and emotional sequences. while efr remains a relatively recent task, it is closely related to the widely studied task of emotion-cause pair extraction (ecpe) @xcite . the objective of ecpe is to identify a text span that elicit a specific emotion @xcite . earlier endeavours to address ecpe using deep learning faced challenges associated with position bias @xcite . @xcite introduced ueca-prompt, a bert-based universal prompt tuning method. subsequently, @xcite proposed the decc framework, which incorporates inducing inference and logical pruning to guide llms to reason. both prompt-based approaches outperformed previous works on this task. the promising results observed in ecpe using prompt-based methods motivates us to adapt them to the efr task. prompt-based learning refers to prompting pretrained language models to tackle downstream tasks @xcite . recently, llms like gpt (openai, 2023) and llama @xcite demonstrate exceptional performance across various nlp tasks, even with zero-shot or few-shot prompts @xcite . several prompting techniques have emerged recently. chain-of-thoughts (cot) prompting, one of the most popular techniques, replicates human cognitive process by integrating intermediate reasoning steps @xcite . instead of attempting to reach the answer in a single leap, this approach encourages the model to divide complicated problems into smaller, more manageable components, imitating the way humans think. tree-of-thoughts prompting extends cot by constructing a tree of logical steps towards the solution @xcite . multimodal cot combines text and vision into a two-phase cognitive process @xcite . on the other hand, instead of fixed prompts, llms themselves can be used to dynamically generate prompts for downstream tasks @xcite or to produce and execute programming code as intermediate steps @xcite . interestingly, llms are demonstrated to be capable of generating and analysing recursive reasoning, a unique cognitive ability akin to human thinking processes @xcite . despite these emerging techniques, llms often generate outputs that deviate from the ground truth labels @xcite . to address this challenge, instruction tuning emerges as a solution, employing supervised learning on a set of instructions to narrow the gap between the output generated by the base llms and the desired output @xcite . additionally, human and augmented feedback play a crucial role in mitigating this issue. akyurek et al. (2023) introduced a reinforcement learning framework equipped with a critique generator to guide gpt-3 in improving its output. @xcite proposed the active-prompt method, which entails human manual annotation of uncertain rational chains. furthermore, @xcite devised a novel framework that freezes llms and integrates reasoning steps from an external program. these prior studies underscore the significance of furnishing high-calibre instructions and feedback, as well as employing suitable prompting techniques, to achieve the desired output with llms. ## our system in this section, we describe the general approach and the implementation of our system. ## conclusion & future work the paper presents an instruction based llm finetuning framework to address the efr task. our strategy employs a multilayered finetuning pipeline, utilising three diverse instruction sets to steer the model towards recognising emotion flip triggers and, and finalised with the application of the sc inference strategy. the framework benefits significantly from the provision of high-quality instructions, as evidenced by the progressively improved performance of our model as betterquality feedback and instructions are incorporated into the finetuning pipeline. the robustness of our framework is demonstrated by its proficient handling of both english-only and hindi-english code-mixed datasets, affirming its effectiveness in varied linguistic scenarios. through these findings, we trust that our study makes a meaningful impact on the field of prompt-based learning techniques by harnessing the evolved capabilities of llms. moving forward, our focus will be on an indepth exploration of various instruction types to devise the optimal way to amalgamate them for the most generalisability. furthermore, we plan to develop a systematic method for constructing a processing pipeline tailored to this task and potentially applicable to related nlp tasks. this pipeline will be designed to encompass a cot prompts, incorporate feedback mechanisms, and integrate self-evaluation instructions to ensure a robust, repeatable process for enhancing model performance.