Affect Recognition in Conversations Using Large Language Models (2024)

Abstract: Affect recognition, encompassing emotions, moods, and feelings, plays a pivotal role in human communication. In the realm of conversational artificial intelligence, the ability to discern and respond to human affective cues is a critical factor for creating engaging and empathetic interactions. This study investigates the capacity of large language models (LLMs) to recognise human affect in conversations, with a focus on both open-domain chit-chat dialogues and task-oriented dialogues. Leveraging three diverse datasets, namely IEMOCAP (Busso et al., 2008), EmoWOZ (Feng et al., 2022), and DAIC-WOZ (Gratch et al., 2014), covering a spectrum of dialogues from casual conversations to clinical interviews, we evaluate and compare LLMs’ performance in affect recognition. Our investigation explores the zero-shot and few-shot capabilities of LLMs through in-context learning as well as their model capacities through task-specific fine-tuning. Additionally, this study takes into account the potential impact of automatic speech recognition errors on LLM predictions. With this work, we aim to shed light on the extent to which LLMs can replicate human-like affect recognition capabilities in conversations.

## introduction affect refers to the broad range of subjective experiences related to emotions, moods, and feelings @xcite . it encompasses the various ways individuals perceive, experience, and express their emotional states and is an essential aspect of human experience and communication @xcite . the ability to recognise human affect is an important ability of conversational artificial intelligence @xcite . it empowers the dialogue agent to go beyond mere information exchange and engage users on an emotional level. by leveraging affect recognition techniques, they can discern the emotional nuances in user inputs, including sentiment, mood, and subtle cues like sarcasm or frustration @xcite . this capability allows the system to respond with greater sensitivity, empathy, and relevance, leading to more meaningful and satisfying interactions @xcite . large language models (llms) have demonstrated promising performance in many tasks @xcite . they have also shown promising capability in adapting to new tasks via prompting @xcite , in-context learning @xcite , as well as task-specific finetuning @xcite . with the advancement in llms, it is possible to use llms as the backend of dialogue systems @xcite @xcite . this brings up the question: can llms recognise human affects in conversations in a similar capacity as human beings? in the context of conversational ai, dialogues can be broadly categorised into two classes: 1) chitchat or open-domain dialogues where users interact with the system for entertainment and engagement, and 2) task-oriented dialogues (tods) where users converse with the system for specific goals @xcite . under tods, depending on the type of user goals, dialogues can be further grouped as information-retrieval, medical consultations, education, and many more. regarding the affective information in conversations, we are particularly interested in the following: (1) categorical emotion classes from generic emotion models such as "basic emotions" proposed by @xcite , (2) custom categorical emotion classes defined for a particular context, such as the emotion labels defined by @xcite to encode task performance simultaneously in tods, and (3) depression, a medical illness that negatively affects how a person feels, thinks and acts, and causes feelings of sadness and/or a loss of interest in activities the person once enjoyed (amer-ican psychiatric association, 2020). the emergence of llms has signified a shift of paradigm from training small models for one specific task to large models for multiple tasks. therefore, in this work, we investigate the affect recognition ability of a range of llms on vastly different types of dialogues and labels foot_0 to ascertain the validity of this direction. specifically, ## methodology the ability of human-beings to recognise affect can be reflected in many ways. yet, being able to narrate what emotion has been expressed in the utterances of the other interlocutor is a straightforward and strong sign of such an ability. therefore, we took llms' ability to verbalise the emotion given the dialogue context as a proxy to both qualitatively and quantitatively analyse llms' ability for affect recognition. ## results and discussions in this section, we aim to answer the questions below. full results can be found in appendix b. ## conclusion in this study, we explore the performance of llms for affect recognition in three distinct types of dialogues: chit-chat dialogues, information-seeking tods, and medical consultation dialogues for depression. we conduct benchmark experiments on these datasets using five llms: llama-7b, alpaca-7b, llama-3-8b, gpt-3.5, and gpt-4. we also explore various setups, including zeroshot learning, few-shot in-context learning, and task-specific fine-tuning, all facilitated by specially designed prompts. additionally, we examine the impact of asr errors on llms' zero-shot performance. our zero-shot experiments underscore that while llms have made significant strides in various natural language understanding tasks, they still have some distance to cover in order to match the supervised sotas in affect recognition tasks. adding emotion definitions explaining the eliciting conditions in tods to the prompt and fine-tuning llms for instruction-following could narrow the performance gap from supervised sotas. performing zero-shot affect recognition from utterances containing asr errors shows that llms are robust to such errors for emotion recognition but not for depression detection. therefore, when considering llms as a back-end module of a spoken dialogue system, it is crucial to exercise extra caution when processing dialogues laden with highly specific and nuanced affective content. our icl experiments exemplify that larger models would benefit more from an increased number of icl samples, highlighting the need to explore the optimal combination of the icl sample size in the prompt and the model size. for smaller llms, effectively utilising lengthy context remains as a challenge. through task-specific fine-tuning, we achieve performance levels close to sota on iemocap and emowoz, using only 50% of the training data, with llama-7b, alpaca-7b, and llama-3-8b. this highlights the great potential of fine-tuning llms for simpler tasks and integrating them as functional modules into dialogue systems. overall, llms have opened new avenues for affect recognition in conversations and building affectaware dialogue systems. despite the limited performance under zero-shot set-up, their robustness to asr errors, few-shot icl capabilities, and erc capabilities after fine-tuning offer exciting research opportunities for exploring affect recognition in conversations and building human-like conversational agents. we would also like to highlight the challenge and also opportunities towards handling long context and nuanced emotion cues in llms.