Is C hat GPT a Good Multi-Party Conversation Solver? (2023)

Abstract: Large Language Models (LLMs) have emerged as influential instruments within the realm of natural language processing; nevertheless, their capacity to handle multi-party conversations (MPCs) – a scenario marked by the presence of multiple interlocutors involved in intricate information exchanges – remains uncharted. In this paper, we delve into the potential of generative LLMs such as ChatGPT and GPT-4 within the context of MPCs. An empirical analysis is conducted to assess the zero-shot learning capabilities of ChatGPT and GPT-4 by subjecting them to evaluation across three MPC datasets that encompass five representative tasks. The findings reveal that ChatGPT’s performance on a number of evaluated MPC tasks leaves much to be desired, whilst GPT-4’s results portend a promising future. Additionally, we endeavor to bolster performance through the incorporation of MPC structures, encompassing both speaker and addressee architecture. This study provides an exhaustive evaluation and analysis of applying generative LLMs to MPCs, casting a light upon the conception and creation of increasingly effective and robust MPC agents. Concurrently, this work underscores the challenges implicit in the utilization of LLMs for MPCs, such as deciphering graphical information flows and generating stylistically consistent responses.

## introduction large language models (llms), including notable instances such as chatgpt @xcite and @xcite , are ushering in a new era in the field of natural language processing (nlp), showcasing remarkable zero-shot and fewshot generalization capabilities. the methods of pretraining language models on extensive text corpora @xcite , followed by alignment fine-tuning to ensure adherence to human instructions @xcite , has significantly amplified their proficiency in language comprehension, generation, interaction, and reasoning. the impressively high performance exhibited by these models has been extensively documented in related works @xcite @xcite . multi-party conversations (mpcs) represent a prevalent and natural facet of human communication. in these exchanges, more than two participants engage in interactive discourse on a variety of topics. such a dynamic introduces new challenges and opportunities for dialogue systems. here, the key requirement isn't merely generating coherent and relevant utterances, but also making strategic determinations about when to intervene and whom to address in the conversation. we queried chatgpt about its potential strategies for addressing the inherent challenges of mpcs with "can you solve multi-party conversation tasks?". chatgpt's response was as follows: "i do not have built-in mechanisms to keep track of individual participants in a conversation. therefore, it's important to explicitly mention the name or identifier of the participant you are addressing when providing instructions or asking questions.", which was intriguing and touched upon some of the critical aspects we are investigating in this paper. considerable efforts have been made to explore the capabilities of llms across a variety of nlp tasks @xcite . despite these advancements, the effectiveness of llms in handling mpcs remains largely underexplored. in this study, we scrutinize the potential of llms such as chatgpt and gpt-4 in managing mpcs by implementing five distinct tasks including emotion detection, addressee recognition, speaker identification, response selection, and response generation, across three different mpc datasets. our experiments reveal that both chatgpt and gpt-4 can achieve performance on par with supervised methods when evaluated on the emorynlp and meld datasets. however, the performance of these models on the more complex ubuntu irc dataset remains less than satisfactory. to address this, we introduce a strategy known as mpc structure incorporation, which weaves the speaker and addressee structure information of mpcs into the llms. this addition leads to a significant performance boost of chatgpt and gpt-4 across all three datasets, underpinning the value of this approach for improving llm effectiveness in mpc scenarios. in summary, our contributions in this paper are three-fold: 1) an exploratory study to examine the performance of chatgpt and gpt-4 in handling mpcs within a zero-shot context is carried out. this is the first study of its kind to investigate how these llms perform in mpc scenarios. 2) an mpc structure incorporation approach is proposed, which enhances the performance of chatgpt and gpt-4 in managing mpcs. this strategy integrates the speaker and addressee structure information of mpcs into llms, leading to substantial performance improvements. 3) we delve into the potential of llms in handling mpcs and shed light on the challenges that need to be tackled in future research. this discussion forms the basis for continued investigations into improving llm effectiveness in mpc contexts. ## related work large language models recently, the development of large language models (llms) has made tremendous progress, as evidenced by models such as @xcite , palm @xcite , llama @xcite , chat-gpt (openai, 2022), and gpt-4 (openai, 2023). these llms exhibit emergent abilities, including in-context learning, mathematical reasoning, and commonsense reasoning @xcite . a recent line of work focuses on instruction learning, either via generating high-quality instructions data @xcite or by boosting llms with instructions @xcite . ## approach to explore an out-of-box mpc solver, our emphasis lies in the zero-shot setting. instruction for each task is shown in figure 1 . for each task, llms are first instructed with the prompt "you have been presented with a sequence of multi-party conversational turns, organized in chronological order.". then, llms are instructed to complete the task with task-specific prompts. to stabilize ## conclusion in this paper, we have explored the abilities of generative llms for mpcs, which have been largely underexplored. we empirically analyze the zero-shot learning ability of chatgpt and gpt-4 by evaluating them on three popular mpc datasets covering five representative tasks. on emorynlp and meld datasets, chatgpt and gpt4 achieve comparable performance to the supervised training models. however, chatgpt performs poorly on the evaluated ubuntu irc tasks, while gpt-4 shows promising results. however, there is still a large gap relative to the supervised training on si task of ubuntu irc. taking into account the structure of the mpc, it is evident that both chatgpt and gpt-4 exhibit enhanced performance across nearly all tasks when equipped with speaker information. however, in the context of addressee information, chatgpt's performance may decline if it is encumbered with extraneous data. conversely, gpt-4 aptly leverages this information to accomplish the task with heightened proficiency. devoting efforts towards efficaciously intertwining the graphical structure inherent in mpcs with llms through prompts or even supervised finetuning constitutes one of future work.