Do LLM s suffer from Multi-Party Hangover? A Diagnostic Approach to Addressee Recognition and Response Selection in Conversations (2024)

Abstract: Assessing the performance of systems to classify Multi-Party Conversations (MPC) is challenging due to the interconnection between linguistic and structural characteristics of conversations. Conventional evaluation methods often overlook variances in model behavior across different levels of structural complexity on interaction graphs. In this work, we propose a methodological pipeline to investigate model performance across specific structural attributes of conversations. As a proof of concept we focus on Response Selection and Addressee Recognition tasks, to diagnose model weaknesses. To this end, we extract representative diagnostic subdatasets with a fixed number of users and a good structural variety from a large and open corpus of online MPCs. We further frame our work in terms of data minimization, avoiding the use of original usernames to preserve privacy, and propose alternatives to using original text messages. Results show that response selection relies more on the textual content of conversations, while addressee recognition requires capturing their structural dimension. Using an LLM in a zero-shot setting, we further highlight how sensitivity to prompt variations is task-dependent.

## introduction multi-party conversations (mpcs) are multi-turn discussions involving more than two participants @xcite , which are typical of online platforms such as reddit or twitter/x @xcite . being able to capture the content of such discussions, when multiple users are involved and the conversation is composed of several turns, is a challenging task because both textual and structural information need to be modeled. indeed, designing systems for mpc understanding is challenging not only because the textual dimension spans multiple turns, but also because we need to capture structural aspects, such each turn in a conversation includes a speaker, an addressee and a textual message. from the conversation, we extract the interaction graph to diagnose model capabilities by performing two tasks: addressee recognition and response selection. as who writes to whom. understanding how these two components should be integrated to classify mpc, and how effectively llms can contribute to this task, is still an open question. in this paper, we examine the ability of an llm to perform mpc classification tasks in a zero-shot setting as well as to capture relevant information from an existing conversation. specifically, we address the tasks of response selection and addressee recognition and we use llama2-13b-chat @xcite not only to classify the last turn of mpcs but also to summarise the previous conversation and to describe users, so that this information can be included in the prompts for zero-shot classification. our choice of these two classifica-tion tasks is based on the following key considerations: i. the tasks deal with two specific aspects that a model working on mpcs needs to address, i.e. response selection for linguistic aspects and addressee recognition for structural and non-linguistic aspects; ii. these tasks can be performed on any conversational corpus in any domain, without the need of manual annotation. this makes our framework widely applicable. in figure 1 we report a graphical representation of an mpc, the pieces of information we retrieve and the tasks we perform. understanding the effects of conversation summarisation and user descriptions is important because they could make processing more efficient, replacing multiple turns with a more concise text representation. furthermore, using summarisations and user descriptions instead of the original conversations would make data sharing easier and more privacy-preserving, addressing growing concerns about this issue @xcite . for instance, it would comply with data minimisation principles, as required by the european general data protection regulation. replacing original conversations with summaries and user descriptions would also make it nearly impossible to train generative models that imitate specific users @xcite . our research questions are therefore as follows: rq(1): how do llms perform in classification tasks involving mpcs in a zero-shot setting, using different input combinations to capture textual and structural information? rq(2): what is the model sensitivity to different prompt formulations when classifying mpcs? rq(3): how does structural complexity of the conversation affect classification performance? to address rq(1), we evaluate llama2-13b-chat on response selection and addressee recognition in a zero-shot scenario (section 3). these tasks capture different types of information: response selection relies on textual information to choose the next message in a conversation, while addressee recognition requires more structural awareness to infer speaker characteristics and conversation flow. for each conversation, we design input combinations of conversation transcripts, interaction transcripts, generated summaries, and generated user descriptions, with the latter two being generated by llama2-13b-chat (section 4). we address also rq(2) by designing prompts of different levels of verbosity for each combination and task. finally rq(3) is addressed by designing a diagnostic approach, where the two tasks are evaluated on mpcs with a different number of speakers and structural characteristics (section 5). this allows us to analyse the connection between task scores and structural characteristics of mpcs. the software to perform the experiments and the processed data are available on a dedicate github repository. foot_0 ## related work researchers have worked on mpc understanding tasks either by trying to model an entire conversation or by focusing on relations within the conversation @xcite . recent mpc understanding studies focus on response selection (rs) and addressee recognition (ar) tasks @xcite to compare different classification approaches. indeed, rs is strictly related to textual (linguistic) information, while ar focuses on interaction information, thus permitting to analyse the performance of classification models from two different angles. however, both tasks can ideally benefit from cross-information between linguistic and interaction cues. for both rs and ar, researchers have fine-tuned transformer-based models incorporating speaker information @xcite @xcite , used graph neural networks (gnns) for interaction modeling @xcite , or leveraged dialogue dependency parsing @xcite . recently, @xcite , explored zero-shot capabilities of @xcite and @xcite in mpcs, focusing only on the overall classification performance. indeed, there is a gap in the nlp literature concerning the evaluation of mpc systems based on structural aspects. past research has focused on textual information, for instance by using candidate rankings @xcite or just looking at conversation length and number of users @xcite . @xcite provided a first exploration of the role of conversation structure in stance detection, show-ing that it benefits classification only when large training data are available. summarizing the dynamics and trajectories of mpcs, where a model's understanding of the conversation structure and interactions is critical, has been recently addressed by @xcite . the authors also evaluate the summaries of conversation dynamics with a classification task (i.e., forecasting the future derailment of the conversation as in @xcite . @xcite point out that conventional summaries heavily focus on the textual content and what individual speakers say while ignoring the interactions between speakers and the conversation flow. the work that is most similar to our contribution is @xcite , since they also use a generative model in a zero-shot setting to address rs and ar. the main difference is that, instead of focusing only on generic accuracy scores, we propose a diagnostic approach for evaluating models for mpc understanding. we use response selection and addressee recognition as proxy tasks and focus particularly on the contribution of structural information, by i. creating diagnostic datasets, each with a fixed number of users, and ii. putting in relation the classification performance to specific network metrics (i.e., degree centrality and average outgoing weight of the speaker node). ## tasks our experiments revolve around two tasks that do not need a manual annotation as long as the used mpc data include speaker, addressee and related utterances. response selection (rs) is the task of choosing the text of the next message given a conversation c, the id of the speaker of the next message and a set of candidate responses. in our experiments, we cast response selection as a binary classification task, since the system has to choose between two possible candidates (similar to the r2@1 task in @xcite . addressee recognition (ar) is the task of predicting the addressee of the next message given a conversation c, the id of the speaker of the next message and a set of candidate addressees. the set of candidate addressees include all speakers involved so far in the conversation c plus a "dummy" option, which introduces a user unseen in the conversation to check whether the classifier choice is fully random. in both cases, the next speaker is given, and the classifier has to select what will be the content of the message (rs) or who will be the addressee (ar). ## mpc classification workflow in this section we describe the classification workflow implemented to perform response selection (rs) and addressee recognition (ar). the workflow is shared between the two tasks. ## diagnostic approach to address the three research questions introduced in section 1, we aim to develop a diagnostic approach that isolates specific phenomena and minimizes confounding factors. a key aspect under analysis is the interplay between interaction structure in the conversation and classification performance. we identify two metrics to capture conversation complexity in terms of interaction graph and we also create subcorpora from a large conversation corpus, called diagnostic datasets, each with specific characteristics to test in relation to classification performance. in figure 4 we report a schematic representation of our evaluation pipeline and the components involved. ## experiments given the four types of input presented in section 4.1, we design five input combinations to test in our prompts for both tasks: i. only the conversation transcript (conv); ii. the conversation transcript and the interaction transcript (conv+struct); iii. the interaction transcript and the conversation summary (struct+summ); iv. the interaction transcript and the user descriptions (struct+desc); v. the interaction transcript, the conversation summary and the user descriptions (struct+summ+desc). for the ar task, we test a sixth combination, vi. struct, which corresponds only to the interaction transcript. struct is not relevant for rs since it does not include any linguistic information. all combinations and prompt schemes are tested across the 4 diagnostic datasets. ## macro results and structural evaluation macro-results on the best run. in figure 5 , we present the macro accuracy for both tasks across all 4 diagnostic datasets. the columns show the highest accuracy achieved among the 3 prompt schemes with varying level of verbosity. in the ar task, the number of classes (i.e. addressees) on each ubuntu subset changes, ranging from four (ubuntu3) to seven (ubuntu6), since the set of possible addressees includes the speakers involved in each conversation, plus the dummy label (see section 3). for this reason, results across different ubuntu subsets on ar should not be compared, and the lowest accuracy is achieved on ubuntu 6, being its classification based on seven possible addressees. in ar, the conv+struct and struct combinations consistently perform best across all datasets. instead, the conv combination, serving as our 'text-only' baseline consistently shows the worst performance. if we consider replacing the original conversation with a summary (summ) or user description (desc), we observe that the former outperforms the other on all datasets, although adding the original conversation to the structure (conv+struct) still ourperforms both alternatives. in the rs task, the conv and conv+struct combinations consistently perform the best across all datasets. among the combinations with summary and/or user description, struct+summ+desc performs the best (in ubuntu3/4) or extremely close to struct+summ, which is the best in ubuntu5/6. finally, the struct+desc input combination yields the lowest classification performance for both tasks on all datasets, except for ar on ubuntu3. this analysis shows that the interaction transcript (i.e., the structural information) is fundamental for achieving the best result in ar. on the other hand, the conversation transcript is fundamental for achieving the best results in rs, which in fact is a more text-oriented task, based on information mainly available in the conversation itself. however, using summaries of conversation may be a viable alternative, achieving results closer to the best input combinations in the setting with more users (i.e. ubuntu6) for both tasks. prompt sensitivity. in this section we compare the highest accuracy and average accuracy among the 3 prompt schemes, for each input combination. given b the accuracy of the best prompt scheme and a the average among the 3 prompt schemes, we define as relative gap the relative worsening from the best to the average: gap @xmath0. a larger relative gap suggests greater sensitivity of the model to the prompts used, which leads to fluctuations in the classification results. in overall, we observe that for ar, where structural information is more relevant, classification perfor-mance tends to vary more with different prompt verbosity compared to rs, where linguistic information has a higher weight. if we analyse what is the effect of prompt verbosity on classification performance, we observe that in the majority of settings and configurations, the verbose version of the prompt is the best performing one for ar. for rs, instead, there is no evidence of benefit from using more or less verbose options. ## discussion our comparative evaluation shows three main findings. ## conclusions in this study, we evaluate the zero-shot performance of an llm (i.e., llama2-13b-chat) on two tasks based on multi party conversations, namely response selection and addressee recognition. our goal is to provide an in-depth analysis of different experimental settings tested for the two tasks, which include three different prompt types and six configurations to model the conversation text and its structure. our analysis is performed on four diagnostic datasets with a fixed number of users. for each of them, we compute two network metrics, i.e. degree centrality and average outgoing weight, to analyse how structural complexity interacts with classification performance. we devote particular attention to evaluating how strategies to replace the original conversation text could be effectively used in the prompts. this is very relevant to ensure a safe use of mpc corpora: if the same classification performance could be achieved by removing the original conversation, data resharing would not imply the risk of making personal or sensitive data available. furthermore, malicious use of mpcs, for example using them to train models with fake personas, would not be possible. although promising, this research direction has not achieved fully satisfactory results. the goal of our work is not much to yield the best possible classification accuracy on ar and rs, but rather to provide an in-depth analysis of the possible dimensions contributing to the classifier performance on the two tasks. we believe that the interplay between textual and structural information in mpcs should be better analysed in current evaluations, merging contributions from nlp and the network science community.