Optimal Summaries for Enabling a Smooth Handover in Chat-Oriented Dialogue (2022)

Abstract: In dialogue systems, one option for creating a better dialogue experience for the user is to have a human operator take over the dialogue when the system runs into trouble communicating with the user. In this type of handover situation (we call it intervention), it is useful for the operator to have access to the dialogue summary. However, it is not clear exactly what type of summary would be the most useful for a smooth handover. In this study, we investigated the optimal type of summary through experiments in which interlocutors were presented with various summary types during interventions in order to examine their effects. Our findings showed that the best summaries were an abstractive summary plus one utterance immediately before the handover and an extractive summary consisting of five utterances immediately before the handover. From the viewpoint of computational cost, we recommend that extractive summaries consisting of the last five utterances be used.

## introduction dialogue systems are widely utilized in chatbots and call centers to respond automatically to users @xcite . however, it is often difficult for such systems to deliver fully autonomous dialogue. to ensure a good dialogue experience, human operators sometimes need to intervene in a dialogue if communication difficulties arise. we call this process handover or intervention and define it as joining a dialogue in the middle to achieve the original objective of the dialogue. in this study, we investigate which type of summary should be presented to the human operator in an intervention for a smooth handover. specifically, we conducted a large-scale experiment focused on chat dialogues to investigate the most useful summary for handover among seven types of dialogue summaries consisting of abstractive, extractive, and keyword summaries. our findings showed that the best summaries were an abstractive summary plus one utterance immediately before the handover and an extractive summary consisting of five utterances immediately before the handover. from the viewpoint of computational cost, we recommend that extractive summaries consisting of the last five utterances be used. ## related work the handover in dialogues from systems to human operators has been researched extensively in the context of call routing. in call routing, the dialogue is transferred to an appropriate operator and the system hands over the dialogue @xcite . however, there has been little research on the actual type of information to be shown to an operator during call routing. various frameworks have been proposed in which a semi-autonomous dialogue system performs most of the dialogue and hands over to a human operator when necessary @xcite @xcite . however, it is not clear exactly what type of information or summary would be the most useful for a smooth handover. automatic summarization has long been studied @xcite , and various datasets have been released @xcite @xcite and are currently in use @xcite . recently, large-scale pre-trained language models have been utilized to generate abstractive summaries @xcite using a large corpus of summaries @xcite @xcite . in this study, we examine what kind of summary is useful for a specific situation: handover. normal interlocutor intervention interlocutor interlocutor b interlocutor a interlocutor d interlocutor c we can't afford to skip shoveling snow off the roof. yes, i do. fan heaters are very economical. i see, a fan heater. ︙ ︙ dialogue history dialogue summary dialogue in existing corpus handover experiment the house may be damaged if you don't do it. do you use kerosene for your heater? ## approach to determine the type of summaries needed for a smooth handover in dialogue, our approach is to present a variety of summaries to the operator during the intervention process, examine the smoothness of the dialogue after the intervention, and quantify the effects of each summary type. to this end, it would be best to collect dialogues in a situation where one interlocutor changes to another in the middle of the dialogue. however, such experimentation would be extremely costly. therefore, in this paper, we simulate the dialogue handover. specifically, instead of performing the handover in real-time, we present the dialogue history of an existing dialogue to one of the interlocutors and a summary of that dialogue to the other interlocutor and have them continue the dialogue. we call this experiment a handover experiment. figure 1 shows an overview of the handover experiment. as a dialogue history, we prepared a chat dialogue between two interlocutors, a and b, from an existing corpus. the participants in the handover experiment are interlocutors c and d. interlocutor c is given the dialogue history and interlocutor d is given the dialogue summary created from the dialogue history, and they continue the dialogue on the basis of the information given to each. in this paper, we refer to interlocutor c as the "normal" interlocutor and interlocutor d as the "intervention" interlocutor. we investigated the usefulness of a summary by utilizing various summary types and analyzing their effects. the questionnaire responses of interlocutors and the number of dialogue breakdowns @xcite (a state in which dialogue cannot be continued smoothly) after the handover were used for the quantification of the effects. to cover typical summaries, we focused on the following summary types. ## analysis in this section, we investigate why abslast1 and extlast5 received the highest scores in the questionnaire. specifically, we first identified utterances in which dialogue breakdown occurred and classified them according to an existing taxonomy of errors in chat-oriented dialogue systems @xcite , and then clarified the types of errors most common for each summary type. the specific procedures are described below. ## conclusion in this study, we conducted a large-scale experiment to determine which summaries are the most useful for handing over a chat dialogue from seven types of dialogue summary consisting of abstractive, extractive, and keyword summaries. our findings showed that the best summaries were an abstractive summary plus one utterance immediately before the handover and an extractive summary consisting of five utterances immediately before the handover. from the viewpoint of computational cost, summaries that do not require learning, such as keyword summary and extractive summary, are useful. considering the results of the questionnaire and the analysis of dialogue breakdowns, we conclude that presenting the extractive summary consisting of the last five utterances is currently the most useful. as future work, it will be necessary to perform experiments to verify the effects of summary lengths. we also want to perform similar experiments with automatically generated summaries so that we can grasp the utility of abstractive summaries in actual handover situations. in addition, we would like to verify the actual usefulness of the summaries by conducting real-time handover experiments. although we targeted chat dialogues in this study, useful summaries for dialogues other than chat, such as task-oriented dialogue, should also be investigated.