Topic Shift Detection for Mixed Initiative Response (2021)

Abstract: Topic diversion occurs frequently with engaging open-domain dialogue systems like virtual assistants. The balance between staying on topic and rectifying the topic drift is important for a good collaborative system. In this paper, we present a model which uses a fine-tuned XLNet-base to classify the utterances pertaining to the major topic of conversation and those which are not, with a precision of 84%. We propose a preliminary study, classifying utterances into major, minor and off-topics, which further extends into a system initiative for diversion rectification. A case study was conducted where a system initiative is emulated as a response to the user going off-topic, mimicking a common occurrence of mixed initiative present in natural human-human conversation. This task of classifying utterances into those which belong to the major theme or not, would also help us in identification of relevant sentences for tasks like dialogue summarization and information extraction from conversations.

## introduction conversational systems have become a part and parcel of our everyday life and virtual assistants like amazon's alexa foot_0 , google home foot_1 or apple's siri foot_2 are soon becoming conventional household items @xcite . most of the conversational systems were built with the primary goal of accessing information, completing tasks, or executing transactions. however, recent conversational agents are transitioning towards a novel hybrid of both task-oriented and a non-task-oriented systems @xcite from the earlier models that resembled factual information systems @xcite . but with this transition, they are failing to engage in complex information seeking tasks and conversations where multiple turns tend to get involved @xcite . these new-age open-domain dialogue systems also suffer from a different kind of user behaviour called "anomalous state of knowledge" @xcite where the user has vague information requirements and is often unable to articulate it with enough precision. this leads to the user deviating from their original path and traversing into a sub-topic without their knowledge @xcite . thus, we need a context-dependent user guidance without presupposing a strict hierarchy of plans and task goals of the user. such a guidance, without topic information provided beforehand, is a difficult task to achieve in an open-domain system. in this work, we observe how a human-human open-domain conversation with an initial topic to begin with, handles topic drift and its rectification in a conversation. we work on the switchboard dataset @xcite and annotate 74 conversations with 'major', 'minor' and 'offtopic' tags (section 4). a key result of our finding was that most of the topic shift detection models [ @xcite , @xcite , @xcite ] have previously defined topic set to assign to utterances. but as we see in switchboard dataset, modeling such a pre-defined set is not a property of an open-domain non-task-oriented conversational system. we create a novel model which can, with a precision of 84%, predict the utterances that belong to the major topic and those which are deviating from the same, without a predetermined topic set. this is a major contribution as it can help in informational retrieval in conversational systems @xcite , dialogue summarization @xcite and in the case study that we explored viz. introducing a system initiative in a conversation. ## task definition mixed initiative (mi) is an important aspect for effectively solving multi-agent collaboration problems and is generally referred to as a flexible interaction strategy where each agent can contribute to a task that it is best at @xcite . here, we'll look into an example of topic shift in a conversation, which sheds light on this issue in a conversation that is common in our day-to-day lives. m t              a: hello, what are your hobbies? b: my hobbies, umm, i used to dance a lot in high school, what are yours? a: i used to paint, but these days i am just occupied with whatever my kids are occupied with at that moment.              ot                        b: ooh that's nice, how many kids do you have? a: i have two kids, one boy aged 6 and a daughter aged 3 what about you? b: yes, two twin girls aged 4. a: aww that's such a lovely age. b: ya it is, but they can also be a little handful at times. m i a: anyways, let's go back to the topic at hand, tell me more about your hobbies? the above example shows how the topic transitioned between the two users, from hobbies which was their major topic given by a prompt, to talking about their kids. we see from the marked area that they transitioned from the major topic (mt) to an off-topic (ot) and rectified the topic shift as well. this shift occurs abruptly, with stark difference in the semantic space between the two topics. such a topic diversion and rectification is a natural phenomenon in a human-human conversation. ## related work a good conversation is one which focuses on a balance between staying on topic and changing it in an interactive multi-turn conversation system @xcite . detection of what constitutes as ontopic can be viewed as segmentation of conversation into relevant and irrelevant of the conversation @xcite . earlier work in segmenting conversations into topics expected a high lexical cohesion within a topic segment @xcite . however, we see that they fail to have regard of sentence-level dependencies leading to fragmented segmentation @xcite . various supervised methods approached this task as a classifi-cation problem @xcite but annotations for them can be expensive and not scalable for large datasets. unsupervised methods on goaloriented conversations also have limited ability to learn from the dataset @xcite . modelling this problem into detection of global topic structure and local topic continuity @xcite results in a weakly supervised approach, using a hierarchical lstm, to analyse dialogue context and content. however, a major drawback in that method is that the topic sets are predefined and the utterances are bucketed into the same. in an unbounded natural conversation, specifying the topic set in advance is not a feasible task. our proposed topic segmentation would help us introduce a system initiative module by figuring out when to give refinement or guidance and how to best contribute in solving a user's problem @xcite , by detecting the major topic of the conversation and steering the user towards it in case of a diversion. ## annotation framework we use the human-transcribed conversations from the nxt-format switchboard corpus @xcite in our task. in this dataset, participants are given a topic prompt and were asked to converse with each other for around ten minutes. this dataset was chosen for annotation, amongst others, as some did not have enough turns to observe a topic shift [ @xcite , @xcite ] or had fixed topics of conversation @xcite , @xcite ] neither of which were favourable for us to model an off-topic shift detection for open-domain conversations. in switchboard, we observe the freedom with which the participants drift from the given topic prompt, leading to different off-topic threads in the conversation and several statements by the users to steer the conversation back to the original topic. to model this property, we annotated the dataset, into three labels -major, minor and off-topic tags. dialogues are inherently hierarchical in structure, but we see that human annotators cannot definitively agree on a hierarchical segmentation @xcite . thus we adopt a flat model of annotation where a strong shift from the original topic of conversation is annotated as off-topic and a subsidiary shift is labelled as minor topic. a conversational speech is not as structured as written text; it consists of overlaps of turns between the participants and interruptions. that is why each turn is divided into an utterance consisting of a single independent clause @xcite . this also helps us in narrowing down each utterance to have a single topic of discussion and thus a single tag to belong to. for our ease of annotation, we have considered incomplete sentence as complete sentences and annotated accordingly. we have also made a conscious decision to drop one word sentences. ## experiments and results prior to designing the topic classifier, we wanted to understand the characteristics of switchboard corpus and visualize the classes that we have defined in section 4. we plotted the t-sne embeddings(van der @xcite for the 3 classes in fig 1 @xcite . we observe that minor and off-topic classes are entangled and thus decided to merge these two classes into a rest class. the t-sne plot for the data with the merged class can be seen in fig 1 (right), and the classes are now less entangled. our task is now a binary classification task with the two classes being major and rest. this is further backed by the poor results obtained on the application of classification models to classify each classes individually, which we omit for brevity. ## case study the system response generated in this case study is a system initiative (si) given to a snippet of the switchboard corpus, prompting the user to go back to the major topic of the conversation, when it detects a topic shift from it. ## conclusion in this paper, we looked at generating a system initiative module in a conversational system that does not interrupt the user and also works towards achieving the common goal of the user. we present a dataset that helps in training an xlnet-based model to correctly detect a digression from the major topic of the conversation. we have also looked at an application of this model as a case study where we detect topic shift and generate a system initiative for the rectification of the same. a predictable limitation of our system lies in not detecting minor and off-topic individually. this categorisation would help in giving a leeway in case of a shift to a minor topic thread and a system rectification initiative in case of a shift to an off-topic thread .