TIAGE : A Benchmark for Topic-Shift Aware Dialog Modeling (2021)

Abstract: Human conversations naturally evolve around different topics and fluently move between them. In research on dialog systems, the ability to actively and smoothly transition to new topics is often ignored. In this paper we introduce TIAGE, a new topic-shift aware dialog benchmark constructed utilizing human annotations on topic shifts. Based on TIAGE, we introduce three tasks to investigate different scenarios of topic-shift modeling in dialog settings: topic-shift detection, topic-shift triggered response generation and topic-aware dialog generation. Experiments on these tasks show that the topic-shift signals in TIAGE are useful for topic-shift response generation. On the other hand, dialog systems still struggle to decide when to change topic. This indicates further research is needed in topic-shift aware dialog modeling.

## introduction existing dialog models @xcite @xcite have been reported to perform well in generating on-topic utterances in dialog scenarios. however, those models still struggle to proactively generate appropriate topic-shift utterances in conversations @xcite . it is beneficial for dialog systems to be able to shift topics fluently. as shown in figure 1 , topicshift behaviors are commonly observed in human conversations @xcite . fluent topic shifts therefore are crucial for dialog models to be able to model or mimic human conversational patterns. proactively using topic shifts can help chatbots guide conversations to a pre-defined target @xcite . furthermore, switching topics allows chatbots to maintain engaging conversations with users. without the ability to actively 1 code and data available at: https://github.com/huiyuanx ie/tiage. shift topics away from tired topics, chatbots risk generating dull responses or repeating themselves regarding a specific topic. to facilitate research on topic-shift dialog modeling, we curate a topic-shift aware dialog dataset (tiage) by augmenting the personachat dataset @xcite with topic-shift annotations. to the best of our knowledge, tiage is the first dataset that focuses on topic-shift behaviors in open-domain dialog data. tiage contains a human annotated dataset with 7,861 gold standard topic-shift annotations, and a weak supervision dataset to adapt pretrained nlg systems to personachat-style data. the inter-annotator agreement for topic-shift annotations in tiage is 0.479. with tiage, we propose three tasks to study topic-shift behaviors: topic-shift detection, topicshift triggered response generation and topic-aware dialog generation. the topic-shift detection task asks models to detect whether the ongoing topic has shifted or should shift. the other two tasks focus on modeling topic-shift behaviors in response generation. specifically, the topic-shift triggered response generator receives a fixed topic-shift signal to generate topic-shift responses, whilst the topic-aware dialog generation task requires dialog systems to predict the topic-shift trigger by themselves. our experiments reveal that the topic-shift signals in tiage indeed improve dialog systems' ability to generate topic-shift responses. however, it is difficult for dialog models to predict when it is appropriate to change topics. these observations highlight the need for better modeling of topic shifts in dialog generation. we hope our benchmark can motivate further research on topic-shift aware dialog modeling. ## related work existing work in dialog systems falls into two broad categories. task-oriented dialog systems @xcite help users complete tasks in specific domains. opendomain dialog systems @xcite allow agents to have open-ended conversations with users. most existing dialog models @xcite @xcite emphasize end-to-end response generation, and do not explicitly address the topicshift problem in dialog generation. early work in topic detection and segmentation @xcite focused on identifying cue phrases (such as on a different note) or examining lexical cohesion to segment topical chunks. other work @xcite investigated topic detection and tracking (tdt) in a stream of broadcast news stories. more recent work @xcite has explored utilizing neural networks to address topic segmentation. although some of the existing work @xcite has investigated topic detection in dialog-style data, the generation aspect of topicshift modeling in dialog settings is still unclear. ## topic-shift aware dialog dataset in this section we introduce the rationale for our choice of data source, the human annotation process of topic-shift labelling in tiage and its data statistics. we also analyze the linguistic patterns of topic-shift utterances in tiage. rationale for our choice of data source. we construct tiage by augmenting the personachat dataset @xcite with topic-shift human annotations. we view personachat as a suitable dataset for topic-shift annotation for the following reasons: (1) the personachat data was collected online in a textual form by mimicking chit-chat scenarios, where natural shifts of topics are more likely to happen; (2) dialogs in this dataset contain more than 10 dialog turns, and longer dialog contexts tend to exhibit a conversational flow with more topics; and (3) despite the fact that some participants in personachat may have rushed into changing topics to quickly exchange their profile information, we observed that most of the participants still manage to change topics in a more natural and coherent way, making this dataset a favorable choice to study topic-shift behaviors. human annotation process. for the annotation pool, we have a total number of 25 human annotators. we randomly selected 500 dialogs from the original personachat dev/test datasets, resulting in 7,861 dialog turns to label. each dialog turn was randomly assigned to and independently labeled by 2 annotators. for each dialog turn, we asked annotators to indicate whether they think the conversational topic is changed at that turn. during the annotation process, all annotators were talked through the general aim of this annotation task and given the same annotation guidelines (see appendix a.1 for details). since topic is co-constructed, it is rather limiting to analyze a turn for itself when trying to identify topic transitions. to facilitate the recognition of slowly transitioned topics, we encouraged the annotators to take into account both the previous two turns and the following two turns of the target dialog turn to make a decision. this helped decision making for cases where topics are slowly developed and transitioned. after annotating, we obtained a dialog dataset with gold standard topic-shift labels for 7,861 dialog turns. the cohen's kappa score for all annotations is 0.479 foot_0 . annotated examples of tiage dialogs are shown in appendix a.2. dataset statistics. as shown in ## tasks of topic-shift modeling along with dialog utterances, tiage also provides gold standard topic-shift labels for dialog turns. this enables us to model topic shifts in dialog scenarios. we first introduce two tasks: topicshift detection and topic-shift triggered response generation. they can be considered as intermediate steps of the topic-aware dialog generation task. ## evaluation results we report here the evaluation results for baseline systems on the above three tasks. topic-shift detection. we test topic-shift classifiers on the annotated anno test split. from ## conclusion and future work we construct the tiage dataset with human annotated topic-shift labels on the basis of the per-sonachat dataset. based on tiage, we introduce three tasks: topic-shift detection, topic-shift triggered response generation and topic-aware dialog generation. empirical results show that topic-shift labels in tiage are useful for topic-shift response generation. however, it remains a challenging task for dialog models to predict good-quality topicshift signals based on dialog context only. further research is needed on selecting appropriate topics to shift to among multiple references. natural topic shifts can be both a precaution against, and a remedy to, dull and repetitive response generation in real-world dialog applications. tiage with its topic-shift annotations can help direct future investigation on the incorporation of topic-shift tactics in dialog models, which allows more effective control over topic-shift aware dialog generation.