Annotating Coherence Relations for Studying Topic Transitions in Social Talk (2020)

Abstract: This study develops the strand of research on topic transitions in social talk which aims to gain a better understanding of interlocutors’ conversational goals. Lưu and Malamud (2020) proposed that one way to identify such transitions is to annotate coherence relations, and then to identify utterances potentially expressing new topics as those that fail to participate in these relations. This work validates and refines their suggested annotation methodology, focusing on annotating most prominent coherence relations in face-to-face social dialogue. The result is a publicly accessible gold standard corpus with efficient and reliable annotation, whose broad coverage provides a foundation for future steps of identifying and classifying new topic utterances.

## introduction in natural language interactions, speakers' conversational goals determine the course of conversation. conversational goals are therefore an essential component of any dialogue model that possesses a genuine capability of natural language understanding and reasoning. while this component is predefined or able to be inferred from visible linguistic content in the dialogue systems for task-oriented conversation, it is inadequately represented or missing in the current dialogue systems implemented for social talk. for instance, open-domain dialog systems that evolve from task-oriented dialog systems usually extend the space of information to be exchanged, e.g. in terms of intents and topics, and treat social talk as multi-domain task-oriented talk; while end-to-end trained neural chatbots focus more on the utterance generation task, and do not have any explicit representation of conversational goals. to gain better insight into conversational goals of the interlocutors in social talk, @xcite conducted a pilot annotation study of new-topic utterances (ntus), which begin a new topic not related to the content of prior discourse. while such utterances are legitimate in social talk, current models of dialogue would treat them as incoherent conversational moves because these models only focus on utterances within a topically coherent discourse segment. by identifying ntus and studying patterns of the disjunctive topic changes (dtcs) which are signaled by these utterances, @xcite was able to introduce new sequence-based social intents that are absent in traditional taxonomies of speech acts but clearly reflect the goal-directed aspect of social talk, and therefore advocate the enrichment of dialogue models for social talk. we build on @xcite to evaluate and refine their annotation methodology for identifying ntu candidates. here, we report on a full annotation project which features a publicly accessible dataset of face-to-face casual dialogues in american english, refined annotation guidelines, detailed analyses of the annotation process, and fully adjudicated annotation results which can serve as a test set for computational evaluation. this paper is organized as follows. section 2 provides an overview of related prior research. section 3 presents our work constructing a gold standard corpus of discourse relation annotation for the purpose of identifying ntus. section 4 reports quantitative and qualitative analyses of our annotation process and methodology, while section 5 concludes and presents a plan for future work. the annotation of coherence relations in @xcite is different from previous attempts in two main aspects. first, as its ultimate goal is to identify ntus, which do not participate in any coherence relations with prior discourse, the annotation aims at annotating coherence relations that cover as many utterances in the discourse as possible, rather than exhaustively annotating all available coherence relations of interest as in @xcite . second, it relies on the superset of all well-known and tested coherence relations, rather than a specific taxonomy which may require ad hoc additions along the annotation process, as in @xcite . it is noted that the pilot annotation in @xcite does not fully comply with their proposed methodology. instead of annotating targeted coherence relations from the ground up, they start with the disco-spice corpus's 1,273 annotated coherence relations defined in the early version of the penn discourse treebank (pdtb) 3.0 scheme @xcite . that optimizes the workload but does not provide an accurate view of the proposed annotation process because the goals of two annotation projects are different (as mentioned above). this work seeks to elaborate and enrich lưu and malamud (2020)'s methodology for marking coherence relations as a way to identify ntu candidates in social talk, making the following contributions: ## conclusion and future work in summary, we have constructed a gold-standard corpus of most salient discourse relations as a way to identify ntu candidates in conversation. our accelerated annotation methodology and the use of established relation taxonomies allowed for efficient and reliable annotation process with broad coverage. our next steps are, first, to compare the topic transitions inferred from our annotation and riou (2015a)'s work to evaluate her methodology; and second, to identify ntus and patterns of dtcs to enrich the classification of ntus in @xcite . we plan to use this new classification as a step towards developing a dialogue agent capable of true understanding of the flow of social talk.