Interview: Large-scale Modeling of Media Dialog with Discourse Patterns and Knowledge Grounding (2020)

Abstract: In this work, we perform the first large-scale analysis of discourse in media dialog and its impact on generative modeling of dialog turns, with a focus on interrogative patterns and use of external knowledge. Discourse analysis can help us understand modes of persuasion, entertainment, and information elicitation in such settings, but has been limited to manual review of small corpora. We introduce **Interview**—a large-scale (105K conversations) media dialog dataset collected from news interview transcripts—which allows us to investigate such patterns at scale. We present a dialog model that leverages external knowledge as well as dialog acts via auxiliary losses and demonstrate that our model quantitatively and qualitatively outperforms strong discourse-agnostic baselines for dialog modeling—generating more specific and topical responses in interview-style conversations.

## introduction much of the news, information, and punditry the general public listens to and reads consists of media dialog-a category of open-domain conversations between an interviewer and interviewee centered on world events and situational context. a system for modeling media dialog from the perspective of one of these roles can help us better understand how media persuades and informs the public @xcite . thus, while recent work in dialog modeling has focused on goal-oriented @xcite , spontaneous @xcite , or synthetic open-domain chit-chat @xcite @xcite , we aim to analyze discourse patterns in media dialog and their impact on dialog modeling. media dialog differs linguistically and in purpose from unstructured, spontaneous conversation such as open-domain chitchat, and both the topical content and interlocutor intent are heavily influenced by the social, cultural, and temporal setting @xcite . the study of media dialog has traditionally focused on individual and manual review of small-scale (<200k word) news corpora @xcite , and we see an opportunity to scale some forms of discourse analysis to tens of thousands of such documents. in this work, we perform the first large-scale automatic analysis of structural components (response-type patterns) and question type categorization on media dialog, specifically for english news interviews. we show that predicting discourse features can improve generative dialog modeling performance, demonstrating the degree to which discourse structure impacts an interviewer's choice of response type and content. news interviews are also heavily situation-grounded and contextualized by past events and world knowledge. we explore methods to associate each conversation with a selection of world facts, and show that by modeling interviewers as knowledge-grounded speakers mediating a conversation we are able to generate relevant and specific utterances fitting their role. our main contributions in this work are: ## related work media dialog-specifically, the news interviewhas seen study primarily in the field of speech transcription, diarization, and speaker role modeling @xcite . these works have typically focused on techniques to annotate broadcast audio transcripts @xcite in order to cluster different news stories from a continuous broadcast stream @xcite . while @xcite and @xcite note that transition points between speaker roles (e.g. anchor and guest) can determine the high-level topical flow of a news conversation, we investigate the impact of discourse patterns on the semantics of specific utterances. such research is currently limited by a lack of accessible corpora for the study of media dialog at scale. the defense advanced research projects agency has undertaken efforts to collect and transcribe broadcast conversations @xcite . however, it proves difficult to adopt these datasets as widely available benchmarks on dialog modeling tasks, as they come with a substantial cost (@xmath01000 per annum per dataset). more recent efforts to amass such data have either focused on collecting large volumes of conversation fragments with noisy transcripts @xcite or human transcripts for a smaller set of long-form open-domain radio programs @xcite . we contribute an open-access largescale corpus of broadcast media dialog annotated with response types, demonstrating that these are useful for modeling interviewer utterances. we explore the application of discourse analysis @xcite on this large media dialog corpus in order to discover, confirm, and leverage discourse patterns regarding interrogative forms, speaker agency, and references to external knowledge. as noted by @xcite in their deep study of israeli news television, structure in media dialog (in contrast to spontaneous natural conversation) is uniquely determined by its speaker role dynamics. @xcite investigate the detection of one such dynamic: agreement/disagreement between speakers. @xcite classify discourse relations (e.g. comparative, temporal) between two turns of dialog, but do not study discourse structure. in this work we extend our analysis to other properties of interviewer utterances (e.g. subjectivity, polarity, dialog act patterns) @xcite in the context of generative dialog modeling. structured approaches for dialog modeling employ a simple concatenation of dialog history in a transformer-based architecture @xcite . we draw inspiration from @xcite who demonstrate the usefulness of a multi-task framework for speakerconditioned dialog modeling. @xcite propose a framework for jointly learning document retrieval and language modeling, and we propose a similar model to learn task-specific annotation of grounding documents. ## interview : a media dialog corpus we collect a new dataset of 105k multi-party interview transcripts for 7 programs on national public radio (npr) foot_2 over 20 years . these transcripts contain in total 3m turns comprising 7.5m sentences (127m words) from 184k speakers, of which 287 are interviewers. to investigate host-mediated media dialog, we curate a subset, interview 2p, with two roles: an interviewer and a guest, comprising 23k two-party conversations encompassing 455k turns, with 1.24m sentences and 21.7m words. in these two-party conversations, each speaker takes an average of nine turns per dialog. guests tend to speak longer on their turns, with 1.6x as many sentences spoken and 2x as many words per turn. meanwhile, hosts ask five times as many questions as guests, with 40% of their dialog turns containing questions. when askdataset structured # dialogs # turns # words radiotalk (2019) 5.98 m* 116 m 2.9 b tal (2020) 663 163,808 7.4 m interview 2p 23,714 454,739 21.7 m interview 105,848 3,199,856 126.7 m ## interview discourse analysis we tackle three aspects of discourse analysis that can be scaled to interview: 1) dialog patterns that emerge through new interviews; 2) large scale annotation of interviewer question types (dialog acts); and 3) obtaining grounding documents that provide situational context for a news interview. we study these discourse features in context of english broadcast news interviews. ## modeling media dialog a model's ability to learn underlying discourse dynamics is reflected in its performance on downstream tasks. here, we assess how well our model learns from dialog structure and question-pattern metadata using utterance generation-a simple predictive task that relies on a holistic understanding of grounding knowledge and a dialog history. this serves as an initial measure of understanding of discourse patterns and grounding even if the exact dialog produced can vary. we treat knowledge-grounded response generation in the media dialog setting as a language modeling task: given a dialog history h and a grounding knowledge document k, we seek to predict the next utterance x by maximizing the likelihood p(x|h, k). the dialog history is composed of turns spoken by both the interviewer and interviewee where each utterance is provided with the role annotation. we only model interviewer (host) responses, which aim to moderate the conversation via questions, follow-ups, and acknowledgements. to understand the effect of dialog structure and question types in response modeling, we introduce two auxiliary losses to influence generationa multi-task setup that has seen success in goaloriented dialog generation @xcite . ## experiments in our experiments, we seek to answering the following: 1) does knowledge grounding help generate more topical host responses? 2) do our two auxiliary discourse losses improve dialog generation performance? 3) do human raters find responses generated by our model coherent and fluent? hyperparameter details are in appendix §a. metrics to measure the fidelity of generated responses, we compute bpe perplexity and bleu @xcite between generated and gold utterances. to assess topical accuracy, we calculate the overlap between noun-phrases and named entities in the generated and gold responses. we are also interested in measuring coherence with respect to the context (i.e., grounding documents and dialog history), calculated via the noun-phrase and named entity overlap between generated responses and context. furthermore, as news interviews are intended to inform audiences, interviewers must ask questions using specific vocabulary and construction. to assess this, we adopt the normalized inverse document frequency @xcite to measure vocabulary specificity via word rarity. finally since we focus on generating interrogative host responses, we also calculate the percentage of questions asked in the generated responses as a measure of model inquisitiveness. ## conclusion in this work, we perform the first large-scale analysis of discourse patterns in media dialog, using a new dataset of 23k annotated news interview transcripts: interview. our results mirror findings from linguistic studies of news interviews @xcite . we demonstrate that adding auxiliary tasks for discourse pattern and interrogative type prediction helps model such media dialog. we observe that responses depend heavily on external knowledge, and present a probabilistic framework for linking factual documents with a conversation. while we focus on discourse pattern analysis, interview also supports analysis of temporal patterns in interviewing, argumentation, and knowledge grounding in long conversations.