Dialogue-Based Relation Extraction (2020)

Abstract: We present the first human-annotated dialogue-based relation extraction (RE) dataset DialogRE, aiming to support the prediction of relation(s) between two arguments that appear in a dialogue. We further offer DialogRE as a platform for studying cross-sentence RE as most facts span multiple sentences. We argue that speaker-related information plays a critical role in the proposed task, based on an analysis of similarities and differences between dialogue-based and traditional RE tasks. Considering the timeliness of communication in a dialogue, we design a new metric to evaluate the performance of RE methods in a conversational setting and investigate the performance of several representative RE methods on DialogRE. Experimental results demonstrate that a speaker-aware extension on the best-performing model leads to gains in both the standard and conversational evaluation settings. DialogRE is available at https://dataset.org/dialogre/ .

## introduction cross-sentence relation extraction, which aims to identify relations between two arguments that are not mentioned in the same sentence or relations that cannot be supported by any single sentence, is an essential step in building knowledge bases from large-scale corpora automatically @xcite @xcite . it has yet to receive extensive study in natural language processing, however. in particular, although dialogues readily exhibit cross-sentence relations, most existing relation extraction tasks focus on texts from formal genres such as professionally written and edited news reports or well-edited websites @xcite @xcite † equal contribution. s1: hey pheebs. s2: hey! s1: any sign of your brother? s2: no, but he's always late. s1: i thought you only met him once? s2: yeah, i did. i think it sounds y'know big sistery, y'know, 'frank's always late.' s1: well relax, he'll be here. argument pair trigger relation type r1 (frank, s2) brother per:siblings r2 (s2, frank) brother per:siblings r3 (s2, pheebs) none per:alternate names r4 (s1, pheebs) none unanswerable in this paper, we take an initial step towards studying relation extraction in dialogues by constructing the first human-annotated dialogue-based relation extraction dataset, dialogre. specifically, we annotate all occurrences of 36 possible relation types that exist between pairs of arguments in the 1,788 dialogues originating from the complete transcripts of friends, a corpus that has been widely employed in dialogue research in recent years @xcite @xcite @xcite . altogether, we annotate 10,168 relational triples. for each (subject, relation type, object) triple, we also annotate the minimal contiguous text span that most clearly expresses the relation; this may enable researchers to explore relation extraction methods that provide fine-grained explanations along with evidence sentences. for example, the bolded text span "brother" in our analysis of dialogre indicates that the supporting text for most (approximately 96.0%) an-notated relational triples includes content from multiple sentences, making the dataset ideal for studying cross-sentence relation extraction. this is perhaps because of the higher person pronoun frequency @xcite and lower information density @xcite in conversational texts than those in formal written texts. in addition, 65.9% of relational triples involve arguments that never appear in the same turn, suggesting that multi-turn information may play an important role in dialogue-based relation extraction. for example, to justify that "pheebs" is an alternate name of s2 in we next conduct a thorough investigation of the similarities and differences between dialoguebased and traditional relation extraction tasks by comparing dialogre and the slot filling dataset @xcite @xcite @xcite , and we argue that a relation extraction system should be aware of speakers in dialogues. in particular, most relational triples in dialogre (89.9%) signify either an attribute of a speaker or a relation between two speakers. the same phenomenon occurs in an existing knowledge base constructed by encyclopedia collaborators, relevant to the same dialogue corpus we use for annotation (section 3.2). unfortunately, most previous work directly applies existing relation extraction systems to dialogues without explicitly considering the speakers involved @xcite . moreover, traditional relation extraction methods typically output a set of relations only after they have read the entire document and are free to rely on the existence of multiple mentions of a relation throughout the text to confirm its existence. however, these methods may be insufficient for powering a number of practical real-time dialoguebased applications such as chatbots, which would likely require recognition of a relation at its first mention in an interactive conversation. to encourage automated methods to identify the relationship between two arguments in a dialogue as early as possible, we further design a new performance evaluation metric for the conversational setting, which can be used as a supplement to the standard f1 measure (section 4.1). in addition to dataset creation and metric design, we adapt a number of strong, representative learning-based relation extraction methods @xcite @xcite and evaluate them on dialo-gre to establish baseline results on the dataset going forward. we also extend the best-performing method @xcite among them by letting the model be aware of the existence of arguments that are dialogue participants (section 4.2). experiments on dialogre demonstrate that this simple extension nevertheless yields substantial gains on both standard and conversational re evaluation metrics, supporting our assumption regarding the critical role of tracking speakers in dialogue-based relation extraction (section 5). the primary contributions of this work are as follows: (i) we construct the first human-annotated dialogue-based relation extraction dataset and thoroughly investigate the similarities and differences between dialogue-based and traditional relation extraction tasks, (ii) we design a new conversational evaluation metric that features the timeliness aspect of interactive communications in dialogue, and (iii) we establish a set of baseline relation extraction results on dialogre using standard learning-based techniques and further demonstrate the importance of explicit recognition of speaker arguments in dialogue-based relation extraction. ## data construction we use the transcripts of all ten seasons (263 episodes in total) of an american television situation comedy friends, covering a range of topics. we remove all content (usually in parentheses or square brackets) that describes non-verbal information such as behaviors and scene information. ## conclusions we present the first human-annotated dialoguebased re dataset dialogre. we also design a new metric to evaluate the performance of re methods in a conversational setting and argue that tracking speakers play a critical role in this task. we investigate the performance of several re methods, and experimental results demonstrate that a speaker-aware extension on the best-performing model leads to substantial gains in both the standard and conversational settings. in the future, we are interested in investigating the generality of our defined schema for other comedies and different conversational registers, identifying the temporal intervals when relations are valid @xcite in a dialogue, and joint dialogue-based information extraction as well as its potential combinations with multimodal signals from images, speech, and videos.