A Large-Scale Corpus of E -mail Conversations with Standard and Two-Level Dialogue Act Annotations (2020)

Abstract: We present a large-scale corpus of e-mail conversations with domain-agnostic and two-level dialogue act (DA) annotations towards the goal of a better understanding of asynchronous conversations. We annotate over 6,000 messages and 35,000 sentences from more than 2,000 threads. For a domain-independent and application-independent DA annotations, we choose ISO standard 24617-2 as the annotation scheme. To assess the difficulty of DA recognition on our corpus, we evaluate several models, including a pre-trained contextual representation model, as our baselines. The experimental results show that BERT outperforms other neural network models, including previous state-of-the-art models, but falls short of a human performance. We also demonstrate that DA tags of two-level granularity enable a DA recognition model to learn efficiently by using multi-task learning. An evaluation of a model trained on our corpus against other domains of asynchronous conversation reveals the domain independence of our DA annotations.

## introduction the recent growth of textual communication media such as e-mails and online forums has led to the great demand for techniques enabling the automatic analysis of conversational structures from texts for information retrieval and intelligent assistance. dialogue acts (das), which are also known as "speech acts" in some studies, are one of such conversational structures. the das of text in conversations are defined as having communicative functions, such as asking questions, requesting some information, and offering suggestions. analyzing the das in conversations helps many downstream applications, including summarization @xcite , question answering @xcite , and conversational agents @xcite . conversations fall into synchronous conversations (e.g., phone calls and meetings) and asynchronous conversations (e.g., e-mails and online forums). in a synchronous conversation all participants engage at the same time with all participants, whereas in an asynchronous conversation participants interact with each other at different times. as is well known, the asynchronous properties make the conversational flow of asynchronous conversations different from those of synchronous conversations @xcite . topics in asynchronous conversations are often interleaved and not contiguous sequence. this complexity in a conversational flow makes da recognition in asynchronous conversations a challenging task compared to synchronous conversations, particularly when the thread structure (reply relations) is missing. the existing corpora for da recognition in asynchronous conversations have some of the shortcomings: (i) scale: whereas large-scale corpora such as the meeting recorder dialog act (mrda) @xcite and the switchboard discourse annotation and markup system of labeling (swbd) @xcite are available in synchronous conversation, most of the available corpora in asynchronous conversations are limited to a few thousands messages or sentences, and are insufficient to train more expressive models of da recognition. (ii) annotating scheme: annotation schemes used in the existing corpora are designed for a particular purpose or a particular application domain. for instance, @xcite focuses on technical help forums and defines domain-specific da tags such as res-olution, which represents a post confirming that an answer operates based on its implementtation. (iii) annotation granularity: many of the existing corpora in asynchronous conversations are annotated with da tags at only either sentence-level or message-level @xcite . however, the level of da tags required for a conversational analysis depends on its applications. for instance, sentencelevel das can help with a summarization of a message, and message-level das are beneficial for mining question-answer pairs from a community question-answering site. therefore, multi-level annotations of das allow for wider range of applications. moreover, multi-level annotations can provide a model of da recognition to learn efficiently by multi task learning. to fulfil all the requirements, we present a large-scale corpus of e-mail conversations with domainagnostic and two-level da annotations. as our source of asynchronous conversations, we use the enron e-mail dataset @xcite ) because e-mail is one of the most used communication media and the number of publicly available raw texts from e-mails is larger than that of other datasets. our corpus consists of over 2,000 threads, 6,000 messages, and 35,000 sentences. to conduct domain-and application-independent annotations, we adopt the iso standard 24617-2 @xcite as our da annotation scheme. the standard is designed for application-and domain-agnostic tags of das. as a segment unit of a da, we adopt both a sentence and a message. to assess the difficulty of da recognition on our corpus, we evaluate several models, including a pretrained contextual representation model, as our baselines. we also evaluate a human performance on our corpus for a model comparison. in addition, we demonstrate that da tags of two-level granularity enable a da recognition model to learn efficiently by applying multi-task learning. to evaluate the domain independence of our da annotations, we investigate the domain adaptation, wherein a model is trained on our corpus as a source domain and is applied to other corpora as the target domains. the contributions of this paper are as follows: ## discussions in this section, we further analyze a da recognition model to provide insight into our corpus. as a target model of the analysis, we choose the bert model owing to its performance. we use only one of the ten-fold data for the following analysis owing to the computational efficiency. ## conclusion aiming to facilitate the development of da recognition systems for asynchronous conversations, we developed a large-scale corpus of e-mail conversations with domain-agnostic and two-level da annotations. a comparison with human and sophisticated neural models demonstrated that there is still plenty of room for advancement in modeling of recognition da. an evaluation on the domain adaptation shows that training on our corpus contributes to a generalization of the models. we also showed that multi-task learning with two-level tags substantially boosts the performance. because an e-mail conversation has a hierarchical structure where words form a sentence, sentences form a message, and messages form a thread, we are planning to explore an efficient end-to-end model that leverages the hierarchical structure. in this paper, we focused annotating communicative functions of das. we will extend annotations to other important components of das such as rhetorical relations.