CEREC : A Corpus for Entity Resolution in Email Conversations (2020)

Abstract: We present the first large scale corpus for entity resolution in email conversations (CEREC). The corpus consists of 6001 email threads from the Enron Email Corpus containing 36,448 email messages and 38,996 entity coreference chains. The annotation is carried out as a two-step process with minimal manual effort. Experiments are carried out for evaluating different features and performance of four baselines on the created corpus. For the task of mention identification and coreference resolution, a best performance of 54.1 F1 is reported, highlighting the room for improvement. An in-depth qualitative and quantitative error analysis is presented to understand the limitations of the baselines considered.

## introduction entity resolution is defined as linking referring spans of text that point to the same discourse entity by @xcite and muc @xcite shared tasks. the corpora used for this task primarily consist of text from news @xcite @xcite , web-logs and transcripted dialogs. this research focusses on the entity resolution task for email conversations. example 1 shows a sample email message and the corresponding entities. the boldfaced tokens represent entities and the numbers beside them represent coreference chain identifiers. an entity is defined as an object or a group of objects in the real world and a span of text referring to an entity is called a mention. when all mentions in a text which refer to the same real-world entity are linked together, they form a coreference chain. 2020 ) first studied entity resolution in email conversations using a small annotated corpus. following the same task definition, this paper builds on their work and makes the following key contributions: the paper is organized as follows: section 2 reviews related work done on email processing and corpora for emails and entity resolution. section 3 describes the corpus creation process and reports statistics on the created corpus. it also explores the addition of features and experiments to evaluate the same. section 4 presents the baseline models, experiments carried out, and the results obtained. error analysis of the results is covered in section 5, followed by a discussion on the problem of missing context in section 6. section 7 concludes the work done in this paper. ## related work email processing has been an active research topic with the earlier works focusing on email classification @xcite @xcite @xcite . this was later followed by work on intent classification @xcite , searching @xcite , clustering @xcite , and summarization @xcite @xcite @xcite @xcite . one of the earliest works highlighting the challenges of the thread-like nature of email conversations was carried out by @xcite . @xcite proposed the creation of extended contribution trees to better understand the conversation structure of email threads. the impact of coreference resolution on conversations in a threaded format was first studied by @xcite using a corpus of blogs and commented news for opinion mining. although the impact was negative, it was attributed to the poor performance of the coreference system. coreference resolution for email conversations is an unexplored problem with, to the best of our knowledge, the only work done by @xcite . numerous corpora have been used for email processing over time. university emails @xcite , email users survey @xcite , private emails @xcite , simulated emails @xcite , and email archives @xcite are few of the initial sources for email corpora. the enron email corpus @xcite was the first large public corpus containing emails of 150 employees of the enron corporation. similarly, the avocado research email collection (douglas oard, 2015) consists of emails from 282 accounts of a now-defunct it company. the task of coreference resolution, specifically entity resolution, has received attention in the natural language research community since the 1960s with noun-phrase and pronomial resolution being the early forms of the task. although multiple corpora released over the years contain a small fraction of telephonic speech text, only a few corpora have focused on the study of the task in a purely conversational setting. character identification corpus @xcite was the first corpus to focus on the entity-linking task in this setting. it was constructed using tv show transcripts with annotations for speakers in a multi-party conversation. aktas ¸et. al (2018) used a twitter corpus to study the performance of stanford statistical coreference system @xcite . they evaluated a corpus with 185 threads containing 278 coreference chains and reported a mediocre performance by the model. ## error analysis this section presents error analysis performed on the predictions obtained by the baselines on a subset of 15 email threads selected randomly from sc. the selected 15 email threads contain a total of 282 coreference chains with 1261 mentions. to gain an in-depth understanding of the errors, human evaluation is performed. errors are broadly divided into four categories. these are similar to the categories used by 7 shows the distribution of errors into these categories for each of the baselines. ## conclusion this paper presents cerec, the first large annotated corpus for the entity resolution in email conversations task. the corpus consists of 6001 email threads with 38,996 coreference chains. the two steps in the construction of the corpus along with the results of the experiments involved and statistics of the resulting corpus are explained. the construction process is carried out with minimal human intervention. we also evaluate the addition of features specific to text in a conversational thread-like setting. two rulebased and two deep learning baselines are used for evaluation of the corpus. qualitative and quantitative error analysis is presented on the predictions obtained using all baselines highlighting the avenues for improvement. future work will consist of evaluating probable solutions for the entity resolution task. we also plan to conduct additional experiments to understand the effect of features presented in this paper using a larger corpus.