E mail S um: Abstractive Email Thread Summarization (2021)

Abstract: Recent years have brought about an interest in the challenging task of summarizing conversation threads (meetings, online discussions, etc.). Such summaries help analysis of the long text to quickly catch up with the decisions made and thus improve our work or communication efficiency. To spur research in thread summarization, we have developed an abstractive Email Thread Summarization (EmailSum) dataset, which contains human-annotated short (<30 words) and long (<100 words) summaries of 2,549 email threads (each containing 3 to 10 emails) over a wide variety of topics. We perform a comprehensive empirical study to explore different summarization techniques (including extractive and abstractive methods, single-document and hierarchical models, as well as transfer and semisupervised learning) and conduct human evaluations on both short and long summary generation tasks. Our results reveal the key challenges of current abstractive summarization models in this task, such as understanding the sender’s intent and identifying the roles of sender and receiver. Furthermore, we find that widely used automatic evaluation metrics (ROUGE, BERTScore) are weakly correlated with human judgments on this email thread summarization task. Hence, we emphasize the importance of human evaluation and the development of better metrics by the community.

## introduction as one of the major natural language generation tasks, automatic summarization has been studied for decades. most research efforts were focused on single-document summarization tasks, e.g., news document summarization @xcite . however, living in an information era, we are facing with diverse content email thread: subject: lunch this week susan: all, regarding our lunch this week to celebrate the one year anniversaries for michelle & david, and mark's birthday, i have a request to make it wednesday instead of tuesday. does anyone have an objection to this? susan david: i have another lunch engagement wed, but i will skip it if everyone else wants to move our lunch. david tamra: susan, wednesday works out better for me as well. i have a doctor's appointment tomorrow during lunch. tamra ## emailsum dataset to collect email thread summarization data, we first need to obtain unlabelled email threads. we resort to existing email collections: enron @xcite , w3c @xcite , and avocado @xcite . however, none of them provides explicit thread structure. therefore, in this section, we will introduce our email thread preprocessing and summary collection procedures. ## models the summarization models we explore in this work take the email thread as input and generate the summary as output. we experiment on email-sum short and emailsum long tasks separately. ## conclusion in this work, we propose an abstractive email thread summarization dataset, emailsum, that contains 2,549 email threads with human-written short and long summaries. we explore different summarization paradigms and find that taking the email thread as a single document and finetuning t5 @xcite sets up a good baseline. transferring from other summarization datasets barely improves it. using hierarchical structure also only marginally improves the performance. semi-supervised learning by using unlabelled email threads improves automatic metrics (rouge) but still loses to the baseline in human evaluation. finally, our human evaluation reveals that the model fails to understand the sender's main intention and the roles of different speakers. automatic metrics are poorly correlated with human judgment, which emphasizes the importance of human evaluation and designing new metrics for this task in the future.