F orum S um: A Multi-Speaker Conversation Summarization Dataset (2021)

Abstract: Abstractive summarization quality had large improvements since recent language pretraining techniques. However, currently there is a lack of datasets for the growing needs of conversation summarization applications. Thus we collected ForumSum, a diverse and high-quality conversation summarization dataset with human written summaries. The conversations in ForumSum dataset are collected from a wide variety of internet forums. To make the dataset easily expandable, we also release the process of dataset creation. Our experiments show that models trained on ForumSum have better zero-shot and few-shot transferability to other datasets than the existing large chat summarization dataset SAMSum. We also show that using a conversational corpus for pre-training improves the quality of the chat summarization model.

## introduction with increasing number of digital communications, there is an increasing need to manage the exploding amount of information. one way of relieving users from information overload in chat applications is through automatic abstractive summarization by selecting important pieces of information and writing them into accurate, fluent and concise summaries. recently there has been a lot of advances in automatic abstractive summarization using large pretrained language models @xcite @xcite and finetuning them on downstream summarization datasets. however, most of the pre-training and finetuning domains are news documents @xcite and there is a lack of attention to summarizing conversations. in this work, we aim to build a high quality conversation summarization system that generalizes well by creating a new dataset and improving pretraining methods for conversation summarization. our contributions include: ubuntu/nyc @xcite ) is an online thread summarization datasets that contains 100 threads from ubuntuforums.org and tripadvisor.com and their human written summaries. bc3 @xcite consists of 40 email threads each annotated with three summaries by three different annotators. each summary sentence is also annotated with references to the corresponding lines in the emails. we employ bc3, ubuntu and nyc datasets to evaluate transferability of models trained forum-sum and samsum datasets. mediasum, summscreen mediasum @xcite and summscreen @xcite are conversations summarization datasets that use speech transcripts as input and automatically mine summaries from interview overviews and tv shows recaps. while being very large and diverse they contain automatically mined summaries which might suffer from lower quality. ami, icsi @xcite and icsi @xcite are meetings transcripts datasets annotated with abstractive summaries. meeting transcripts are much longer than messenger or email threads. for example, an average ami transcript contains 289 turns, while the average number of turns in forumsum dataset is around 10. meetings transcripts contain more repetitions, backchannel responses and interjections. in this paper we focus on summarizing online messaging conversations. therefore we do not use mediasum, summscreen, ami or icsi for transferability studies. ## forumsum dataset motivated by the lack of diverse multi-speaker conversation summarization datasets we collected fo-rumsum: a conversation summarization dataset from internet forums labeled with human-written summaries. first we collected a list of message board websites. we only kept websites that are relatively popular, and are not present in a blocklist of not appropriate websites. the result contains 281 websites. ## experiments and results following @xcite , we conduct our studies using transformer encoder-decoder models at the base size. it had @xmath0, @xmath1, @xmath2, @xmath3.e. transformer blocks), h for the hidden size, f for the feedforward layer size and a for the number of selfattention heads. ## conclusions we collected forumsum, a diverse and high-quality chat summarization dataset with human written summaries. forumsum can be easily expanded to further improve conversation summarization quality using the released process of dataset creation. our experiments show that models trained on forumsum have good zero-shot and few-shot transferability to other conversation summarization datasets measured by rouge scores and human evaluations. we also show that using a conversational corpus for pre-training improves the quality of the conversation summarization model.