W iki S um: Coherent Summarization Dataset for Efficient Human-Evaluation (2021)

Abstract: Recent works made significant advances on summarization tasks, facilitated by summarization datasets. Several existing datasets have the form of coherent-paragraph summaries. However, these datasets were curated from academic documents that were written for experts, thus making the essential step of assessing the summarization output through human-evaluation very demanding. To overcome these limitations, we present a dataset based on article summaries appearing on the WikiHow website, composed of how-to articles and coherent-paragraph summaries written in plain language. We compare our dataset attributes to existing ones, including readability and world-knowledge, showing our dataset makes human evaluation significantly easier and thus, more effective. A human evaluation conducted on PubMed and the proposed dataset reinforces our findings.

## introduction summarization is the task of preserving the key information in a text while reducing its length. recently, many summarization datasets were published and helped push the boundaries of new summarization systems. these datasets differ on several properties, including the domain (e.g., academic or news) and the summary form. pubmed, arxiv, and bigpatent @xcite provide a summary in the form of coherent paragraphs (i.e., each sentence flows smoothly into the next). in contrast, other summarization datasets @xcite @xcite offer a summary in the form of a key points list (i.e., highlights). in this paper, we focus on coherent paragraph summarization datasets. automatic evaluation of summarization systems, e.g., by using the rouge metric, is challenging @xcite and is often inconsistent with human evaluation @xcite @xcite . to understand -and later improve -the quality of summarization systems, it is necessary to conduct a human evaluation. a human evaluation's quality depends on the ease of reading and understanding of the measured text: a simple text does not require annotators with unique expertise, can be evaluated faster, and is easier to annotate correctly. however, existing coherent-paragraph summarization datasets consist of academic papers and cannot be considered easy to read. evaluating such summarization samples requires unique expertise, takes time, and comes at a high cost. in this work, we present wikisum, a new summarization dataset from the wikihow knowledge base foot_0 . the wikisum documents are written in simple english, and the summaries provide "nonobvious tips that mimic the advice a knowledgeable, empathetic friend might give." the key attributes of wikisum are: (1) summaries written as a single, coherent passage. (2) articles and summaries that are easy to read. (3) articles and summaries require less world knowledge to understand. we evaluate the dataset readability and estimate the required world-knowledge in section 3. moreover, we reinforce our results by conducting a human-evaluation of a summarization dataset in section 4. finally, to establish a baseline on the proposed dataset, we benchmark wikisum using recent summarization systems and report their performance on section 5. ## related work the summarization landscape can be roughly divided into three primary summary-forms: (1) single sentence @xcite @xcite -summarize the document in a single sentence; (2) highlights @xcite @xcite ) -a summary in the form of bullets listing the key points in the text; (3) coherent summary @xcite ) -short coherent paragraphs describing the salient information. the summarization datasets from the news domain, which are commonly used for human evaluation, include summaries in the form of highlights or single-sentence summaries. however, summarization datasets written in a co-herent format come from the academic domain, making them extremely difficult to annotate manually. our proposed wikisum is the only dataset written in a coherent format, yet easy for human evaluation. we do not claim that coherent paragraph summaries are better, but rather different; each format has its use cases, and human evaluation should be done on each of the different formats separately. the existing wikihow datasets (koupaee and wang, 2018; @xcite ) can be considered the closest to wikisum, as they originate from the same knowledge base. however, while the existing wikihow datasets split the article to generate the document and summary, wikisum uses the entire article as the document and a summary specifically written by the article's author (called the article quick summary). the former uses the concatenation of the first line of each step, called the step header, as the list of highlights and the remainder of step text's concatenation called "wraptext," as the document 4 . in addition to the different summary-form of the highlight-based wikihow and wikisum, the content of the summaries is significantly different, which can be illustrated by the low bleu-4 (0.06 5 ) between the two. bigpatent @xcite , arxiv and pubmed @xcite are recent summarization datasets with coherent paragraph summaries. these datasets focus on the academic domain and are written for experts. like these datasets, wik-isum is composed of long documents and coherent paragraph summaries. nonetheless, it uses common everyday language and ranges over many domains (see figure 2 ). finally, ## human evaluation we conducted a standard human evaluation on a summarization task, in addition to the automatic readability and the external knowledge metrics. we gathered a pool of 6 annotators, without any prior knowledge of the project, all with a graduate degree (m.sc. or ph.d.) and proficient english readinglevel. we asked them to evaluate summaries generated by pegasus @xcite . the an-dataset time (minutes) difficulty (rating) exhausting (rating) qualified (rating) unknown (%) wikisum 6.8±1.2 1.9±0.3 2.2±0.5 4.2±0.3 0.2±0.1 pubmed 10.0±1.2 3.7±0.3 3.9±0.4 2.2±0.4 3.7±1.4 during the annotation task, we measured the evaluation time and asked the annotators to mark unfamiliar words. in addition, we asked the annotators to rate the following aspects on a 1-5 scale: (a) how difficult was the task? (b) how tiring was it? (c) how qualified are you for this task? after each pair of pubmed and wikihow samples were completed, the annotators selected which dataset they prefer to evaluate. in in the annotation task, we also asked the annotators to mark unfamiliar words in the article. we found a strong correlation between the count of unfamiliar words and the task difficulty, evaluation time, and perceived required qualification (pearson correlation of 0.57, 0.36, -0.48 8 , respectively, 8 many unfamiliar words implied annotators perceived ## model results and discussion to provide both abstractive and extractive baselines for wikisum, we evaluate on pegasus large @xcite , tex-trank @xcite , and the common lead-3 that selects the first three sentences of the document as the summary. we compare the results on wikisum to the arxiv, pubmed, and bigpatent datasets results. to conclude, this paper presents the wikisum dataset, which is drastically simpler for human evaluation than existing summarization datasets where the summary appears as a coherent paragraph. we showed wikisum's simplicity via various readability metrics and demonstrated that the text requires less external knowledge to be understood. finally, we validated our finding via a human evaluation task on wikisum and pubmed. we remark that the quick summaries are indeed used by commercial voice assistants to answer howto questions. as voice assistants gain popularity, so does the importance of such coherent-paragraph summaries.