Artemis: A Novel Annotation Methodology for Indicative Single Document Summarization (2020)

Abstract: We describe Artemis (Annotation methodology for Rich, Tractable, Extractive, Multi-domain, Indicative Summarization), a novel hierarchical annotation process that produces indicative summaries for documents from multiple domains. Current summarization evaluation datasets are single-domain and focused on a few domains for which naturally occurring summaries can be easily found, such as news and scientific articles. These are not sufficient for training and evaluation of summarization models for use in document management and information retrieval systems, which need to deal with documents from multiple domains. Compared to other annotation methods such as Relative Utility and Pyramid, Artemis is more tractable because judges don’t need to look at all the sentences in a document when making an importance judgment for one of the sentences, while providing similarly rich sentence importance annotations. We describe the annotation process in detail and compare it with other similar evaluation systems. We also present analysis and experimental results over a sample set of 532 annotated documents.

## introduction given an input source document, summarization systems produce a condensed summary which can be either informative or indicative. informative summaries try to convey all the important points of the document @xcite , while indicative summaries hint at the topics of the document, pointing to information alerting the reader about the document content @xcite . an informative summary aims to replace the source document, so that the user does not need to read the full document @xcite . an indicative summary, on the other hand, aims to † work done while an intern at microsoft. ‡ work done while an employee of microsoft. ## annotation methodology figure 2 shows a high-level diagram representing the annotation process for artemis. given a document as input, the preprocessing step consists of first dividing the document into sections, each of which is further divided into paragraphs. the section and paragraph boundaries are computed based on a set of heuristics that depend on signals like explicit section headers as well as constraints on the number of sentences shown at each screen. the hypothetical document in figure 2 is divided into two sections with two paragraphs each. the first section contains sentences {1 . . 6}, with two paragraphs containing sentences {1 . . 3} and {4 . . 6} respectively. the second section contains sentences {7 . . 12}, again with two paragraphs containing sentences {7 . . 9} and {10 . . 12}. a salient sentence is defined as a sentence that includes a main concept or idea for summarizing the text, or a fact or an argument emphasized by the author * . several example sentences are provided in the judge guidelines to help them distinguish salient sentences from non-salient sentences. at a high-level, the judges are trained to select sentences that allow a reader to decide whether to read the full document or not. to summarize the document, judges proceed in a bottom-up manner starting from paragraphs (left-toright in figure 2 ). a judge is first asked to summarize each paragraph in a section by selecting a few salient sentences. a minimum number of sentences are required for each paragraph-summary † . once a paragraph has been summarized, the annotation continues to the next paragraph till paragraph-level summaries are created for all the paragraphs in a section. for the document in figure 2 , the judge selected sentences {2, 3} for the first paragraph and sentences {4, 6} for the second paragraph. once all the paragraphs in a section are summarized, the judge is asked to create a summary for the entire section. however, the judge doesn't have to look at all the sentences in the section to build the section-level summary. instead, they only select from the set of sentences previously selected to summarize the paragraphs of the section. for example, for summarizing the first section in figure 2 , the judge only needs to select from the set of sentences {2, 3, 4, 6}, instead of the entire set of sentences {1 . . 6} that comprise the section. in the example, the judge decided to use the sentences {3, 4, 6} for summarizing the first section. once a section is summarized, the annotation proceeds to the next section in a similar manner. once all the sections of a document are summarized, the judge is asked to build the document summary by selecting from sentences that they had previously selected to build the section-level summaries. in figure 2 , the judge selected sentences {3, 4, 7, 11} for the document level summary. finally, the judge is asked to build a short summary for the document by selecting three most salient sentences from their document-level summary. artemis's hierarchical annotation process considerably reduces the cognitive load on the judges. by reusing judgements made at previous steps, judges are able to successively summarize long documents by divide-and-conquer. for creating document sentences #para #sec #doc #short (1) this content should be viewed as reference documentation only, to inform it . . . 0 0 0 0 (2) microsoft employees need to stay aware of new company products, services, processes, and personnel-related developments in an organization that provides . . . the document-level summary in the hypothetical example in figure 2 , the judge only needs to look at the 6 sentences {3, 4, 6, 7, 8, 11} selected for the two section-level summaries, instead of having to go over the entire set of 12 sentences. sentences (1) and (5) were deemed by every judge as not salient. sentence (3) was selected by four judges as salient up to document-summary level, but one of the judges dropped it at shortsummary level. similarly, sentence (4) was selected at paragraph-summary level by five judges, but only two judges kept it till the document and short-summary level. in section 4, we present statistics on a sample annotated document set that characterize the annotation process in more detail. ## related work we now compare artemis with existing summarization evaluation methods. we start with discussing relative utility, which is most related to our methodology, and describe how artemis obtains similar judgments, but with a light-weight process where judges don't need to look at the entire input document when annotating a sentence. following this, we discuss duc evaluations, rouge and the pyramid method. finally, we discuss some of the recent trends in summarization evaluation. @xcite introduce relative utility (ru) as an evaluation metric to account for summary sentence substitutability (sss) problem in coselection metrics. co-selection metrics are evaluation metrics for extractive summarization that depend on text unit overlap with ideal reference summaries created by judges. the sss problem arises because the judges only provide information about the sentences that they selected for a fixed-length summary. however, other sentences in the document might be equally good candidates for the summary. human judges often disagree about which are the top n% of the sentences in a document @xcite . ## annotated data analysis we present analysis on a sample dataset of 532 microsoft word documents crawled from the web with no domain restrictions, thus creating an opendomain dataset. we extracted the text from the word documents for our annotation. the annotation framework does not rely on word document format and can be used to annotate any document for which the raw text can be extracted. the data was annotated by a set of managed judges who were trained extensively for artemis annotation process using detailed guidelines and illustrative examples. for additional quality control, we used a set of gold documents annotated by the development team for initial qualification tests for the judges as well as their ongoing evaluation. we divided the sample dataset into train, dev and test partitions, as shown in ## experiments we evaluate a number of baseline methods on the sample annotated document set, partitioned into train, dev and test as described in for these experiments, the document-level summary created by each judge for a document is treated as an independent reference summary and we evaluate the candidate summary against all the reference summaries using rouge-f scores. we used the code released by @xcite for reproducing cheng&lapata, summarunner and seq2seqrnn systems. the rouge-f score for each system on the test data is shown in the lead baseline achieves a rouge-1 score of 44.94, which is significantly lower than the other systems as well as the oracle. this shows that compared to news summarization, selecting the first few sentences is a much weaker baseline for open-domain summarization. the summarunner system does better than cheng&lapata, potentially due to its incorporating multiple signals for content, salience, novelty and position. seq2seqrnn performs the best, which is consistent with the results reported in @xcite . there is still a gap between these systems and the oracle method, which achieves a rouge-1 score of 73.28. ## concluding remarks in this paper, we described artemis, a novel hierarchical annotation methodology for indicative, extractive summarization. we described the annotation process in detail and compared it with relative utility, duc evaluation methodology, the pyramid method as well as other recent methods for summary content evaluation. we also presented analysis over a sample annotated dataset to characterize various properties of annotation process such as distribution of salient sentences and judge agreement. finally, we showed experimental results for a set of baseline summarization systems using the annotated dataset. indicative summaries are useful in a number of scenarios involving information triage such as document management and information retrieval systems. however, summarization models for such systems need to be able to summarize documents from multiple domains. most existing summarization datasets are single-domain and focused towards news, and hence are not sufficient for training and evaluating models for these applications. artemis provides a low-cost methodology for annotating multi-domain indicative summaries compared to systems such as pyramid and relative utility while producing similarly rich annotations. artemis summary annotations contain sentences that provide information about important topics in the document. the summaries are indicative because they do not aim to convey all the important points for a given information need, but instead, give a sense of what topics are covered in the document. the set of annotations in artemis can be seen as a coarse partitioning between important and non-important sentences in an input document. thus, models trained on these annotations can also be used as an importance signal in a larger pipeline for creating informative summaries.