Reference and Document Aware Semantic Evaluation Methods for K orean Language Summarization (2020)

Abstract: Text summarization refers to the process that generates a shorter form of text from the source document preserving salient information. Many existing works for text summarization are generally evaluated by using recall-oriented understudy for gisting evaluation (ROUGE) scores. However, as ROUGE scores are computed based on n-gram overlap, they do not reflect semantic meaning correspondences between generated and reference summaries. Because Korean is an agglutinative language that combines various morphemes into a word that express several meanings, ROUGE is not suitable for Korean summarization. In this paper, we propose evaluation metrics that reflect semantic meanings of a reference summary and the original document, Reference and Document Aware Semantic Score (RDASS). We then propose a method for improving the correlation of the metrics with human judgment. Evaluation results show that the correlation with human judgment is significantly higher for our evaluation metrics than for ROUGE scores.

## introduction the task of text summarization is to generate a reference summary that conveys all the salient information of an original document. there are two strategies for this type of summarization (i.e., extractive and abstractive summarization). with the extractive approach, the most noticeable key sentences are extracted from the source and compiled into a reference @xcite @xcite . the second approach is abstractive, with which a paraphrased summary is generated from the source @xcite @xcite . the generated summary may not contain the same words that appear in the source document. therefore, measuring factual alignment between the generated summary and source document is important @xcite . most summarization models are evaluated using recall-oriented understudy for gisting evaluation (rouge) @xcite , which measures n-gram overlaps between generated and reference summaries. rouge has proven to have a high correlation with manual evaluation methods, such as pyramid @xcite and tac aesop @xcite . however, @xcite showed that the correlation significantly decreased when only one reference summary was provided. additionally, considering the process by which a person manually summarizes a document, rouge is limited, because it does not reflect semantic meanings between generated and reference summaries. for example, when a person summarizes a document, they tend to use words that are implicit while not always using the explicit words from the original document. as the rouge score is computed based on an n-gram overlap, the score can be low even if two words have the same semantic meaning. to overcome this limitation, an evaluation method that considers the semantic information of both the generated and reference summary is required. it is important to examine the factuality between the generated summary and source document, because the generated summary may contain false information. each person summarizes information in different manners, and it is difficult to agree, even after cross-checking @xcite . therefore, the source document should also be considered with generated and reference summary. in this study, we propose metrics for evaluating a summarization model that consider both the source document and reference summary together with the generated summary (see ## related work evaluation methods of text summarization are divided into two strategies: manual and automatic. manual evaluation is expensive and difficult @xcite . several studies have been conducted to develop automatic methods that facilitate fast and low-cost evaluations. there are two types of automatic evaluation methods: extrinsic and intrinsic. an extrinsic automatic method evaluates a summarization model based on how it affects the completion of tasks comprising the judgment of document relevance @xcite . the intrinsic automatic method evaluates quality via a property analysis or by calculating its similarity to a manually generated summary. intrinsic methods include the pyramid method @xcite , the basic-elements method @xcite , and rouge @xcite . the pyramid method inspects various human-made summaries and creates summary content units, each with a scoring weight. the basic-elements method is similar to the pyramid method. rouge evaluates the similarity of the lexical overlap between the candidate and reference summary. as the rouge score is computed based on the n-gram overlap, it does not account for synonymous words or phrases. many approaches have been proposed to overcome this limitation. paraeval @xcite , rouge-we @xcite , rouge 2.0 @xcite , and rouge-g (shafieibavani et al., 2018) have been used to extend rouge to support synonymous constructs. paraeval uses a matching method based on paraphrase tables. rouge-we uses a lexical matching method with a semantic similarity measure and the cosine distances between tokens. rouge 2.0 uses wordnet as a synonym dictionary and computed token overlaps with all synonyms of matched words. rouge-g uses lexical and semantic matching from wordnet. these approaches have limitations because they require hand-crafted lexical and synonym dictionaries, which are particularly difficult to construct in korean. our research is similar to @xcite , which utilized bert to compute semantic score between the generated and reference sentence. however, @xcite does not consider the document, whereas our research considers the document to be characterized in the evaluation of summarization tasks. overall, our research is different from previous approaches in that 1) we propose a method to evaluate generated summary by considering documents as well as reference summary. 2) in addition, our evaluation model is robust to out of vocabulary (oov) words because it leverages a pre-trained neural network (sbert) based on byte pair encoding (bpe) @xcite tokenization method from unsupervised learning. considering the fact that korean is an agglutinative language, this feature is essential. 3) finally, our evaluation model can be further trained to capture more contextualized information both on reference summary and document. text summarization models can be divided into abstractive, extractive, and hybrid. abstractive models reword phrases and create summaries having novel phrases constructed from the original document. recent text summarization approaches have leveraged multi-task and multi-reward training @xcite @xcite , attention-with-copying mechanisms @xcite @xcite , and unsupervised training strategies @xcite . the extractive method extracts the most-suitable sentences (or words) from the source document and copies them directly into the summary. many researchers @xcite @xcite have utilized domain expertise to develop heuristics for refining summary texts. recently, neural-based text summarization models have been proposed to train the model for predicting whether a span of text should be included in the summary @xcite @xcite . reinforcement learning-based summarization models have also been proposed to directly optimize models @xcite @xcite . the hybrid approach uses both abstractive and extractive methods. with this approach, the summarization process is divided into two phases: content selection and paraphrasing @xcite @xcite . ## methodology from ## results in this section, we first report the performance of the summarization model using the rouge and proposed evaluation metrics (subsection 3.1). next, we report how the proposed evaluation metrics correlated to human judgment. we also report the correlation of the proposed evaluation metrics to rouge to show that the proposed methods complement rouge. finally, through qualitative evaluation, we demonstrate the limitations of rouge and the superiority of the proposed evaluation metrics. the abstractive summarization model is based on the neural architecture of @xcite . we trained the summarization model on the daum/news dataset. to evaluate the summarization model, we used rouge and the proposed evaluation metrics. the fine-tuned fwa-sbert was then used to evaluate the proposed semantic scores (s(p, r), s(p, d), and rdass). ## conclusion in this paper, we pointed out the limitation of the widely used rouge evaluation metric when adopting korean summarization. since korean is an agglutinative language, the generated summary having the same semantic meaning with reference summary can be variously expressed. therefore, only leveraging rouge metric can produce inaccurate evaluation results. to overcome this limitation, we proposed rdass (reference and document aware semantic score) evaluation metric. the rdass can reflect deep semantic relationships of a generated, reference summary, and document. through extensive evaluations, we demonstrated that the correlation with human judgment is higher for the proposed evaluation metric (rdass) than for rouge scores. in future work, we will demonstrate the effectiveness of the proposed method in english summarization dataset.