Semantic Similarity Based Evaluation for Abstractive News Summarization (2021)

Abstract: ROUGE is a widely used evaluation metric in text summarization. However, it is not suitable for the evaluation of abstractive summarization systems as it relies on lexical overlap between the gold standard and the generated summaries. This limitation becomes more apparent for agglutinative languages with very large vocabularies and high type/token ratios. In this paper, we present semantic similarity models for Turkish and apply them as evaluation metrics for an abstractive summarization task. To achieve this, we translated the English STSb dataset into Turkish and presented the first semantic textual similarity dataset for Turkish as well. We showed that our best similarity models have better alignment with average human judgments compared to ROUGE in both Pearson and Spearman correlations.

## introduction automatic document summarization aims to produce a summary that conveys the salient information in the given text(s). automatic summarizers provide reduction in the size of the text, as well as, combine and cluster different sources of information, while preserving the informational content. there are two approaches to summarization: extractive and abstractive. extractive summarization yields a summary by extracting important phrases or sentences from the document. in contrast, abstractive summarization provides a much more human-like summary by capturing the internal semantic meaning and generating new sentences. rouge is a widely used evaluation metric in text summarization. it compares the system summary with the human generated summary or summaries, by considering the overlapping units such as n-gram, word sequences and word pairs @xcite . however, in abstractive summarization systems, the generated summary does not necessarily contain the same words in the gold standard summary. on the contrary, an abstractive summarization model is expected to generate new words that may not even appear in the source. for agglutinative languages, the ineffectiveness of rouge metric becomes more apparent. for instance, both of the following sentences has the meaning "i want to call the embassy": büyükelc ¸iligi aramak istiyorum. ## related work the most widely used evaluation metric for summarization is rouge which compares the system summary with the human generated summary or summaries by considering the overlapping units such as n-gram, word sequences and word pairs @xcite . recently, there has been a range of studies focusing on the evaluation of factual correctness in the generated summaries. falke et al. (2019) has studied whether textual entailment can be used to detect factual errors in generated summaries based on the idea that the source document should entail the information in a summary. the authors investigated whether factual errors can be reduced by reranking the alternative summaries using models trained on nli datasets. they found that out-of-the-box nli models do not perform well on the task of factual correctness. kryscinski et al. (2020) proposed a model-based approach on the document-sentence level for verifying factual consistency in generated summaries. zhao et al. (2020) addressed the problem of unsupported information in the generated summaries known as factual hallucination. durmus et al. (2020) and wang et al. (2020) suggested question answering based methods to evaluate the faithfullness of the generated summaries. in addition to the studies focusing on summarization evaluation, there are some recently proposed metrics to evaluate generated text with the gold standard. @xcite proposed bertscore that uses bert @xcite to compute a similarity score between the generated and reference text. several recent works proposed new evaluaiton metrics for machine translation (bleurt @xcite , comet @xcite , yisi @xcite , prism @xcite ). ## experiments for semantic textual similarity in order to assess the semantic similarity between a pair of texts, there are two main model structures: 1) sentence representation models that try to map a sentence to a fixed-sized real-value vectors called sentence embeddings. 2) cross-encoders that directly compute the semantic similarity score of a sentence pair. in this paper, we experimented with state-of-theart sentence representation models that are applicable to turkish (language-specific and multilingual models) and bert cross-encoders. in sentence representation models, we obtained the semantic similarity scores using cosine similarity. all models were tested on the stsb-tr test dataset. ## experiments for summarization to investigate the effectiveness of our semantic similarity models for summarization evaluation, we computed the correlations of rouge scores and our best performing four similarity models with human judgments for a state-of-the-art abstractive model. we reported semantic similarity scores for extractive baselines as well in order to observe their alignmnet with the rouge scores. ## results quantitative analysis we computed pearson and spearman correlations of human judgments with semantic similarity and rouge scores. correlation values can be seen in while the main idea of this paper is to evaluate abstractive summarization, we also showed that an extractive lead-3 baseline yields better semantic similarity scores compared to the abstractive mt5 although it outperforms the extractive baselines in terms of bertscore and rouge scores. ## conclusion in this study, we presented the first turkish semantic textual similarity corpus, called stsb-tr, by translating the original english stsb dataset via machine translation. we showed that the dataset has high quality translations and does not require costly human annotation. we applied state-of-theart models to the stsb-tr dataset, and used the best performing four models as evaluation metrics for the text summarization task. we used natural language inference (nli) models and observed that we can improve our semantic similarity models. we found high correlations between human judgments and our models, compared to bertscore and rouge scores. our qualitative analyses showed that the proposed models can capture the semantic similarity of reference and predicted summaries which cannot be caught by rouge scores. we conclude that our models can be applied as evaluation metric to abstractive summarization in turkish.