Evaluation of Review Summaries via Question-Answering (2021)

Abstract: Summarisation of reviews aims at compressing opinions expressed in multiple review documents into a concise form while still covering the key opinions. Despite the advancement in summarisation models, evaluation metrics for opinionated text summaries lag behind and still rely on lexical-matching metrics such as ROUGE. In this paper, we propose to use the question-answering(QA) approach to evaluate summaries of opinions in reviews. We propose to identify opinion-bearing text spans in the reference summary to generate QA pairs so as to capture salient opinions. A QA model is then employed to probe the candidate summary to evaluate information overlap between candidate and reference summaries. We show that our metric RunQA, Review Summary Evaluation via Question Answering, correlates well with human judgments in terms of coverage and focus of information. Finally, we design an adversarial task and demonstrate that the proposed approach is more robust than metrics in the literature for ranking summaries.

## introduction opinion summarisation takes input documents like online reviews or social media posts where users express their opinions on topics and condense them into a single piece of text. the summary should reflect the core opinions expressed in the source documents. recent studies in opinion summarisation have shown an advancement moving from extractive @xcite ; i.a), i.e. copying sections from the original reviews to produce a summary, to abstractive @xcite @xcite , i.e. generating new phrases that reflect the information covered in the original text. despite the advancement in summarisation models, evaluation metrics for opinionated text summaries lag behind. for evaluation of review summaries, traditional token-matching rouge @xcite is still widely used, supplemented with hu-man evaluation on the relative ranking of systemgenerated summaries in terms of quality dimensions such as fluency, coherence, non-redundancy, informativeness, and sentiment @xcite @xcite . it is well understood that rouge cannot capture the same meaning expressed in different token sequences. neural model-based metrics have been proposed for general text summarisation evaluation. especially qaeval @xcite has been proposed for evaluating the information quality of abstractive summaries with respect to the reference summaries. a key step for the success of qaeval for summarisation evaluation is extracting answers and generating questions covering a significant amount of important summarisation content units (scus). generally, noun phrases(np) and named entities(ner) are used to generate question-answer pairs in qa models @xcite @xcite , but their applicability for review summary evaluation is yet to be examined. in this paper, we propose to evaluate review summaries with question answering (qa) based on neural models, with a focus on evaluating the information quality of review summaries. modern abstractive summarisation systems can generate sentences of high linguistic quality -grammatically correct, easy to read and understand-but it is more important to evaluate the information quality of summaries. specifically system summaries of high quality information should express opinions consistent with those in the reference summary. we propose to evaluate the information quality of review summaries, in terms of coverage(recall) and focus(precision) @xcite , where coverage is the amount (proportion) of salient information of the reference summary that the system summary contains, and focus is the amount (propor-tion) of salient information in the system summary. to improve the qa framework for more effective review summary evaluation, we propose to identify opinion-bearing text spans to generate qa pairs rather than relying on nps and ners. in addition to the evaluation of the information quality of summaries, we further propose evaluating the robustness of evaluation metrics for ranking summaries through an adversarial task. our evaluation metric runqa, namely review summaries evaluation via question answering, was evaluated against qaeval and other metrics on an amazon review summarisation dataset @xcite we found that runqa significantly outperforms qaeval and other metrics for evaluating the information quality of summaries, especially in terms of precision. we also found that runqa is the most robust for ranking summaries. ## related work we first discuss automatic metrics for general text summarisation evaluation and then especially discuss the qa-based metrics. ## dataset and baselines we conducted experiments to evaluate the qa model and benchmark runqa against qaeval and other metrics. the dataset we use in our experiments is from @xcite . it was obtained from amazon product reviews, where 60 products from 4 different categories (15 for each category) were randomly selected. for each product there are 8 source reviews, 1 system generated summary using the copycat model @xcite , and 3 reference summaries obtained from amazon mechanical turk 1 . the summary of the dataset is in baseline metrics include the lexical overlapbased rouge family of metrics @xcite , embedding-based metrics bertscore @xcite and moverscore @xcite , as well as the qa model-based metric qaeval @xcite . ## experiments for summary information quality introduced by @xcite , coverage and focus are metrics designed to evaluate information in summaries. coverage(recall) measures the amount of key information expressed in the reference summary that is also captured by the candidate summary. focus(precision) measures the amount of primary information expressed in the candidate summary that is also true in the reference summary. to compute coverage and focus, we need to gather gold-standard scores by recruiting human annotators. then calculate the correlation between the human annotations and the scores generated by the metrics. we followed @xcite and @xcite , and used the customised direct assessment method to collect human scoring annotations using the amazon mechanical turk. the annotation interface is shown in figure 4 . to control possible bias in annotation, each hit contained a balanced number of annotations for both coverage and focus using different products (30 different products). on top of the 30 required annotations, each hit also contained 6 quality control questions. where 3 are reference summary: this purse is well-designed in terms of appearance, but not in terms of usability and reliability. it is smaller than advertised... candidate summary: i love this purse! it is a bit smaller than i thought it would be, but i love it! it's a perfect size for me. question 1: this purse is well-designed in terms of what? reference answer: appearance candidate answer: perfect size question 2: this purse is well-designed in terms of appearance, but not in terms of what else? reference answer: usability candidate answer: na question 3: how big is it than advertised? reference answer: smaller candidate answer: smaller reference summary: this camera bag, constructed as a backpack with padded straps and back is functional and comfortable to wear as well... it is well designed and made of durable materials. . . candidate summary: this is the best backpack i have ever owned. it's very comfortable and holds a lot of stuff... question 1: what is this camera bag constructed as with padded straps and back? reference answer: a backpack candidate answer: backpack question 2: this camera bag, constructed as a backpack with what type of straps and back is functional and comfortable to wear as well? reference answer: padded candidate answer: backpack question 3: what type of material is it well designed and made of? reference answer: durable candidate answer: na hits were restricted to workers from englishspeaking countries, with over 10,000 approved hits and a 98% approval rate. we first collected more than required, then filtered out annotations that failed the quality control tests. it leaves us with an uneven number of annotations per hit (ranging between 3 and 7). for quality control, we implemented several tests. first, work is only considered if a worker passed 4 of the quality control questions. on top of the distractor questions in each hit, we further examine workers' time spent on the task and its variation of scores similar to @xcite . if the amount of time spent or the variation between the scores is suspiciously low, we disregard all of the worker's annotations. lastly, the annotations are removed for workers with a pearson correlation to other workers(agreement score) of less than 0.2. after quality control, a mean pearson correlation of 0.41 is achieved, with a reasonable average time spent (16.35 minutes) and a quality score (94.74%). like @xcite and @xcite , for annotations pass quality controls we standardise the annotation scores to a z-score of each worker before averaging. this helps remove personal bias introduced by different annotators. then take an average among workers who completed the same hit and use the score as the final score for that hit. we collected both the coverage and focus scores for 180 (60 × 3) summary pairs. we use various automatic metrics to generate scores for the candidate summaries based on the reference summaries, and then calculate their correlation with human annotations. each product has 3 reference summaries, we average both the human and metric scores to one final score for each product. the original bertscore suggested when comparing against multiple references, the maximum score should be used as the final score. we calculated both the maximum and average bertscore and found that the average score better correlates with human judgements. therefore, we use average bertscore instead of maximum in this paper. we present the pearson, spearman, and kendall correlations between human annotations for coverage and focus, with various metrics. results are shown in tables 4 and 5 , all results are significant with p-@xmath0.01.