An Exploration of Post-Editing Effectiveness in Text Summarization (2022)

Abstract: Automatic summarization methods are efficient but can suffer from low quality. In comparison, manual summarization is expensive but produces higher quality. Can humans and AI collaborate to improve summarization performance? In similar text generation tasks (e.g., machine translation), human-AI collaboration in the form of “post-editing” AI-generated text reduces human workload and improves the quality of AI output. Therefore, we explored whether post-editing offers advantages in text summarization. Specifically, we conducted an experiment with 72 participants, comparing post-editing provided summaries with manual summarization for summary quality, human efficiency, and user experience on formal (XSum news) and informal (Reddit posts) text. This study sheds valuable insights on when post-editing is useful for text summarization: it helped in some cases (e.g., when participants lacked domain knowledge) but not in others (e.g., when provided summaries include inaccurate information). Participants’ different editing strategies and needs for assistance offer implications for future human-AI summarization systems.

## introduction text summaries provide short overviews of long documents or document collections, allowing readers to understand the content without the need to read full documents. for example, news summaries outline key points so that readers do not have to read the entire article. for scientific papers, abstracts allow readers to easily understand the extent of the work and decide whether the paper is relevant to them. while these human-written summaries are typically high quality, a human's time and energy is limited and such tasks require heavy cognitive load @xcite . therefore, increasing research effort has explored machine models to generate summaries automatically @xcite @xcite . while recent advances in learning algorithms and data have resulted in models that can generate relatively high-quality summaries, human summarization is still the gold standard. further, training models require large, high-quality summarization datasets that are expensive to curate. taking advantage of the complementary strengths of humans and ai, can they collaborate to improve summarization performance? in the area of machine translation, a common method of human-ai collaboration is human post-editing of ai-generated text, which increases human productivity and improves the quality of translation @xcite . however, in spite of its potential impact, studies of post-editing for summarization have been very limited, e.g. @xcite , in the medical domain. to bridge this gap, we performed a large-scale human subject experiment (72 participants) investigating the utility of post-editing provided summaries in text summarization for informal (reddit) and formal (news) datasets. we expect tradeoffs in quality and efficiency (i.e., it might take longer to write better summaries), so we are interested in whether post-editing can actually improve efficiency or quality over manual methods, as well as the effects on users' experience. this work is an important step toward understanding the benefits and drawbacks of post-editing as opposed to manual text summarization. our main contributions can be summarized as follows @xcite (1) we present the first large, human subjects experiment of post-editing for text summarization; (2) we show how post-editing impacts summary quality, efficiency, and user experiencewhere it is useful and where it is not; and (3) we create, and make public, two new datasets, each with 360 human-evaluated summaries for news and reddit posts-either written manually or post-edited on provided summaries. ## evaluating post-editing for text summarization we explored how providing summaries for postediting affects (rq1) final summary quality, (rq2) efficiency, and (rq3) user experience, compared to fully manual or fully automatic approaches for two domains: social media and news. participants reviewed documents and summarized them, either without any assistance (manual) or provided with a human-written or ai-generated summary that they could edit (post-edit). a distinct set of annotators then evaluated the quality of the summaries. we included both human-written and ai-generated summaries in our study to explore post-editing for different summary types and qualities. ## results we report on the impact of post-editing on summary quality, efficiency, and user experience. ## discussion this work is the first large-scale study of postediting for text summarization, providing valuable insights on the benefits and drawbacks. we discuss these, as well as outline future research directions and design recommendations for post-editing summarization systems. finally, we discuss the limitations of our experiments. post-editing was useful when domain context is needed. as our participants were not well-versed in the british news content of xsum, the provided summary was "useful" as a starting point (as described in qualitative responses), so that they did not have to write from scratch. this is similar to machine translation literature, which suggests that monolingual editors, despite lacking the knowledge of the other language, can still effectively improve the quality of translation via post-editing @xcite . beyond post-editing, systems could provide additional support when users lack domain knowledge or context, such as inline web searching to learn about unknown terms or phrases (e.g., the rules of cricket). post-editing was less useful when the provided summaries were low quality. low quality, particularly inaccurate or incoherent, provided sum-maries can be confusing and hard to edit, making them less "useful" as summarization assistance. ideally, such summaries are not provided for postediting, as manual summarization would be better in those cases. future systems should explore techniques for determining whether to provide a summary or not, based on desired summary qualities. finally, systems might provide transparency, e.g., highlighting the important details in the original text @xcite . then users can then decide for themselves whether or not those details are important and the summary should be trusted or ignored. post-editing can lead to over-reliance and stifle creativity. humans have a tendency to overrely on ai systems @xcite : prior work on text generation found users consider the provided text as an "authority" and thus feel apprehensive to make significant edits @xcite . in our experiment, some participants reported a similar tendency to over-rely on the provided summary or were distracted from writing their own version of the summary; some even developed their own combative strategies: writing their summaries manually first and then referring to the provided assistance. therefore, future systems might allow different workflows, where summaries are shown before or after manual summarization (or not at all). post-editing systems should cater to varied users' preferences and needs. users have varied summarization strategies and needs for assistance as a result of personal preferences and their experience with the domain. in our study, some preferred more control over the final summary, making lots of edits, while others made fewer edits, and a few did not use the provided summaries at all. these differences in writing style (and their effects on post-editing in text summarization) are also noted in prior work @xcite . therefore, systems should give users control over the assistance they receive and alternative workflows, based on their preferences and needs. users' needs also vary by their target audience; users might desire summaries that are longer or shorter, or more formal or informal based on who they expect will read them. post-editing can help, allowing users to tailor summaries to different audiences with the same underlying content. future post-editing systems might provide multiple summary options, with diverse content and/or style, from which users can choose. limitations. we note possible limitations of our results due to the length and nature of the task: participants only interacted with the summarization system for a short time (less than an hour) and for a task, for which they lacked ownership; also, including more participants would have given more statistical power for comparing conditions. future work should perform experiments with more realistic and longer-term summarization engagements. regarding our datasets, we chose two to differentiate between formal writing (i.e., news articles) and informal writing (i.e., social media posts). however, we did not experiment with more societally critical summarization tasks, such as medical or legal documents. while post-editing was useful when more domain context was needed, it is unclear how our findings would generalize to more high-risk scenarios. ## conclusion to take advantage of the complimentary strengths of ai-which can produce summaries quicklyand humans-which can write summaries wellwe explored how human-ai collaboration (i.e. postediting) impacts summary quality, human efficiency, and user experience for text summarization. through the first large-scale study on post-editing for text summarization, we provide valuable insights on the benefits and drawbacks: compared to summarizing manually, post-editing was helpful for formal news articles, where participants lacked domain knowledge, while post-editing was less helpful for informal social media posts, for which the reference tl;dr summaries sometimes included inaccurate information. we also observed differences in participants editing strategies and needs as well as concerns of over reliance, all of which deserve future exploration. we hope this initial exploration provides a starting point for future research on post-editing in text summarization. criteria explanation essence the summary is a good representation of the post. clarity the summary is reader-friendly. it expresses ideas clearly. accuracy the summary contains the same information as the longer post purpose the summary serves the same purpose as the original post. style the summary is written in the same style as the original post.