What Makes a Good and Useful Summary? I ncorporating Users in Automatic Summarization Research (2022)

Abstract: Automatic text summarization has enjoyed great progress over the years and is used in numerous applications, impacting the lives of many. Despite this development, there is little research that meaningfully investigates how the current research focus in automatic summarization aligns with users’ needs. To bridge this gap, we propose a survey methodology that can be used to investigate the needs of users of automatically generated summaries. Importantly, these needs are dependent on the target group. Hence, we design our survey in such a way that it can be easily adjusted to investigate different user groups. In this work we focus on university students, who make extensive use of summaries during their studies. We find that the current research directions of the automatic summarization community do not fully align with students’ needs. Motivated by our findings, we present ways to mitigate this mismatch in future research on automatic summarization: we propose research directions that impact the design, the development and the evaluation of automatically generated summaries.

## introduction the field of automatic text summarization has experienced great progress over the last years, especially since the rise of neural sequence to sequence models (e.g., @xcite @xcite . the introduction of self-supervised transformer language models like bert @xcite has given the field an additional boost (e.g., @xcite @xcite . the-often implicit-goal of automatic text summarization is to generate a condensed textual version of the input document(s), whilst preserving the main message. this is reflected in today's most common evaluation metrics for the task; they focus on aspects such as informativeness, fluency, succinctness and factuality (e.g., @xcite ; nenkova and @xcite @xcite @xcite . the needs of the users of the summaries are often not explicitly addressed, despite their importance in explicit definitions of the goal of automatic summarization @xcite . mani defines this goal as: "to take an information source, extract content from it, and present the most important content to the user in a condensed form and in a manner sensitive to the user's or application's needs." different user groups have different needs. investigating these needs explicitly is critical, given the impact of adequate information transfer @xcite . we propose a survey methodology to investigate these needs. in designing the survey, we take stock of past work @xcite who argues that in order to generate useful summaries, one should take the context of a summary into account-a statement that has been echoed by others (e.g., @xcite . to do this in a structured manner, spärck jones introduces three context factor classes: input factors, purpose factors and output factors, which respectively describe the input material, the purpose of the summary, and what the summary should look like. we structure our survey and its implications around these factors. in figure 1 we give an example of incorporating the context factors in the design of automatic summarization methods. our proposed survey can be flexibly adjusted to different user groups. here we turn our focus to university students as a first stakeholder group. university students are a particularly relevant group to focus on first, as they benefit from using pre-made summaries in a range of study activities @xcite , but the desired characteristics of these pre-made summaries have not been extensively investigated. we use the word premade to differentiate such summaries from the ones that users write themselves. automatically generated summaries fall in the pre-made category, and should thus have the characteristics that users wish for pre-made summaries. motivated by our findings, we propose important future research directions that directly impact the design, development, and evaluation of automatically generated summaries. we contribute the following: c1 we design a survey that can be easily adapted and reused to investigate and understand the needs of the wide variety of users of automatically generated summaries; c2 we develop a thorough understanding of how automatic summarization can optimally benefit users in the educational domain, which leads us to unravel important and currently underexposed research directions for automatic summarization; c3 we propose a new, feasible and comprehensive evaluation methodology to explicitly evaluate the usefulness of a generated summary for its intended purpose. ## related work in section 1 we introduced the context factors as proposed @xcite . each context factor class can be divided into more fine-grained subclasses. to ensure the flow of the paper, we list an overview in appendix a. below, we explain and use the context factors and their fine-grained subclasses to structure the related work. as our findings have implications for the evaluation of automatic summarization, we also discuss evaluation methods. lastly, we discuss the use-cases of automatic summaries in the educational domain. ## survey procedure and participants here we detail our survey procedure. for concreteness, we present the details with our intended target group in mind. the context factors form the backbone of our survey and the setup can be easily adjusted to investigate the needs of different target groups. for example, we ask participants about a pre-made summary for a recent study activity, but it is straightforward to adapt this to a different use-case that is more suitable for other user groups. ## results for each question we examine the outcomes of all respondents together and of different subgroups ( ## conclusion in this paper we focused on users of automatically generated summaries and argued for a stronger emphasis on their needs in the design, development and evaluation of automatic summarization methods. we led by example and proposed a survey methodology to identify these needs. our survey is deeply grounded in past work by spärck jones (1998) on context factors for automatic summarization and can be re-used to investigate a wide variety of users. in this work we use our survey to investigate the needs of university students, an important target group of automatically generated summaries. we found that the needs identified by our participants are not fully supported by current automatic summarization methods and we proposed future research directions to accommodate these needs. finally, we proposed an evaluation methodology to evaluate the usefulness of automatically generated summaries.