Extending Multi-Document Summarization Evaluation to the Interactive Setting (2021)

Abstract: Allowing users to interact with multi-document summarizers is a promising direction towards improving and customizing summary results. Different ideas for interactive summarization have been proposed in previous work but these solutions are highly divergent and incomparable. In this paper, we develop an end-to-end evaluation framework for interactive summarization, focusing on expansion-based interaction, which considers the accumulating information along a user session. Our framework includes a procedure of collecting real user sessions, as well as evaluation measures relying on summarization standards, but adapted to reflect interaction. All of our solutions and resources are available publicly as a benchmark, allowing comparison of future developments in interactive summarization, and spurring progress in its methodological evaluation. We demonstrate the use of our framework by evaluating and comparing baseline implementations that we developed for this purpose, which will serve as part of our benchmark. Our extensive experimentation and analysis motivate the proposed evaluation framework design and support its viability.

## introduction large bodies of texts on a topic oftentimes contain extensive information that is challenging for a potential reader to handle. traditionally, information seeking tasks, like search, question-answering (qa) and multi-document summarization (mds), are single-round input-output processes that can serve the information seeker only to a limited extent. this calls for an interactive setting where a user can guide the information gathering process. for search and qa, this type of research has been gaining momentum recently in areas such as exploratory search @xcite and conversational qa @xcite . for mds, where interaction would allow a user to affect summary content, only sporadic works have been seen over the years (e.g., @xcite @xcite @xcite @xcite . a key gap in the development and adoption of interactive summarization (denoted here intsumm) solutions is the lack of evaluation methodologies and benchmarks for meaningful comparison of systems, similarly to those for static (non-interactive) summarization (e.g., @xcite . the previous works on interactive or customizable summarization of multi-document sets are distinct, with proprietary evaluations that do not admit comparison. furthermore, the evaluation processes are often not scalable and replicable, or do not give a comprehensive enough assessment. in this paper we develop an end-to-end evaluation framework for intsumm systems. the framework starts with real user session collection on a system, via a concrete process of controlled crowdsourcing that we designed for this task. the sessions are then measured to produce absolute scores for the system, allowing for robust system comparison. our framework supports a general notion of expansion-based interactive summarization, where the textual summary gradually expands in response to user interaction. figure 1 presents an intsumm system that we implemented to illustrate this notion ( §5.1). to ensure our evaluation framework is sound, we developed the framework in multiple cycles accompanied by user studies and extensive crowdsourcing experimentation. our main contributions are as follows. we demonstrate the use of our full evaluation framework on two intsumm systems that we implemented, which apply different algorithms but share a common user interface, with @xcite mds dataset. analysis shows favorable results in terms of internal consistency between sessions, users, and different evaluation measures, indicating that our solutions may serve as a promising benchmark for future intsumm research. see §5. the evaluation procedures and systems are available publicly. 1 ## background traditional mds has been researched extensively (e.g. @xcite @xcite . it encompasses variants of query-focused summarization @xcite , orienting the output summary around a given query (e.g. @xcite @xcite @xcite , and incremental update summarization (dang and @xcite , generating a summary of a document set with the assumption of prior knowledge on an earlier set (e.g. @xcite @xcite . evaluation approaches predominantly include automatic rouge @xcite ) measurement, i.e. word overlap against reference summaries, and manual responsiveness @xcite scores or pairwise comparison @xcite between summaries. in the related qa task @xcite , a system extracts an answer for a targeted question. similarly, in the interactive setting, a conversational qa @xcite system extracts answers to a series of interconnected questions with a clear informational goal. to check correctness in both cases, a system answer is simply compared to the true answer via text-comparison. on the contrary, in the exploratory style of intsumm, where the knowledge desired is less certain, evaluation must consider dynamically accumulating information. exploratory search @xcite addresses the need for converting big data to knowledge via human-machine cooperation. for example, interactive information retrieval @xcite focuses on fine-tuning document retrieval interactively, and complex-interactive-qa (ciqa) @xcite involves interacting with a system to generate a passage that answers a complex question. evaluation is a major challenge in dealing with these tasks @xcite @xcite . firstly, real users must use the system being evaluated by completing a task-appropriate assignment, requiring large-scale user studies that highly increase the cost and complexity of evaluation. furthermore, varying user behavior could mean distorted session comparison. then, a system is measured on the basis of its final outputs, mostly disregarding the evolvement of the interactional session. among interactive summarization, in the querychain focused summarization task @xcite , a chain of queries yields a sequence of short summaries, each refraining from repeating content. the task's evaluation relies solely on pre-defined sequences of queries with a respective reference summary per iteration (laboriously prepared by experts) that disregards previous outputs by the system. other interactive summarization systems, such as @xcite ; @xcite , present a preassembled summary with several levels of detail, allowing a user to drill down to or expand on information of interest. these works do not evaluate in a manner that is comparable to others, and do not consider information variation due to interaction. they perform small-scale userstudies for preference between their system and static variants, or a single automatic assessment of the fully expanded final summary. we address all of the above-mentioned evalua-tion issues, specifically targeting the intsumm task, where the interaction-induced outputs are purely textual summary snippets of the input document set. ## intsumm evaluation an intsumm system is evaluated by measuring its performance on multiple sessions produced as a result of human operation. the input of a session, σ, is a set of documents, d, on which to explore. a session comprises an automatically generated initial summary, σ 0 , and a sequence of user-posed requests, q i , and corresponding output responses, r i . the responses can be viewed as expansions of σ 0 . consequently, the overall interactive summary resulting from σ defines a sequence of incrementally expanding snapshots [σ 0 , σ 1 , . . . , σ |σ| ] where σ @xmath0. each snapshot may thus be regarded as a static summary, allowing static summarization measures to be applied on it. for compared intsumm systems s 1 , ..., s m , we require at least u sessions of distinct users interacting with s i on each test document set @xmath1, ..., d n }. assuming such sessions, we next define automatic and manual evaluation measures, and defer details on adequate session collection to §4. importantly, all measures are based on established evaluation mechanisms used in static summarization and interactive systems, that we extend or adapt for the intsumm setting, and that are practically linear in time to the length of the session sequence. together, the set of measures we define provide an encompassing assessment adequate for the evaluation of interactive summary systems. ## session collection the evaluation of interactive systems requires real user sessions, as explained in §3. using a prototype intsumm system, described in §5.1, we conducted several cycles of session collection which uncovered multiple user-related challenges, in line with previous work on user task design @xcite @xcite . in particular, recruited users may make undue interactions due to insincere or experimental behavior, yielding noisy sessions that do not reflect realistic system use. additionally, without an objective informational goal, a user interacts with the system according to subjective interests, producing sessions that are objectively incomparable. controlled crowdsourcing method. employing experts to use an interactive system in a user study is usually unnecessary and hinders scalability and accessibility for researchers, making crowdsourcing an appealing and less expensive alternative. while crowdsourcing is ordinarily used for annotation jobs, we show its suitability for system session collection. we designed a three-stage controlled crowdsourcing protocol that mitigates the aforementioned session collection challenges, while filtering out unsatisfactory workers (further details in appendix b). the first stage is a trap task whose aim is to efficiently filter out insincere workers, and, conversely, discover workers with an ability to apprehend salient information within text. the second stage assigns practice tasks that familiarize the workers to the intsumm system interface to prevent experimentation in the actual sessions to be evaluated. here, the users are also presented with a grounding use-case, or 'cover-story' as termed by @xcite . the use-case states an objective common goal to follow in interacting with the system, to minimize the effect of subjective preferences, and allow comparison against respective reference summaries with a similar objective goal. an example use-case to follow, applied in our experiments, is "produce an informative summary draft text which a journalist could use to best produce an overview of the topic". the use-case is strongly emphasized during practice sessions with integrated guidelines. workers completing two practice assignments with predominantly relevant interactions are invited to continue on to the final stage. the evaluation session collection stage involves interacting with the evaluated system, for a minimum amount of time per session (e.g., 150 seconds in our experiments), to produce a summary on a topic in light of the same assigned use-case as in the practice stage. each worker may explore a topic once, and the overall goal is recording sufficiently many sessions per combination of system and topic. generally in interactive tasks, systems are manually examined over a rather small number of instances (e.g. topics), with only a few users per instance, due to the high cost and complexity of collecting such sessions with experimenters. for example, @xcite assessed their system on 10 topics, and the ciqa benchmark @xcite had 6 topics per tested subtask. our session collection technique provides a more scalable approach, facilitating larger collection processes (e.g., in our experiments in §5 we used 20 topics @xmath2. we note that, in use cases or domains where experts are required, the proposed three-stage session collection protocol is still fully relevant. it is not limited to the crowdsourcing setting, and can be applied within controlled user-studies if needed. wild versus controlled crowdsourcing. we illustrate the benefit of the controlled crowdsourcing procedure described above by comparing its results with a "wild" crowdsourcing preliminary experiment. the latter experiment applied basic workerfiltering (99% approval rate and 1000 approved assignments on amazon mechanical turk foot_2 (amt)) and did not apply the trap and practice tasks. for quality control, a post-session questionnaire was assigned in order to catch insincere workers. analysis of the collected sessions showed a substantial improvement in querying behavior in controlled crowdsourcing over sincere wild crowdsourcing (filtering out the insincere wild crowdworkers) -the former scored higher than the latter on every evaluation metric. ## experiments we carried out experiments that assess our full evaluation framework and demonstrate its utility. as the few existing intsumm systems were not readily available or suitable for adaptation to our experimental setup, we developed an intsumm system of our own, shown in figure 1 , with two different algorithmic implementations for comparison. we gathered user sessions with our controlled crowdsourcing procedure and evaluated their quality with our defined measures. ## discussion and conclusion we proposed a comprehensive evaluation framework for user-guided expansion-based interactive summarization -a vital ingredient for the methodological advancement of interactive summarization research which was unaccounted for until now. our controlled crowdsourcing procedure makes intsumm system session collection accessible, scalable and replicable. the evaluation measures in our framework provide a thorough assessment with absolute scores that enable comparison of intsumm systems. our framework provides the means to advance intsumm research on system development and improved evaluation. all solutions, including our implemented baseline systems, are publicly available to enable comparison of new intsumm systems to ours on any mds dataset. in future work, it is worthwhile to separately assess the effectiveness of individual interaction modes, including ones incorporated in our implementation and others, e.g., full questions input by users. these would require further experimentation, additional evaluation metrics, and the possible use of datasets from tasks other than mds. within our expansion-based framework, we can consider additional measures of textual consistency, coherence, and relevance of responses to queries. we may also test additional approaches for summarization: e.g., abstractive summarization for flexible synthetic summary generation, requiring further evaluation of factuality and truthfulness. beyond our framework, that targets objective quality, intsumm systems should also be evaluated according to their compatibility with personalized, subjective use.