Welcome to the Real World: Efficient, Incremental and Scalable Key Point Analysis (2023)

Abstract: Key Point Analysis (KPA) is an emerging summarization framework, which extracts the main points from a collection of opinions, and quantifies their prevalence. It has been successfully applied to diverse types of data, including arguments, user reviews and survey responses. Despite the growing academic interest in KPA, little attention has been given to the practical challenges of implementing a KPA system in production. This work presents a deployed KPA system, which regularly serves multiple teams in our organization. We discuss the main challenges we faced while building a real-world KPA system, as well as the architecture and algorithmic improvements we developed to address these challenges. Specifically, we focus on efficient matching of sentences to key points, incremental processing, scalability and resiliency. The value of our contributions is demonstrated in an extensive set of experiments, over five existing and novel datasets. Finally, we describe several use cases of the deployed system, which illustrate its practical value.

## introduction getting the gist of a large collection of opinions, such as user reviews and open-ended survey responses, typically requires significant manual work. while word clouds @xcite and key phrases @xcite are somewhat helpful in providing a highlevel view of the data, they are often too crude to fully replace manual analysis. plain-text summaries, on the other hand @xcite @xcite , lack a quantitative dimension, as they do not measure the prevalence of each point in the summary. applying generative ai to summarize large datasets may raise additional issues, such as faithfulness @xcite and scalability. key point analysis (kpa) has been recently proposed as a compelling alternative to the above approaches @xcite . kpa maps the input texts to a set of automatically-extracted short sentences and phrases, termed key points (kps), which provide a concise plain-text summary of the data. the prevalence of each kp may be quantified as the number of its matching sentences. figure 1 shows an example of kpa results summarizing a few thousands of responses to a community survey conducted in the city of austin. kpa has gained significant academic interest, with 17 teams participating in the 2021 kpa shared task @xcite . however, little attention has been given in previous work to practical aspects of building a real-world kpa system. in this work, we make a step towards closing this gap, by describing a deployed kpa system that is being regularly used by multiple teams in our organization (ibm). specifically, we focus on the following issues, which we found to be the most critical when deploying kpa in production: efficient matching. most of the run time in kpa is spent on matching sentences to kps or kp can-didates. we propose a method that combines slow, accurate matching with fast, less accurate matching. this allows run time reduction by a factor of five, while achieving comparable accuracy ( §4). incremental kpa. in many practical scenarios, kpa needs to be performed periodically, over data that is being accumulated over time. we introduce the notion of incremental key point analysis, and propose a modification to the kpa algorithm that enables efficient incremental processing ( §5). ## kpa algorithm overview our system is based on the kpa algorithm of @xcite , which was the best end-to-end performer on the 2021 shared task @xcite . the input for the algorithm is a collection of comments, split into sentences. the algorithm comprises the following steps: the algorithm employs two supervised transformer models: an argument quality model @xcite for the first step, and a matching model @xcite for the second step (for both matching the sentences and identifying semantically-similar candidates foot_0 ). following @xcite , we incorporated two additional models into the kpa system: first, a stance (sentiment) classification model, which labels the input sentences as positive, negative or neutral. 2 when this optional step is performed, we subsequently exclude the neutral sentences and run kpa separately on the positive and negative sentences. this improves both the run time and the matching accuracy. second, we developed an additional kp quality supervised model, specifically designed to select candidates with desirable properties (have a clear stance, discuss a single topic, not too general/specific). this classifier is used in conjunction with the argument quality classifier. the bottleneck of the kpa algorithm is the matching step, in which a transformer-based matching model is applied to compute the match score for each (sentence, kp candidate) pair, resulting in quadratic complexity. other models (stance, quality) are only applied once per sentence, and therefore have marginal effect on the overall runtime. @xcite proposed the following modifications to reduce the matching run time for large datasets: (a) limit the number of kp candidates, and (b) select the kps based on a subset of sentences; then match the rest of the sentences only to the selected kps. even with these improvements, matching run time remains a major issue when deploying a kpa system that should serve many users, and process large datasets. thus, a core challenge in developing a realworld kpa system is improving matching efficiency, without degrading its quality. in the following sections we describe the setup of our experiments, and specifically the data ( §3) and the various matching approaches we experimented with ( §4.1), followed by experimental results ( §4.2). then, we describe additional solutions we implement for real-world kpa: addressing incremental updates ( §5), and a scalable system architecture ( §6). ## data in this section we briefly describe the datasets that were used to train the matching model, and to assess its quality. out of our three training sets and five test sets, two (argkp and argkp-21) are existing public datasets. the remaining six datasets were created as part of this work, utilizing diverse public and internal data sources. each of the datasets includes correct (positive) and incorrect (negative) examples of mapping sentences to kps. the label of each instance was obtained by consolidating multiple human annotations. some statistics on the train and test sets are given in the dev set was used for tuning the matching thresholds, as described in section 4. we release two novel kpa benchamrks, argkp-large and municipal, along with this paper. foot_5 ## use cases the deployed kpa system serves multiple teams across ibm. below we describe several use cases, in which kpa has been applied to extract finegrained insights from large textual datasets, saving the time and cost of manual analysis. feedback on internal applications. ibm uses a common tool to collect internal user feedback and verbatim comments. each product owner or application team has to dedicate time and efforts to manually read, categorize user comments and feedback, summarize and identify actions to be taken to address them. the my cognitive adviser (mca) tool was created as a common solution to be used by owners of internal applications. using kpa services, it analyzes and displays applicationspecific feedback summarized by key points, sentiment, associated sentence count and trends over time. automated analysis is provided monthly or quarterly, depending on the volume of feedback. it allows users to understand the top pain points for their applications, without having to manually review, categorize and label each user feedback. productivity improvement ideas. ibm finance and operations (f&o) launched an organizationwide initiative to improve productivity within the company, by organizing and executing 900+ workshops resulting in 7500+ ideas from employees (posted in slack and mural boards). kpa was used to analyze and summarize these ideas to identify the top 15 key areas for productivity improvements within the company. this analysis gave the f&o executives a very quick synopsis of the workshops, enabling them to hear the voice of their employees, without having to spend weeks manually reviewing and categorizing the ideas. employee engagement survey. kpa has also been applied to analyze the annual ibm employee engagement survey. over 300k employees wrote more than 550k sentences in total. these sentences were automatically classified into positive and negative, and kpa was applied to each set separately to identify positive and negative key points. we also compared year-to-year differences in key points distribution, as well as differences between suborganizations, and between detractors and advocators. kpa results enabled the hr survey research team to extract actionable and valuable insights from this very large dataset, with significantly less effort. ## discussion and conclusion kpa is a promising approach for large-scale quantitative summarization of opinions, with many practical applications. yet, moving from academic experimentation to a real-world implementation poses significant research and engineering challenges. we described a deployed kpa system, for which we presented a comprehensive architecture, as well as algorithmic improvements that enable efficient matching and incremental processing. our contributions have been evaluated in terms of both quality and run time over the most comprehensive set of kpa benchmarks to date, including both internal and public datasets. it is important to note the compound effect of combining the above individual improvements. for example, as shown in in future work, we plan to further explore the use of generative llms for kpa, aiming to achieve both hiqh-quality results and feasible run time.