Decision-Focused Summarization (2021)

Abstract: Relevance in summarization is typically de- fined based on textual information alone, without incorporating insights about a particular decision. As a result, to support risk analysis of pancreatic cancer, summaries of medical notes may include irrelevant information such as a knee injury. We propose a novel problem, decision-focused summarization, where the goal is to summarize relevant information for a decision. We leverage a predictive model that makes the decision based on the full text to provide valuable insights on how a decision can be inferred from text. To build a summary, we then select representative sentences that lead to similar model decisions as using the full text while accounting for textual non-redundancy. To evaluate our method (DecSum), we build a testbed where the task is to summarize the first ten reviews of a restaurant in support of predicting its future rating on Yelp. DecSum substantially outperforms text-only summarization methods and model-based explanation methods in decision faithfulness and representativeness. We further demonstrate that DecSum is the only method that enables humans to outperform random chance in predicting which restaurant will be better rated in the future.

## introduction human decision making often requires making sense of a large amount of information. for instance, doctors go through a myriad of medical notes to determine the risk of pancreatic cancer, and investors need to decide whether a stock price will increase based on hundreds of analyst reports. in these cases, summarization can potentially support human decision making by identifying the most relevant information for these decisions @xcite . ideally, decision-focused summarization should incorporate insights about how decisions can be inferred from text. however, typical summarization methods in nlp define relevance based on our method (decsum) covers the full distribution, while presumm, a text-only summarization method, concentrates on the right side, and integrated gradients, a model-explanation method, misses the middle part. the textual information exclusively. an example desideratum is textual non-redundancy @xcite , which encourages the summaries to cover diverse information in the input documents. fully optimizing this text-only criterion can be counter-productive for decision making: information about a knee injury does not really help understand the risk of pancreatic cancer, and the disclaimers in financial analysts may not be the most relevant for investment decisions. in this work, we investigate the potential of leveraging a supervised decision model for extractive decision-focused summarization. a predictive model that learns to make a decision given the full text can encode valuable insights about how the decision can be inferred from text. given that @xcite shows that many policy problems depend on predictive inference, incorporating model-based insights into summarization can be widely applicable to many decisions in high-stake scenarios such as finance and healthcare. we propose novel desiderata for decisionfocused summarization in addition to textual nonredundancy and formalize them based on model behavior. first, decision faithfulness suggests that the selected sentences should lead to the same decision as using the full text based on the model. this desideratum is analogous to sufficiency in evaluating the interpretability of attribution methods @xcite , as attribution methods should ideally identify sentences that would "explain" the model's decision with all sentences. this observation also highlights the connection between explanation and decision-focused summarization. in addition to faithfulness, decision representativeness resembles textual non-redundancy in the decision space. fig. 1 illustrates the decision distribution of all individual sentences in the input documents, i.e., model predictions given each sentence, and sentences chosen by different methods. ideally, the selected sentences should be representative of this overall decision distribution. our method is designed to optimize this desideratum, whereas text-only summarization methods and model-based explanation methods do not aim to select sentences that represent the whole distribution. to evaluate our proposed method, we formulate a future rating prediction task on yelp, inspired by investment decisions. the task is to predict a restaurant's future rating given the first ten reviews. automatic metrics demonstrate that our method (decsum) outperforms text-only summarization methods and model-based explanation methods in decision faithfulness and decision representativeness. decsum also improves textual nonredundancy over the baselines, although at the cost of grammaticality and coherence. human evaluation further shows that decsum is the only method that enables humans to statistically outperform random chance in predicting which restaurant will be rated better in the future. to summarize, our main contributions are: ## method in this section, we formalize decision-focused summarization and three desiderata. we then provide a greedy algorithm to optimize the three desiderata. ## experiment setup our approach is contingent on a machine learning model that can make decisions based on the input text. in this section, we discuss our dataset split and choice of this ml model, baselines summarization approaches, and evaluation strategies. ## results in this section, we compare the quality of summaries from our proposed decision-focused summarization with other existing approaches, both through automatic evaluation metrics and human evaluation. automatic metrics show that decsum provides better decision faithfulness, decision representativeness, textual non-redundancy than other baselines, but sacrifices other text-only qualities such as coherence and grammaticality. human evaluation shows that decsum also leads to better human decision making. ## related work we review additional related work in three areas: query/aspect-based summarization, forecasting with nlp, and evaluation of summarization. our problem formulation is closely related to query-focused summarization @xcite @xcite . in fact, @xcite also uses the term "decision" and provides summaries for each decision made in a meeting. note that relevance in query-focused summarization is still based on textual information, whereas we incorporate potential insights about a decision from a supervised model into the summarization framework. for example, query-focused summarization for pancreatic cancer may summarize all sentences that mention pancreas, but a supervised model may learn that smoking relates to pancreatic cancer and our approach then includes smoking history in the summary. similar to our work, aspect-based summarization uses a predictive model to provide summaries for food, service, decor for reviews @xcite . another related direction is identifying helpful sentences in product reviews @xcite . it is useful to highlight our motivation in support decision making in challenging tasks towards effective human-ai collaboration @xcite @xcite . unlike tasks such as textual entailment where models aim to emulate human intelligence, forecasting future outcomes, such as stock markets @xcite and message popularity @xcite , is challenging both for humans and for machines. humans and machines may offer complementary insights in these tasks. we chose restaurant rating prediction as an example about which laypeople may have valid intuitions. we thus also propose novel desiderata, decision faithfulness and decision representativeness. evaluation of summarization is very challenging, partly because the goal of summarization is usually vague @xcite . popular metrics such as rouge require reference summaries @xcite , but it is unclear that humans can provide useful summaries for decision making in challenging tasks given their limited performance and the scale of inputs. our formulation adopts a task-driven evaluation, i.e., human performance on the decision task which the summaries are supposed to support. this resembles application-based evaluation of explanations in interpretability @xcite . ## conclusion we propose a novel task, decision-focused summarization, and demonstrate that decsum outperforms text-only summarization methods and modelexplanation methods in both automatic metrics and human evaluation. there are many exciting future directions in advancing decision-focused summarization to support human decision making. in particular, our human evaluation demonstrates a substantial gap between human performance and model performance. one possible approach is to leverage visualizations similar to fig. 1 to enable interactive summarization so that users can see the decision variance and explore the textual information beyond a static set of sentences. as humans are final decision makers in a wide variety of highstake scenarios, ranging from healthcare to justice systems, it is critical to investigate human-centered approaches to support human decision making. ethics considerations. our work promotes intelligent models that can be used to support human decision making. we advocate the perspective of augmented intelligence: the goal of our system is to best support humans as final decision makers instead of maximizing model performance. however, in decisions with fairness concerns (e.g., bailing decisions), important future directions include examining fairness-related metrics for the summaries and human-ai interaction.