Improving Factual Consistency of Abstractive Summarization on Customer Feedback (2021)

Abstract: E-commerce stores collect customer feedback to let sellers learn about customer concerns and enhance customer order experience. Because customer feedback often contains redundant information, a concise summary of the feedback can be generated to help sellers better understand the issues causing customer dissatisfaction. Previous state-of-the-art abstractive text summarization models make two major types of factual errors when producing summaries from customer feedback, which are wrong entity detection (WED) and incorrect product-defect description (IPD). In this work, we introduce a set of methods to enhance the factual consistency of abstractive summarization on customer feedback. We augment the training data with artificially corrupted summaries, and use them as counterparts of the target summaries. We add a contrastive loss term into the training objective so that the model learns to avoid certain factual errors. Evaluation results show that a large portion of WED and IPD errors are alleviated for BART and T5. Furthermore, our approaches do not depend on the structure of the summarization model and thus are generalizable to any abstractive summarization systems.

## introduction in order to improve customer order experience, most e-commerce stores allow customers to submit reviews or feedback via their post-order communication channels. such customer feedback, usually in the form of short paragraphs of free texts, contains information reflecting the issues that customers experienced in their purchases. this information can be shared with sellers to bring their awareness on the problems in their products. however, customer feedback often include other contents that are irrelevant to the product issues. such redundant information requires extra efforts for ## proposed approaches our error analysis of customer-feedback summarizaton showed that most of the factual errors belong to two error types: wed and ipd. hence, in our proposed approaches, we first apply rule-based transformations and introduce synthetic factual errors of the two error patterns into the target summaries. we then modify the training objective by source: (...) i've bought cheese from this store for many times, and they were very good. so i think other products must be good too. then i ordered several bottles of milk. but they are clearly expired (. . . ) reference summary: milk delivered is expired. corrupted summary: cheese delivered is expired. source: (. . . ) the eggs i purchased have bad smells. they don't look like fresh eggs. (. . . ) reference summary: eggs have bad smells, and don't look like fresh eggs. corrupted summary: eggs have good smells, and don't look like fresh eggs. ## conclusion in conclusion, we study the error patterns in the customer-feedback summaries generated by bart and t5. we propose to augment the training data with artificially corrupted summaries and use contrastive learning methods to enhance the model faithfulness. human analysis shows that significant portion of wed and ipd errors from bart and t5 are reduced. because our methods do not involve modifying the model structure, they can also be applied to other abstractive summarization frameworks.