Leveraging Customer Feedback for Multi-modal Insight Extraction (2024)

Abstract: Businesses can benefit from customer feedback in different modalities, such as text and images, to enhance their products and services. However, it is difficult to extract actionable and relevant pairs of text segments and images from customer feedback in a single pass. In this paper, we propose a novel multi-modal method that fuses image and text information in a latent space and decodes it to extract the relevant feedback segments using an image-text grounded text decoder. We also introduce a weakly-supervised data generation technique that produces training data for this task. We evaluate our model on unseen data and demonstrate that it can effectively mine actionable insights from multi-modal customer feedback, outperforming the existing baselines by 14 points in F1 score.

## introduction customer feedback is essential for businesses to design and improve their products and services, according to customer expectations. @xcite observe that the multi-modal feedback is growing rapidly. however, most existing solutions @xcite @xcite do not take into account the rich information that image and video feedback contain, which can enhance the actionability and improve the customer experience. to address this challenge, we propose a novel multi-modal architecture that extracts pairs of text segments and corresponding images that are relevant and actionable for a given product from customer feedback. these pairs can help businesses to increase the actionability, improve the product catalogue quality, enhance the customer experience and thereby reduce returns and replacements. ## related work in recent years, multi-modal tasks, combining various data types such as images and text, have garnered significant interest in artificial intelligence and natural language processing @xcite @xcite . among these tasks, visual question answering (vqa) stands out as a prominent domain @xcite , wherein the aim is to generate textual answers to questions based on images. vqa has evolved significantly over the years, thanks to various advancing contributions. early works on vqa @xcite laid the groundwork, delineating the fundamental framework and challenges of the task. subsequent research delved into innovative methodologies, such as leveraging transformer networks @xcite , and devising techniques for seamlessly integrating visual and textual information @xcite @xcite . the field progressed with more advanced models and datasets, such as vqav2 @xcite , which improved the robustness and performance benchmarking of vqa models. moreover, the strides in pre-training on both vision and language have significantly influenced the landscape, as evidenced by groundbreaking models like oscar @xcite , beit-3 @xcite , and vlmo @xcite , which have achieved remarkable results. additionally, techniques such as counterfactual data augmentation @xcite have further enhanced the performance of vqa models. meanwhile, in the nlp domain, verbatim extraction and text summarization tasks have attracted a lot of attention. models like bert @xcite and @xcite have shown remarkable abilities in extracting and summarising textual feedback. abstractive text summarization models, such as the t5 @xcite , have emerged as the state-of-the-art (sota) solutions for generating short and coherent summaries from longer texts. our work falls at the intersection of these two domains, where we propose a novel problem at the intersection of vqa and text summarization, where the input is a customer feedback text and corresponding images, and the objective is to extract the verbatim (exact segments of the textual feedback) that best highlights or talks about the feedback image. we design a multi-modal approach that builds on vqa models and extends them with text generation capabilities, introducing a new technique for multi-modal insight extraction. our work bridges the gap between multi-modal understanding and verbatim extraction, and enhances the interpretability of images within textual context, especially for customer feedback and images. ## problem statement given a customer feedback consisting of a set of images @xmath0, i 2 , ...i n } and a text t , we extract m verbatims from t , denoted by the aim is to extract all the relevant and actionable verbatims {v @xmath1. ## proposed approach in this section, we first present our data generation method that creates training data from raw feedback. then, we introduce our model architecture that extracts pairs of verbatims and images that are relevant and actionable for a given product. ## conclusion in this paper, we introduced a novel architecture mine, for extracting insights from multi-modal customer feedback encompassing both text and images. we proposed a weakly-supervised data generation approach that leveraged raw feedback text and images to create relevant verbatim and image pairs. we also proposed an unsupervised insight extraction approach that used an image-grounded text encoder to learn latent representations of the feedback text and corresponding images, and an image-text grounded text decoder to extract actionable verbatim relevant to image. we evaluated our approach on real-world dataset and demonstrated its effectiveness in extracting actionable insights from multi-modal feedback with minimal supervision, achieving a 14-point improvement in f1-score over the existing baselines and uni-modal insight extraction. future work: our research opens up several avenues for further exploration. as a future direction, we would like to extend our method to video feedback by incorporating the temporal dimension and extracting relevant video snippets, as mention in the feedback text. we also aim to structure the extracted insights in hierarchical way that facilitates decision making for users. moreover, we plan to localise the regions of interest in the feedback images that correspond to the verbatim, using bounding boxes, to help identify the specific features or issues that customers refer to in their feedback. finally, we intend to generate suggestions for product enhancements or fixes based on the extracted insights and the customer sentiment.