Too much of product information : Don’t worry, let’s look for evidence! (2023)

Abstract: Product question answering (PQA) aims to provide an instant response to customer questions posted on shopping message boards, social media, brand websites and retail stores. In this paper, we propose a distantly supervised solution to answer customer questions by using product information. Auto-answering questions using product information poses two main challenges:(i) labelled data is not readily available (ii)lengthy product information requires attending to various parts of the text to answer the question. To this end, we first propose a novel distant supervision based NLI model to prepare training data without any manual efforts. To deal with lengthy context, we factorize answer generation into two sub-problems. First, given product information, model extracts evidence spans relevant to question. Then, model leverages evidence spans to generate answer. Further, we propose two novelties in fine-tuning approach: (i) First, we jointly fine-tune model for both the tasks in end-to-end manner and showcase that it outperforms standard multi-task fine-tuning. (ii) Next, we introduce an auxiliary contrastive loss for evidence extraction. We show that combination of these two ideas achieves an absolute improvement of 6% in accuracy (human evaluation) over baselines.

## introduction around the world, customers post millions of questions across digital mediums to obtain important information before completing their purchase journey for a given product. plethora of content on product pages makes it very difficult for customers to discover relevant information which leads to questions. answering customer questions instantly is very crucial for organizations to ensure a seamless buying experience, thereby increasing customer engagement and reducing purchase abandonment possibly due to lack of information. in this paper, * work done during internship at amazon we aim to build a scalable solution to auto-answer customer questions using product pages. auto-answering product questions using product page content poses two main challenges. firstly, labelled data for the task is not available. the existing (question, answer) pairs openly available on product pages are not sufficient since we need to ensure that answers posted are verified and question is answerable using the product content. secondly, description information for products is very lengthy. details of many top selling products span over six to eight thousand words which is equivalent to 15 a4 sheets @xcite . answering questions using lengthy contexts is a difficult task. to tackle the challenges mentioned above, we first propose a distant supervision based natural language inference (nli) model to prepare training data. we leverage nli model to compute relevance between question, answer pair and question and sentences of product content. if product content contains high relevance sentences, we treat the question as answerable and unanswerable otherwise. to deal with the lengthy context, we factorize pqa task into two sub-tasks. task 1 (generatively) extracts evidence span from the context. whereas, task 2 (answer generation) uses both evidence span and context to generate the answer. next, we also propose several novelties in the training procedure. to capture dependency of answer generation on evidence span explicitly during training, we propose a joint end-to-end training of both the tasks. this is in contrast to standard multi-task training where every task is treated independently. further, to improve performance of evidence selection task, we also introduce auxiliary contrastive loss @xcite which helps model distinguish between supporting evidence and irrelevant sentences. we showcase that the combination of end-to-end training along with contrastive objective outperforms other baselines. to the best of our knowledge, this is the first work that deals with long context pqa with distantly supervised data creation. rest of the paper is organized as follows. we review related work in section 2. in section 3, we discuss distant supervision based training data creation approach. section 4 explains the details of evidence extraction and answer generation task. we discuss experiments in section 5 and results in section 6. finally, we conclude paper with a discussion on industry impact in section 7 and conclusion as well as future directions in section 8. ## related work product question answering has gained a lot of attention as a research problem. @xcite provides a comprehensive survey of research work done so far in this space. we can divide the current approaches into two main categories : a) extractive answering b) generative answering. @xcite proposed extracting answers as spans of text from reviews by post training bert on review data. @xcite leverages multiple heterogeneous sources such as reviews and structured attributes to filter snippets of text for answering a question. @xcite proposed distantly supervised extractive approach for pqa. to generate customer friendly responses, @xcite @xcite explored generative approaches. they leverage llms such as t5 @xcite , flan-t5 @xcite and unified-qa @xcite to generate natural language answers to questions. however, most of the existing models have short context window (<2k max token length) which limits their performance in long context scenario. in this work, we aim to combine the power of extractive and generative approaches for pqa for very lengthy product content. ## distant supervision for automated training data creation in this section, we capture the data requirement, challenges, and distant supervision approach to automatically create training data. as stated earlier, the primary focus of this work is to answer questions using only product page content. obtaining training data manually for thousands of product categories is challenging. given the scope, we are faced with following three primary challenges: a) answer-ability: we need to ensure that question in the training data is answerable using product content. b) unavailability of evidence: the first sub-task requires ground truth evidence for training. there is no such dataset available as of today. c) truthfulness: answers posted can be incorrect since they are not moderated. we need to remove untrustworthy answers from dataset for better quality of training data. in subsection 3.2, we describe detailed approach to deal with challenges mentioned above. ## experiments we conduct various experiments to evaluate the proposed approach proposed on following aspects: a) comparison with qa baselines, b) effectiveness of end-to-end fine-tuning and, c) effectiveness of proposed approach in lengthy context. ## results in this section, we discuss the observations made based on experiment results. joint end-to-end fine-tuning improves performance. from using product content along with evidence as input for answer generation improves answer generation performance. we can see from ## industry impact in this paper, we proposed a practical solution for auto-answering product queries using product information which helps customers make quicker purchase decisions. applications of this work have the potential to auto-answer or reply in real-time to thousands of perennially unanswered questions leading to elimination of redundant work and resource savings. ## conclusion in this paper, we proposed distant supervision based approach that combines the power of extractive and generative techniques for product question answering. there are two key contribution of the approach presented in this paper. first, we proposed a distant supervision and nli based technique to create training data without any manual intervention. next, proposed two-stage answer generation approach which achieves 6% point improvement in accuracy over only answer generation approach. further, we also introduce a novel training mechanism which is a combination of two key ideas: a) joint end-to-end fine-tuning b) contrastive loss for evidence extraction. we systematically studied the impact of each component and showed that the combination of ideas proposed above achieves highest performance. in future, we plan to extend this work to incorporate multi-modal input sources such as product reviews, images and videos. we can also leverage rlhf based techniques to achieve better alignment of the model output with human preferences.