Investigating the Generative Approach for Question Answering in E -Commerce (2022)

Abstract: Many e-commerce websites provide Product-related Question Answering (PQA) platform where potential customers can ask questions related to a product, and other consumers can post an answer to that question based on their experience. Recently, there has been a growing interest in providing automated responses to product questions. In this paper, we investigate the suitability of the generative approach for PQA. We use state-of-the-art generative models proposed by Deng et al.(2020) and Lu et al.(2020) for this purpose. On closer examination, we find several drawbacks in this approach: (1) input reviews are not always utilized significantly for answer generation, (2) the performance of the models is abysmal while answering the numerical questions, (3) many of the generated answers contain phrases like “I do not know” which are taken from the reference answer in training data, and these answers do not convey any information to the customer. Although these approaches achieve a high ROUGE score, it does not reflect upon these shortcomings of the generated answers. We hope that our analysis will lead to more rigorous PQA approaches, and future research will focus on addressing these shortcomings in PQA.

## introduction with the increase in e-commerce shopping, customer-generated product queries are also growing. manually answering the questions in real-time is infeasible, and also some questions go unanswered for an extended period. it is necessary to answer the user queries in the e-commerce business automatically. the user reviews are a vast source of information with diverse opinions, and they can be used to answer user queries. earlier works on product question answering (pqa) focus on retrieval-based approaches and binary answer prediction tasks. @xcite ; @xcite ; @xcite aim to predict the answer as "yes/no" based on the relevant reviews, customer ratings, aspects in the reviews, etc. retrieval-based approaches try to find the most relevant review snippet as the answer @xcite and use a ranked list of review snippets as the response for a given question @xcite . with the success of machine translation @xcite and summarization @xcite , the pqa approaches are shifting towards natural answer generation from relevant product reviews @xcite @xcite @xcite . in this work, we analyse the answer generated from state-of-the-art generative models oaag @xcite and chime @xcite in detail beyond their traditional scores on popular metrics such as rouge @xcite . we find that despite achieving a good score on these metrics, generated answers have several drawbacks that can lead to user dissatisfaction. ## research questions we empirically analyse the oaag model with dynamic fusion and chime model to answer the following research questions: rq1 : are the retrieved review snippets significantly utilized for generating the answers? rq2 : is the model performing similarly for a heterogeneous group of questions? rq3 : is the generative model biased towards more frequently occurring phrases? rq4 : can rouge capture the correctness of generated answers? ## experiments we use two product categories, namely, home&kitchen and sports&outdoors for our analysis from the dataset mentioned in section 2.1 after combining the question-answer and review dataset with the product ids. we will denote the two categories as home and sports, respectively. we use the same data split from oaag foot_0 to retrain the models. since there is no validation dataset, we take the 10% of the train data as validation data. table a .1 in the appendix shows the details of training, validation, and test split. we keep all the hyper-parameters the same as the oaag and chime. we train all the oaag models for 20 epochs and chime models for 3 epochs, and the model that performs the best on the validation set is used to evaluate the test set. we evaluate the model with rouge metric and report the f1 scores for rouge-1 (r1) and rouge-l (rl), which measure the word overlap and the longest common sequence between the reference answer and the generated answer, respectively. we obtain the rouge scores using rouge-score foot_1 package. ## conclusion in this paper, we extensively analyze the generative approach of question-answering in e-commerce using a state-of-the-art oaag model @xcite and chime model @xcite . we find many shortcomings which need to be addressed for a reliable pqa system. we try to address four re-search questions related to the generative approach for pqa, such as how the models utilize the reviews, how it performs on diverse question types, whether it is biased toward frequent phrases in training data, and the correctness of the generated response. we hope that our analysis will lead to more rigorous pqa research.