Product Answer Generation from Heterogeneous Sources: A New Benchmark and Best Practices (2022)

Abstract: It is of great value to answer product questions based on heterogeneous information sources available on web product pages, e.g., semi-structured attributes, text descriptions, user-provided contents, etc. However, these sources have different structures and writing styles, which poses challenges for (1) evidence ranking, (2) source selection, and (3) answer generation. In this paper, we build a benchmark with annotations for both evidence selection and answer generation covering 6 information sources. Based on this benchmark, we conduct a comprehensive study and present a set of best practices. We show that all sources are important and contribute to answering questions. Handling all sources within one single model can produce comparable confidence scores across sources and combining multiple sources for training always helps, even for sources with totally different structures. We further propose a novel data augmentation method to iteratively create training samples for answer generation, which achieves close-to-human performance with only a few thousandannotations. Finally, we perform an in-depth error analysis of model predictions and highlight the challenges for future research.

## introduction automatic answer generation for product-related questions is a hot topic in e-commerce applications. previous approaches have leveraged information from sources like product specifications @xcite , descriptions @xcite or user reviews @xcite @xcite to answer product questions. however, these works produce answers from only a single source. while a few works have utilized information from multiple sources @xcite @xcite , they lack a reliable benchmark and have to resort to noisy labels or small-scaled human evaluation @xcite . furthermore, almost none of them make use of pretrained transformer-based models, which are the current state-of-the-art (sota) across nlp tasks @xcite . in this work, we present a large-scale benchmark dataset for answering product questions from 6 heterogeneous sources and study best practices to overcome three major challenges: (1) evidence ranking, which finds most relevant information from each of the heterogeneous sources; (2) source selection, which chooses the most appropriate data source to answer each question; @xcite (3) answer generation, which produces a fluent, natural-sounding answer based on the relevant information. it is necessary since the selected relevant information may not be written to naturally answer a question, and therefore not suitable for a conversational setting. most published research on product question answering is based on the amazonqa dataset @xcite , which takes the community question-answers (cqas) as the ground truth. this leads to several problems. (1) cqas, even the top-voted ones, are quite noisy. many are generic answers or irrelevant jokes @xcite . (2) cqas are based more on the opinion of the individual customer who wrote the answer rather than on accompanying sources such as product reviews and descriptions. as such, cqas are not reliable references for judging the quality of answers generated from these sources @xcite . (3) there are no annotations for assessing the relevance of the information across multiple data sources. this makes it difficult to evaluate the evidence ranker and generator separately. some works collect annotations for evidence relevance, but only for a single source and with questions formulated post-hoc rather than naturally posed @xcite . to address these shortcomings, we collect a benchmark dataset with the following features: (1) it provides clear annotations for both evidence ranking and answer generation, enabling us to perform in-depth evaluation of these two components separately. (2) we consider a mix of 6 heterogeneous sources, ranging from semi-structured specifications (jsons) to free sentences and (3) it represents naturally-occurring questions, unlike previous collections that elicited questions by showing answers explicitly. as sources differ in their volume and contents, collecting training data covering all sources of natural questions and answers is challenging. to get enough positive training signals for each source, we propose filtering community questions based on the model score of a pretrained qa ranker. questions are only passed for annotation when the confidence scores of top-1 evidence lie within some certain range. this greatly reduces annotation effort by removing most unanswerable questions. after collecting the data, we apply sota transformer-based models for evidence ranking and answer generation, and present a set of data augmentation and domain adaptation techniques to improve the performance. we show that pretraining the model on the amazonqa corpus can provide a better initialization and improve the ranker significantly. for evidence ranking, we apply question generation with consistency filtering @xcite to obtain large amounts of synthetic qa pairs from unannotated product sources. for answer generation, we propose a novel data augmentation algorithm that creates training examples iteratively. by first training on this augmented data and then finetuning on the human annotations, the model performance can be further enhanced. as for the model design, we homogenize all sources by reducing them to the same form of input which is fed into a unified pretrained transformer model, similarly to many recent works of leveraging a unified system for various input formats @xcite @xcite . we show that combining all sources within a single framework outperforms handling individual sources separately and that training signals from different answer sources can benefit each other, even for sources with totally different structures. we also show that the unified approach is able to produce comparable scores across different sources which allows for simply using the model prediction score for data source selection, an approach that outperforms more complex cascadebased selection strategies. the resulting system is able to find the correct evidence for 69% of the question: how much weight will it safely hold? source supporting evidence relevance attribute item_weight:{unit: pounds,value:2.2} ✖ bullet point supports up to 115 pounds ✔ description weight limit: 115 lbs. ✔ osp if you're looking for an inexpensive way to change up ... ✖ cqa we put ours on a swingset. ✖ review it is sturdy and well made. ✖ annotated answer: it can support up to 115 pounds. ## benchmark test set collection we begin by explaining how we collect a benchmark test set for this problem. the benchmark collection is performed in 4 phases: question sourcing, supporting evidence collection, relevance annotation, and answer elicitation. an annotation example is shown in ## training data collection for training data collection, a complete annotation of each set of evidence is not necessary; we need only a rich set of contrastive examples. therefore, we propose to select questions for annotation based on the confidence score of a pretrained ranker (the same ranker we used to select top evidence for cqa and review). we sample 50k community questions about products in the same domain as the testset. we first select questions whose top-1 item of supporting evidence returned by the pretrained ranker has a prediction score @xmath0.8. in this way the selected questions have a good chance of being answerable from the available evidence and the approach should also yield enough positive samples from all sources to train the ranker. this selection step is crucial to ensure coverage of lowresource sources, like osp, which otherwise might have zero positive samples. to avoid a selection process that is biased towards easy questions we further include questions whose top-1 evidence has a score within the range of 0.4∼0.6. intuitively these questions will pose more of a challenge in ranking the evidence and their annotation should provide an informative signal. from each out of the 6 sources, we sample 500 questions with prediction @xmath1.8 and another 500 questions with scores in the range of 0.4∼0.6. for each question, we then annotate the top-5 (if available) evidence items returned by the pretrained ranker. this reduces annotation cost relative to the complete annotation that was done for the test set. the final dataset contains 6000 questions with 27,026 annotated question-evidence pairs being annotated, 6,667 of which were positive. we then submit the positive question-evidence pairs for answer elicitation. after filtering invalid annotations as was done for the benchmark collection, we obtain a set of 4,243 question-evidence-answer triples to train the answer generator. for both evidence ranking and answer generation, we split the collected data by 9:1 for train/validation. ## error analysis based on the human evaluation, we identified the following key problems that exist in the current system. for evidence ranking, the major problems are: the model tends to select strings similar to the ques-tion while ignoring their fine semantics, a common problem from the bias to 'shortcut learning' of neural networks @xcite . (4) uncertain evidence: the model ranks evidence highly, even if this evidence is an uncertain expression. this can be viewed as a special case of over-fitting to string match. we show examples in error type 3 is more challenging. one possible solution is to automatically detect spurious correlations and focus the model on minor examples @xcite . nevertheless, a fundamental solution to fully avoid error 3 is still an open question. for answer generation, we identify the major problems as: (1) number accuracy: the model cannot fully understand the roles of numbers from the limited training examples. (2) hallucination if inference is needed: when it is not possible to generate an answer by simple rephrasing, the model can hallucinate false information. (3) sensitivity to typos: the model is not robust to typos in the question. a tiny typo can easily break the system. we provide examples of these errors in ## conclusion to the best of our knowledge, this work is the first comprehensive study of product answer generation from heterogeneous sources including both semistructured attributes and unstructured text. we collect a benchmark dataset with annotations for both evidence ranking and answer generation. it will be released to benefit relevant study. we find that the best practice is to leverage a unified approach to handle all sources of evidence together and further experimented with a set of data augmentation techniques to improve the model performance. error analysis is provided to illustrate common errors, which we hope will lead to inspire future work.