Using Large Pretrained Language Models for Answering User Queries from Product Specifications (2020)

Abstract: While buying a product from the e-commerce websites, customers generally have a plethora of questions. From the perspective of both the e-commerce service provider as well as the customers, there must be an effective question answering system to provide immediate answer to the user queries. While certain questions can only be answered after using the product, there are many questions which can be answered from the product specification itself. Our work takes a first step in this direction by finding out the relevant product specifications, that can help answering the user questions. We propose an approach to automatically create a training dataset for this problem. We utilize recently proposed XLNet and BERT architectures for this problem and find that they provide much better performance than the Siamese model, previously applied for this problem. Our model gives a good performance even when trained on one vertical and tested across different verticals.

## background and related work in recent years, e-commerce product question answering (pqa) has received a lot of attention. @xcite present a framework to answer product related questions by retrieving a ranked list of reviews and they employ the positional language model (plm) to create the training data. @xcite apply a multi-task attentive model to identify plausible answers. @xcite propose a siamese deep learning model for answering questions regarding product specifications. the model returns a score for a question and specification pair. mcauley and yang (2016) exploit product reviews for answer prediction via a mixture of expert (moe) model. this moe model makes use of a review relevance function and an answer prediction function. it assumes that a candidate answer set containing the correct answers is available for answer selection. @xcite develop a chatbot for e-commerce sites known as superagent. superagent considers question answer collections, reviews and specifications when answering questions. it selects the best answer from multiple data sources. language representation models like bert @xcite and xlnet @xcite are pre-trained on vast amounts of text and then fine-tuned on task-specific labelled data. the resulting models have achieved state of the art in many natural language processing tasks including question answering. @xcite employ bert to answer binary questions by utilizing customer reviews. in this paper, unlike some of the previous works @xcite on pqa that solely rely on human annotators to annotate the training instances, we propose a semi-supervised method to label training data. we leverage the product specifications to answer user queries by using bert and xlnet. ## problem statement here, we formalize the problem of answering user queries from product specifications. given a question q about a product p and the list of m specifications {s 1 , s 2 , ..., s m } of p , our objective is to identify the specification s i that can help answer q. here, we assume that the question is answerable from specifications. ## model architecture our goal is to train a classifier that takes a question and a specification as input (e.g., "color code black") and predicts whether the specification is relevant to the question. we take siamese architecture @xcite as our baseline method. we fine-tune bert and xlnet for this classification task. siamese: we train a 100-dimensional word2vec embedding on the whole corpus (all questions and specifications as shown in bert and xlnet : the architecture we use for fine-tuning bert and xlnet is the same. we begin with the pre-trained bert base and xlnet base model. to adapt the models for our task, we introduce a fully-connected layer over the final hidden state corresponding to the [cls] input token. during fine-tuning, we optimize the entire model end-to-end, with the additional softmax classifier parameters @xmath0, where h is the dimension of the hidden state vectors and k is the number of classes. ## results and discussion error analysis: we assume that for each question, there is only one correct specification, but the correct answer may span multiple specifications and our models can not provide a full answer. for example, in backpack dataset, the dimension of the backpack, i.e., its height, weight, depth is defined separately. so, when user queries about the dimension, only one specification is provided. some specifications are given in one unit, but users want the answer in another unit, e.g., "what is the width of this bag in cms?". since the specification is given in inches, the models show the answer in inches. so, the answer is related, but not exactly correct. users sometimes want to know the difference between certain specification types, what is meant by some specifications. for example, consider the questions "what is the difference between inverter and non-inverter ac?", "what is meant by water resistant depth?". while we can find the type of inverter, the water resistant depth of a watch etc. from specifications, the definition of the specification is not given. as we have generated train data labels in semi-supervised fashion, it also contributes to inaccurate classification in some cases. ## conclusion and future work in this paper, we proposed a method to label training data with little supervision. we demonstrated that large pretrained language models such as bert and xlnet can be fine-tuned successfully to obtain product specifications that can help answer user queries. we also achieve reasonably good results even while testing on different verticals. we would like to extend our method to take into account multiple specifications as an answer. we also plan to develop a classifier to identify which questions can not be answered from the specifications.