Distantly Supervised Transformers For E -Commerce Product QA (2021)

Abstract: We propose a practical instant question answering (QA) system on product pages of e-commerce services, where for each user query, relevant community question answer (CQA) pairs are retrieved. User queries and CQA pairs differ significantly in language characteristics making relevance learning difficult. Our proposed transformer-based model learns a robust relevance function by jointly learning unified syntactic and semantic representations without the need for human labeled data. This is achieved by distantly supervising our model by distilling from predictions of a syntactic matching system on user queries and simultaneously training with CQA pairs. Training with CQA pairs helps our model learning semantic QA relevance and distant supervision enables learning of syntactic features as well as the nuances of user querying language. Additionally, our model encodes queries and candidate responses independently allowing offline candidate embedding generation thereby minimizing the need for real-time transformer model execution. Consequently, our framework is able to scale to large e-commerce QA traffic. Extensive evaluation on user queries shows that our framework significantly outperforms both syntactic and semantic baselines in offline as well as large scale online A/B setups of a popular e-commerce service.

## introduction product pages on an e-commerce service (eg. amazon) are often overloaded with information. customers wanting to search for a piece of specific information about a product find it difficult to sift * this work was done while author was in community shopping team. in this paper, we present an instant qa system with two main contributions: (1) our framework is able to learn a robust relevance function between user queries and cqa pairs by jointly learning semantic and syntactic features-aware representations without the need for explicit human-labeled data, and (2) our framework minimizes the need for realtime model execution by encoding the cqa pairs offline, enabling large scale online deployment. we chose bert @xcite as our transformer encoder due to its recent success in various natural language understanding (nlu) tasks including qa. to address the lack of labeled training data challenge, we use the qa pairs from the cqa section of each product page as training data. however, as shown in our evaluation (section 4.3), such a model does not work well on the user queries asked on the instant qa system on the product pages. we propose a distillation-based distantly supervised training algorithm where we use the answers retrieved by a syntactic match system on a set of user queries asked on the instant qa system. this training helps the model adapt to the specific task at hand by learning the user query distribution as well as the strengths of a traditional syntactic match system. this coupled with training on cqa pairs helps our model learn a robust semantic model that is task aware. our training data does not require any explicit human labeling. to make our system work in real-time we train the bert model in siamese style @xcite with triplets consisting of query, relevant candidate (+ve sample), and irrelevant candidate (-ve sample). hence the query and candidate responses are encoded independently using the same transformer encoder enabling embedding computation of all candidates (across all products) offline. at real-time, only the user query needs to be embedded using the heavy semantic model resulting in a significant reduction of online compute cost. in contrast, the common practice of using bert in qa problems is to concatenate the query and a candidate response and run bert on the fused input. this would require bert to run on all query, candidate cqa pairs on product pages real-time making it prohibitive for online deployment. additionally, we combine the two embeddings (question and answer) in each cqa pair to form one embedding per pair allowing us to reduce the offline storage significantly. we extensively evaluate our framework on user queries asked on the instant qa system at a popular e-commerce system in 4 locales spanning 3 languages. offline evaluation shows that our proposed framework is able to increase the area under the precision-recall curve (pr-auc) by up to 12.15% over the existing system. also in an online a/b test, our system is able to improve coverage by up to 6.92% by complementing the existing system. in this paper, we take the approach of answer retrieval (instead of generation). we solve the orthogonal problem of how to adapt the relevance model to be aware of the user query characteristics (significantly different from the well formed questions posted in the cqa section) in the absence of human labeled data. the improvement in relevance models (between user queries and cqa pairs) proposed can be easily complemented with the existing review awareness models. a drawback of the aforementioned models is they comprise of multiple deep neural components, many of which need to be run real-time making online model deployment and computation cost prohibitive for large scale deployment. our framework only needs to encode the user query realtime, all candidate responses are precomputed stored in an index making it amenable to real-time deployment. ## semantic qa system in this section, we describe our proposed semantic qa system for e-commerce services. unlike tradi-tional qa systems where multiple models are used sequentially to surface the final response (eg. candidate retrieval, followed by answer selection/reranking, followed by span selection), here we use a semantic index and the top results retrieved from the index are the final answers shown to the users. below we describe the problem definition followed by individual components of our system: ## experiments we ran experiments both in offline settings as well as in large scale online setups. we evaluated our models across 4 locales with 3 languages to test whether our distant supervision based training approach is able to generalize across languages and varying data characteristics. ## conclusions & future works in this paper we presented 'semqa', a practical transformer-based framework to provide instant qa efficiently on the product pages of e-commerce services. given a user query, our framework directly retrieves the relevant cqa pairs from the product page, where user queries and cqa pairs have significantly different language characteristics. our model is able to learn a robust relevance function between user queries and cqa pairs by learning representations that leverage the strengths of both syntactic and semantic features, without the need for any explicit human labeled data. our model is able to scale to large scale real-time ecommerce systems and at inference time only requires model encoding of user queries for by index lookups, and candidate responses are encoded offline into the index in a space efficient manner. extensive offline evaluation shows our approach generalizes to multiple locales spanning different languages with a pr-auc gain by upto 12.15% over the existing system at a popular e-commerce service. we also ran a large scale online a/b experiment with 50% of the user traffic and our framework was able to improve coverage by upto 6.92% by complementing the existing system. as a future direction, we would like to expand our semqa system to include responses from additional content on the product pages (reviews, descriptions etc.). we believe some of the existing approaches to leverage reviews (discussed in section 2) can be used to complement our system to expand our relevance model beyond cqa data. another direction of research will be to include features such as accuracy, sentiment, freshness etc. within our proposed semqa system's responses.