Knowledge Distillation based Contextual Relevance Matching for E -commerce Product Search (2022)

Abstract: Online relevance matching is an essential task of e-commerce product search to boost the utility of search engines and ensure a smooth user experience. Previous work adopts either classical relevance matching models or Transformer-style models to address it. However, they ignore the inherent bipartite graph structures that are ubiquitous in e-commerce product search logs and are too inefficient to deploy online. In this paper, we design an efficient knowledge distillation framework for e-commerce relevance matching to integrate the respective advantages of Transformer-style models and classical relevance matching models. Especially for the core student model of the framework, we propose a novel method using k-order relevance modeling. The experimental results on large-scale real-world data (the size is 6 174 million) show that the proposed method significantly improves the prediction accuracy in terms of human relevance judgment. We deploy our method to JD.com online search platform. The A/B testing results show that our method significantly improves most business metrics under price sort mode and default sort mode.

## introduction relevance matching @xcite @xcite is an important task in the field of ad-hoc information retrieval @xcite , which aims to return a sequence of information resources related to a user query @xcite @xcite . generally, texts are the dominant form of user queries and returned information resources. given two sentences, the target of relevance matching is to estimate their relevance score and then judge whether they are relevant or not. however, text similarity does not mean semantic similarity. for example, while "mac pro 1.7ghz" and "mac * chaokun wang is the corresponding author. pro (16-inch, 16gb ram, 1tb storage, 2.3ghz intel core i9) i 2 apple macbook pro mlh42ll/a 13.3inch laptop i 3 little mac lipstick 0.06 oz/ 1.77 ml whirl i 4 apple mac mini with apple m1 chip (8gb ram, 256gb ssd storage) view but not click positives negatives q 1 --i 2 q 1 --i 3 q 3 --i 2 arc-i model: using click behaviors as training labels lipstick 1.7ml" look alike, they describe two different and irrelevant products. therefore, relevance matching is important, especially for long-term user satisfaction of e-commerce search @xcite @xcite . recently, transformer-style models (e.g., bert @xcite and ernie @xcite ) have achieved breakthroughs on many nlp tasks and shown satisfactory performance on relevance matching, but they are hard to deploy to the online environment due to their high time complexity. moreover, these methods cannot deal with the abundant context information (i.e., the neighbor features in a query-item bipartite graph) in e-commerce product search. last but not least, when applied to real-world scenarios, existing classical relevance matching models directly use user behaviors as labeling information (figure 1 ). however, this solution is not directly suitable for relevance matching because user behaviors are often noisy and deviate from relevance signals @xcite . in this paper, we propose to incorporate bipartite graph embedding into the knowledge distillation framework @xcite @xcite @xcite to solve the relevance matching problem in the scene of e-commerce product search. we adopt bert @xcite as the teacher model in this framework. also, we design a novel model called berm, bipartite graph embedding for relevance matching (berm), which acts as the student model in our knowledge distillation framework. this model captures the 0-order relevance using a word interaction matrix attached with positional encoding and captures the higherorder relevance using the metapath embedding with graph attention scores. for online deployment, it is further distilled into a tiny model berm-o. our main contributions are as follows: ## experiments in this section, we present the offline and online experimental results of berm * . ## conclusions and future work in this paper, we propose the new problem of contextual relevance matching in e-commerce product search. different from the previous work only using the 0-order relevance modeling, we propose a novel method of the k-order relevance modeling, i.e., employing bipartite graph embedding to exploit the potential context information in the queryitem bipartite graph. compared to the state-of-theart relevance matching methods, the new method berm performs robustly in the experiments. we further distill berm into berm-o and deploy berm-o to jd.com online e-commerce product search platform. the results of a/b testing indicate that berm-o improves the user's search experience significantly. in the future, we plan to apply our method to other e-commerce applications such as recommender systems and advertisements.