CUPID : Curriculum Learning Based Real-Time Prediction using Distillation (2023)

Abstract: Relevance in E-commerce Product Search is crucial for providing customers with accurate results that match their query intent. With recent advancements in NLP and Deep Learning, Transformers have become the default choice for relevance classification tasks. In such a setting, the relevance model uses query text and product title as input features, and estimates if the product is relevant for the customer query. While cross-attention in Transformers enables a more accurate relevance prediction in such a setting, its high evaluation latency makes it unsuitable for real-time predictions in which thousands of products must be evaluated against a user query within few milliseconds. To address this issue, we propose CUPID: a Curriculum learning based real-time Prediction using Distillation that utilizes knowledge distillation within a curriculum learning setting to learn a simpler architecture that can be evaluated within low latency budgets. In a bi-lingual relevance prediction task, our approach shows an 302 bps improvement on English and 676 bps improvement for low-resource Arabic, while maintaining the low evaluation latency on CPUs.

## introduction large-scale e-commerce search systems, such as those used by companies like amazon, walmart etc., typically employ a multi-step process to retrieve relevant products for a given query @xcite . the first step in this process is to generate a matchset that is approximately relevant to the query, followed by a series of steps that optimize for relevance, customer interest and other associated metrics @xcite . in such a setting, it is imperative to have features that accurately capture relevance between the customer's query-intent and the candidate set of products in the matchset. recently, transformer-based models such as bert have proven to be highly effective in es-timating relevance between a query and a product by using cross-attention @xcite @xcite , self-attention (dual-encoders) @xcite @xcite , or late interaction @xcite @xcite models. the use of cross-attention in transformers has been shown to be effective, as it allows the model to take into account both the query and the product when determining relevance @xcite . however, these models come at a cost, as they require heavy computational resources and have a high latency even during evaluation. in large-scale search systems, it is important to perform real-time relevance predictions, as thousands of products need to be processed for each query, with strict latency requirements. for this reason, dual-encoder models are more suitable, as they can provide real-time relevance predictions with low latency requirements. in such a setting, the task of estimating relevance is reduced to carrying out simple vector operations, typically a dot product of high-dimensional vectors, one representing the query and the others representing products. under such a setting, the vectors representing products are pre-computed and cached, while those for the query are computed on-the-fly. such on-thefly computation of query vectors or embeddings in low latency settings restricts us from using a full stack of transformer layers, as is typical to models like bert. to address the issue of latency in computation of query embeddings, we instead borrow from the architecture used in @xcite to be used for the query arm for our asymmetric dual-encoders, while continuing to use a full stack of transformers for the product arm (as shown in figure 1 ). their simple architecture is comprised of a word embedding layer and a mean-pool layer (based on @xcite , referred loosely as dssm (deep structured semantic model) henceforth), which is more suitable for real-time scenarios with low latency requirements. however, this model lacks the rich semantic representation of models built purely of transformers. to bridge this gap, we leverage knowledge distillation techniques, where we use the dssm of the query arm as a student model to learn the rich semantic representation from the transformer model. to address this issue, we propose cupid: a curriculum learning based real-time prediction using distillation that utilizes knowledge distillation within a curriculum learning setting to learn a simpler architecture that can be evaluated within low latency budgets. our contributions to the literature can be summarized as follows: bi-encoders @xcite , on the other hand, use separate encoders for each input sentence. each sentence is encoded independently, producing two separate representations. these representations are then compared using a similarity function (e.g., dot product, cosine similarity) to determine the relationship between the sentences. both cross encoders and bi-encoders have their advantages and are suitable for different scenarios. cross encoders excel at capturing the interaction between sentences, while bi-encoders are computationally efficient. the choice between the two architectures depends on the specific requirements. for real-time predictions, cross encoders are often infeasible, but bi-encoders excel due to the ability to utilize pre-computed indexes. low latency transformers in recent years, with the state-of-the-art performance of transformers in nlp applications, there has been a demand to make transformers suitable for real-time e-commerce applications. and the reduction in computation time and latency is crucial for transformers to be viable for such use cases. there are three main themes @xcite into which the advances in the design of low-latency transformers can be categorised. (i) architectural level, (ii) component level, and (iii) technique-based. at the architectural level (i), modifications are made at a higher level, such as the use of lightweight transformers like funnel transformer @xcite , lite transformer @xcite , and delight @xcite . additionally, pruning techniques reported in @xcite @xcite aim to reduce the size and computation by eliminating unimportant weights. finally, quantization-based approaches @xcite and model compression @xcite are used to compress weights and activations. at the component level (ii), there is a focus on efficient self-attention, such as in the works of @xcite , and delayed interaction networks @xcite @xcite . lastly, under the category of technique-based (iii), research efforts have been made in areas such as early exit @xcite @xcite and knowledge distillation @xcite . knowledge distillation knowledge distillation (kd) @xcite is a widely researched topic that enables the transfer of knowledge from a complex, pre-trained model to a smaller, more computationally efficient model. with the rise of bert @xcite and the corresponding growth in textual data, there has been growing interest in applying kd to bert models in the field of nlp. distill bert @xcite is one such seminal work along with (tinybert @xcite , fast bert @xcite , task specific bert @xcite , patient knowledge distillation bert @xcite . these propose using kd to transfer knowledge from a large bert model to a smaller model. while the distill bert makes bert models faster by 60%, they cannot be used in real-time because e-commerce applications such as semantic matching @xcite have limited computational resources (especially gpus) and strict latency requirements. the latest twinbert @xcite proposes the distillation of 12-layer bert to 6-layer twin tower bert structure, thus, permitting pre-compute of document embeddings and cache in memory saving additional computational time. however, the twinbert requires gpus during inference to compute the query arm embeddings within the latency budgets. ## our approach in this section, we present our proposed approach cupid that uses bi-encoder transformer models as teachers, and learn asymmetric student models having dssm architecture in the query side and transformer architecture in the product side. using cross-encoders as teachers is natural, however, in this work we show that using bi-encoder teacher yields better performance and allows better transferring of knowledge to bi-encoder student models (refer to section 4.3). bi-encoder teacher model enables better transfer of semantics to a simpler bi-encoder student model with better alignment of query embeddings. later in the section, we present a very simple curriculum learning framework for training bi-encoder student model by progressively increasing the difficulty of task. we present the detailed architecture (also shown in figure 1 ) below. ## experiments and results in this section, we compare the performance of cupid with the state of the art methods of knowledge distillation. we also study the latency of the student models and compare them with the latency of teacher model to show why bert based models are not suitable for real time predictions. we then present results for a real world application of cupid on an internal dataset. finally we show the effect each of the losses have on the performance of the models, and explain the choice of a bi-encoder teacher model compared to a better performing cross-encoder model. ## conclusion in this paper, we propose cupid, a novel approach for knowledge distillation for real-time prediction of relevancy using curriculum learning. it uses a new loss function and two-stage curriculum learning framework to increase the influence of teacher gradually, resulting in a loss function that outperforms imitation learning based kd methods by up to 300 basis points.