Cross-Encoder Data Annotation for Bi-Encoder Based Product Matching (2022)

Abstract: Matching a seller listed item to an appropriate product is an important step for an e-commerce platform. With the recent advancement in deep learning, there are different encoder based approaches being proposed as solution. When textual data for two products are available, cross-encoder approaches encode them jointly while bi-encoder approaches encode them separately. Since cross-encoders are computationally heavy, approaches based on bi-encoders are a common practice for this challenge. In this paper, we propose cross-encoder data annotation; a technique to annotate or refine human annotated training data for bi-encoder models using a cross-encoder model. This technique enables us to build a robust model without annotation on newly collected training data or further improve model performance on annotated training data. We evaluate the cross-encoder data annotation on the product matching task using a real-world e-commerce dataset containing 104 million products. Experimental results show that the cross-encoder data annotation improves 4% absolute accuracy when no annotation for training data is available, and 2% absolute accuracy when annotation for training data is available.

## introduction product matching refers to the task of determining whether two different entries in the product catalog refer to the same real-world product. it is a core task for an e-commerce company where product catalog entries come from different sources and duplicates need to be identified and managed. there are two popular types of deep learning models that had been applied to the recent product matching work; cross-encoder @xcite and bi-encoder @xcite . figure 1 depicts the architectures to demonstrate the difference between two models. when you have textual data for two products such as title pairs, the cross-encoder encodes them jointly, and the interaction between the two occurs through all encoder layers. the bi-encoder encodes both of them separately and there is no interaction between them until computing similarity. the cross-encoder can capture more context due to its more complex interaction between the two inputs of data. however, because of its complexity, the model requires much more computational resources and time. a previous work @xcite reported that it takes 65 hours to find the most similar pair from a collection of 10,000 sentences using the cross-encoder (which requires us building 100,000,000 pairs for crossencoder to process), while it only takes 5 seconds to encode all 10,000 sentences and compute cosine similarity for every possible pairs using the bi-encoder; that is 46,800 times difference in processing time. since an e-commerce platform could easily have millions of products in its product catalog, such a volume will be a great challenge to use an approach that requires lots of computing resources, such as the cross-encoder. as a result, the bi-encoder is more viable for product matching in actual production systems. in this paper, we propose cross-encoder data annotation, which is a technique that benefits from both model architectures. we first apply a crossencoder on training data for a bi-encoder, and then train a bi-encoder using the data with high prediction scores from the cross-encoder. this process is a way to utilize the knowledge learned in the cross-encoder to support the bi-encoder training. this approach is inspired by knowledge distillation @xcite , where information learned by the cross-encoder is passed to the biencoder through the loss function. this can also be considered as a semi-supervised approach for product matching, where the new data is unlabeled. in this paper, we make three major contributions. first, we demonstrate that our cross-encoder data annotation is effective for the product matching task on a real-world e-commerce dataset. selecting the subset that both cross-encoder prediction and human annotation consider positive can train a better performance model in comparison to baseline. this can happen in an e-commerce company where different existing trained models are available for research purposes and new data are coming in. when building a model with the human annotated new data, our approach is applicable. second, we show that even when the human annotation of the bi-encoder training data is not available, we can use the prediction result from the cross-encoder as the data annotation, and build a model that performs better than a model based on product search results. this scenario can happen when reliable annotation for the new data is not available, and we can use the prediction of the cross-encoder to replace human annotation. finally, from the first contribution, we get to train a better model with less training data. this is not a common practice in deep learning since more training data is usually preferred. our analysis shows that through the cross-encoder data annotation, we focus our training data on less queries. since the bi-encoder tries to learn the difference between the two, the cross-encoder data annotation removes extreme samples in training data, which acts similarly to noise removal. ## product matching given a product entry, a system is required to find entries in a product corpus that represent the same product, despite that the content in the entries are different. a product entry contains a set of information for a specific product, such as title, description, image, or categories. the multiple entries for the same product can be created by different vendors using the same e-commerce system. assuming that our product corpus c contains m product entries for n different products @xmath0, p 2 , p 3 ..., p m }, when given a query product p q , we want to find a target product p t that matches the query product in the corpus. the product corpus can be the product database for a deployed e-commerce system, which means the corpus size can easily contain over tens of millions of product entries. we consider product matching as an information retrieval problem. we learn the similarity between the product entries, and use the similarity between query product p q and target product p t to decide whether two product entries are for the same product. this approach does not have to retrain the model every time the number of products n changes. by increasing the n in the product corpus c, we can increase the coverage of our product matching system. ## real-world e-commerce scenario as a solution to the real-world product matching, there are two different types of approaches. the first is an approach based on a bi-encoder and another is based on a cross-encoder. we refer to the bi-encoder as known bi-encoder, and the crossencoder as known cross-encoder. these known models require annotated data, which can be considered as the system that already exists in the ecommerce platform or systems that researchers develop as a proof-of-concept. each time a new item is received, sellers on the ecommerce platform upload the product data to the product database. since the uploaded data sometimes contain incorrect information, a business unit in the e-commerce company periodically inspects if product data is correct. these inspection results can be regarded as human annotated data. therefore, regardless of the annotation, new data is accumulated in the e-commerce platform. to incorporate the latest product information into a production system, we want to build a new bi-encoder using the new data. we exploit the known bi-encoder and known cross-encoder to train the new bi-encoder effectively, which is our key contribution for this work. in the rest of this section, we describe the known bi-encoder, the cross-encoder, and how to train the new bi-encoder using these existing models. ## experiments our experiments are focused on the new bi-encoder. there are two main scenarios for our experiments, depending on whether the human annotations of the new products are available. for each scenario, we compare the experimental result with and without the cross-encoder data annotation. when the human annotations of the new products are available, we intersect human annotations with cross-encoder data annotations to improve the quality of training data. when the human annotations of the new products are not available, we use the cross-encoder data annotation for model training. in future work, we will compare the new biencoder with a bi-encoder model trained with all query products used for training the known biencoder, known-cross encoder, and the new biencoder. the reason why we skip this comparison is that the goal of our experiments is to see how changing the annotation of the same data such as doing intersection with other annotations can improve models. ## analysis since our experiments use the identical modeling approach, we focus on understanding the composition of training data and how such differences in training data affects the prediction performance. ## related work product matching is a fundamental task for an ecommerce platform. given the text in product entries, we want to match it to a specific product so duplicates can be identified. earlier works tried to solve it by extracting defined product attributes and perform matching based on the extraction results @xcite . recently, more efforts have shifted toward focusing on text @xcite . this avoids the need of doing attribute extraction and can directly be used on product titles and descriptions. there are two main directions for these efforts. one is considering the product matching task as an extreme classification problem, and another considering it as a zero-shot learning problem. as an extreme classification problem @xcite , the paper built a multi-class classifier that categorizes each input product information into a class, whereas each class represents a different product. the challenge is how to manage a multi-class classifier with several millions of classes, and the need to retrain the classifier every time a new product has entered the database. on the other hand, when considering a zero-shot learning problem, it focuses on learning the difference between the product texts @xcite . this makes it easier to apply the model on new products and more ideal in a production scaling environment. both cross-encoder and bi-encoder had been used for solving product matching as a zero-shot learning problem. how-ever, the computational complexity of the crossencoder makes it hard to scale millions of items. for the bi-encoder models, different loss functions @xcite had been applied to the task but the approaches are fundamentally similar. in addition, there are several works in different domains that inspired our paper. knowledge distillation @xcite is proposed to let a teacher model instruct a student model through learning. while their focus is on training, we applied similar ideas for data annotation. the biencoder approach for retrieval was also used in the open-domain qa task @xcite . though we do not need two separate encoders for the question and answer separately, the retrieval task is still similar in implementation. there are also works @xcite focused on analysis representation for text retrieval. our cross-encoder and bi-encoder models also use the difference in the representation created by different encoder for the product matching task. ## conclusion we demonstrated that we can use a cross-encoder to provide data annotation and to improve product matching performance on a bi-encoder. while such an approach can be useful when human annotation for new data is not available, it can also improve the quality of human-annotated data by conducting intersection. our empirical analysis suggests that the intersection of our cross-encoder annotation and human annotation creates more focused training data that improves the quality of the product embedding space. as a result of this annotation technique, we obtained a new way to improve human annotation quality or building bi-encoder model without human annotation for product matching.