Towards Generalizeable Semantic Product Search by Text Similarity Pre-training on Search Click Logs (2022)

Abstract: Recently, semantic search has been successfully applied to E-commerce product search and the learned semantic space for query and product encoding are expected to generalize well to unseen queries or products. Yet, whether generalization can conveniently emerge has not been thoroughly studied in the domain thus far. In this paper, we examine several general-domain and domain-specific pre-trained Roberta variants and discover that general-domain fine-tuning does not really help generalization which aligns with the discovery of prior art, yet proper domain-specific fine-tuning with clickstream data can lead to better model generalization, based on a bucketed analysis of a manually annotated query-product relevance data.

## introduction semantic search has been widely studied recently as a promising competitor to token-based inverted index search that had dominated information retrieval (ir) and question answering (qa) for decades. however, in e-commerce product search, the quality of the learned embedding space is not well understood, especially on the quality of generalization to unseen queries, unseen products, or even unseen query-product pairs. the generalization is particularly interesting and important because in wayfair 80% of queries are new in the monthly basis, and new products are added everyday. in this paper, we will investigate the generalization issue by seeking answers to the two following questions: arguably, some pre-trained models, especially the ones pre-trained with domain-specific click-stream data, may outperform the others on generalization. to validate the hypothesis, we train twotower semantic search models using three different categories of base models for comparison: not pretrained (roberta-base @xcite , robertalarge), general-domain pre-trained with text similarity (simcse @xcite ) and domainspecific pre-trained with text similarity (simcse that are continuously pre-trained with clickstream data). to observe how the models perform, we take a publicly available product search dataset @xcite and partition the data into buckets according to whether the query/product/queryproduct pairs have been visited during training. by comparing the performance of the above models, we observe that query representations may have been insufficiently learned during two-tower semantic search fine-tuning as opposed to product representations, and the model variant which is pretrained on query-query text similarity has a visible impact on models' generalization power. this paper is organized as follows: 1) we first introduce related work, then 2) explain the proposed pre-training approach, and finally 3) discuss results and analysis. ## approach this section describes how models are pre-trained, fine-tuned, and evaluated on clickstream data. section 3.1 sets up preliminaries and math notations for the product search problem; section 3.2, 3.3 and 3.4 describe three different pre-training approaches; section 3.5 discusses how the model is fine-tuned and evaluated in the product search problem. ## results we evaluated the models' performance on the wands dataset. as noted in rq1: how well does the learned semantic space generalize? rq2: are we able to improve generalization by pretraining? in ## analysis to understand why queries and products follow different generalization pattern in terms of ndcg scores observed in rq1, we further analyzed the the "overal" bucket contains all (query, product) pairs in the testing dataset, the "seen" bucket contains (query, product) pairs from the testing dataset that also exist in the training dataset, and the "unseen" bucket contains (query, product) pairs from testing dataset that does not exist in the training dataset. the "unseen" bucket is further split into four different sub-buckets, according to whether the specific query or product exists in the training dataset or not. "q" represents query, "p" represents product; "+" represents existing in the training dataset, "-" represents not existing in the training dataset. bucket ratio represents the size of current bucket with respect to the entire test dataset. in ndcg scores, "*" indicates that number is statistically smaller than that column's largest (bold-faced) number, tested by one-side t-test with 0.05 p-value. ndcg results at other k's are presented in the appendix a. first, the models perform better on seen queryproduct combinations than unseen combinations (the query and the product has been individually seen in training, but not the combination). the bucket 2 from figure 2 shows a well learned cosine score distribution for seen query-product pairs, and the scores concentrate around a high similarity range around 0.65. in contrast, for bucket 4, although queries and products are individually seen by the models during training, the combination is novel to them, and it increases the models' uncertainty, which is demonstrated by a significant similarity score spread-out accompanied by a shift to a lower median around 0.3. in general, the figure shows the difficulty of generalization of deep learning-based semantic search models on even the simpler task of dealing with unseen combinations. this suggests the generalization may be beyond the discussion of learning representations for an individual query or product, and rather about the geometry of the semantic space to properly maintain the distance between seen and unseen texts after semantic search as asymmetric embedding learning that "disturbs" a well-defined text representation space to encode text rankings. second, as shown in figure 3 , the visibility of queries and products in training data poses opposite impacts to a model's behavior. figure 3: query-product cosine score distribution for bucket 4, 5 and 6. their mean cosine scores are 0.3562, 0.3655, and 0.3556 respectively. in bucket 4, both query and product are seen, but the combination is not seen by the model in training. in bucket 5, only the query is seen by the model in training. in bucket 6, only the product is seen by the model in training. by comparing the above two cases, we have seen that queries may have learned slower than products in semantic search. the reason may be due to the fact that queries are usually much richer and more diverse (we have tens of times more unique queries than unique products), requiring a text encoder to observe way more cases for a query to learn representation well; whereas products are relatively fewer, well-formed and limited in diversity in product search. however, for a company that has billions of products in inventory, this observation may not hold true. yet, the observation indeed resonates with that of question answering and web search, which suggests learning generalizable semantic search through improving query representation learning may still be a promising direction to follow. ## conclusions and future work this paper examines the encoder generalization problem in the setting of semantic product search. we observe that query representation learning still is the bottle neck compared to the product representation learning under the semantic search training objective, the problem of which is surfaced by the superior performance of the domain-specific query pair pre-trained encoder model. we also surprisingly found that large, general-domain and unsupervisedly regularized roberta variants such as sim-cse do not guarantee superior performance compared to roberta-base or large, which may have suggested that semantic search is an asymmetric relationship learning task rather than symmetric, similarity learning task, since the semantic space has to encode much more complex relations (e.g. node priorities in a list) beyond point-wise similarity. although being a proof of concept, the proposed pre-training approach could start the conversation about how to properly pre-train encoders for semantic product search, and the results suggest that the knowledge hidden in the click graph can be better explored by defining proper training tasks, either transforming the task into a sequential task learning setting or a multi-task setting. in the fu-ture, a transfer learning setting that leverage richer domain knowledge and semantic structures from relevant sources such as class and class taxonomy of queries may be promising. also, data augmentation to enhance query representation learning, or novel learning curriculum/objective/regularization that can help balance query and product learning in the semantic search objective may also be valuable for exploration.