Improving Relevance Quality in Product Search using High-Precision Query-Product Semantic Similarity (2022)

Abstract: Ensuring relevance quality in product search is a critical task as it impacts the customer’s ability to find intended products in the short-term as well as the general perception and trust of the e-commerce system in the long term. In this work we leverage a high-precision cross-encoder BERT model for semantic similarity between customer query and products and survey its effectiveness for three ranking applications where offline-generated scores could be used: (1) as an offline metric for estimating relevance quality impact, (2) as a re-ranking feature covering head/torso queries, and (3) as a training objective for optimization. We present results on effectiveness of this strategy for the large e-commerce setting, which has general applicability for choice of other high-precision models and tasks in ranking.

## introduction search is one of the primary means used by customers to find products in e-commerce and therefore it is critical to ensure the relevance quality of search results. a search result may be considered to have low relevance quality (search defect) if it mismatches the customer's query intent. such defects may range from the mild case of mismatch in brand or color (i.e., substitutes) to the more egregious case of a completely irrelevant result of a different product type. addressing search defects is a critical task as it can damage customer trust and perception of the e-commerce system, and in general hinder the ability to sell products. as a simplified view, product search may consist of two distinct phases. first, given a search query, a set of candidate products are determined based on various matchset generation techniques (e.g. lexical/semantic matching, historical associations from past query reformulations, and others). next, a ranking model is used to generate a score for each (query,product) pair upon which a descending sort determines a ranked list. in the ideal case, the construction of the matchset would be strictly restricted to products that are only relevant to the customer's query, however, this can be challenging to enforce without potentially limiting recall, which itself presents issues that negatively impact search experience. in practice, products in the matchset may still contain complementary or related items to the customer-intended one due to partial matches or from noisy historical associations. the ranking phase can be used to mitigate the impact of search defects that may exist in the matchset by demoting such results out of the first several pages. in contrast to matchset restrictions, demotion in ranking can be seen as a softer approach since a product will not be entirely eliminated from search results but simply moved beyond the topresults. given a dataset with relevance labels for query-product pairs, relevance quality can be optimized within ranking to demote products estimated to be less relevant to the customer's query. however, this strategy will have a dependence on similarity measures that often need to favor online efficiency over a higher-precision computationallyexpensive counterpart. for example, common ranking features may include query-product lexical/semantic match features and behavioral features that incorporate historical customer interactions; for all these cases, efficient computation over the entire matchset will be required, whether by simple online computations (e.g. tf-idf, cosine similarity) or by retrieving pre-computed offline (intermediate) results or a combination of both. in this work we leverage a high-precision model bounded by offline computational resources for addressing our ranking-based task. specifically, we develop a high-precision cross-encoder bert model for semantic similarity between customer query and products that is optimized for predicting relevance quality and we survey its effectiveness for three applications where offline-generated scores could be used: (1) as an offline metric for estimat-ing relevance quality impact, (2) as a re-ranking feature covering head/torso queries, and (3) as a training objective for optimization. we present results on effectiveness of this strategy for the large e-commerce setting, which has general applicability for choice of other high-precision models (i.e. other than bert) and tasks in ranking (other than relevance quality). ## related work generating textual representations that are effective for downstream tasks is an active area of research that has seen significant improvement in the last several years @xcite @xcite . bert @xcite is one such model based on a multi-layer bidirectional transformer encoder @xcite that has shown state-of-the-art performance on various nlp tasks. in this work, we leverage bert for two-sentence classification with cross-encoders that is known to have strong predictive performance while at the same time presenting computational challenges for large-scale product search @xcite . various strategies can be used towards improving scalability of model inference such as model distillation/compression @xcite @xcite @xcite and factorizing inputs into separate model paths (i.e. two-tower or bi-encoder models) @xcite @xcite , but these optimizations typically come at the expense of model performance. for example, using a factorized model enables opportunities to cache pre-computed embeddings and use efficient distance functions on embeddings vectors (e.g. cosine similarity), but this architecture will sacrifice interactions between inputs within early model layers that tend to be helpful. indeed, the trade-off between precision and inference speed is not limited to bert nor transformer-based models; in general, higher precision for a given task may be achievable when there is flexibility to use more complex models (e.g. more parameters, ensemble methods), and consume more costly features (e.g. embeddings generated from a secondary model and fed as input). given its applicability and known computational challenges for our task, in this work we use bert directly as our highprecision model (i.e. without any computational efficiency optimizations) and apply it for several applications where offline-computed scores are permitted. ## query-product semantic similarity for our task we would like to develop a measure of semantic similarity g : (q, a) → r, given an arbitrary query @xmath0, where we assume the cardinality of q and a may be infinite. we select g by favoring a model that maximizes predictive performance bounded by offline resources as opposed to meeting stricter online inference requirements. as mentioned in the previous section, we adopt bert and frame our problem as twosentence classification @xcite to incorporate textual inputs for the query and product. for two-sentence classification using bert as a cross-encoder the input sequence is prepared by prefixing a 'cls' token followed by the query textual representation, a special separator token ('sep'), and finally the product textual representation via product title. this input sequence is segmented into sub-words using wordpiece algorithm @xcite and fed through bert-base pre-trained model (12 transformer blocks, 768 hidden units, and 12 self-attention heads). we further pre-train the bert-base model using both masked lm and next sentence prediction tasks as described in @xcite on product metadata such as title and description. for building a classification model, the output embedding for 'cls' token is passed to a final linear classification layer. all model weights, including transformer block layers, are trained jointly using binary crossentropy loss. the labeled dataset consisting of judgments tuples (query,product,label) is based on historical query-product samples with relevance judgments (relevant vs irrelevant), which is split into train/validation/test datasets. for evaluating relevance quality, we use ndcg metric (normalized discount cumulative gain) that achieves the highest value of 1 when the rank order respects the ideal relevance label ordering, which is then averaged over all queries. using this high-precision predictor, we explore several applications in search ranking where we can leverage this model using offline-generated scores for our end task. these are discussed in the next section. given that relevance quality is an important metric across search, it is useful to monitor it alongside other important metrics (e.g. revenue, latency) for each experiment that impacts ranking of products and search experience. here we propose using our semantic predictor for estimating online relevance impact and seek to measure which offline estimator (our high-precision predictor or baseline) better reflects the changes observed for the online metric. we represent our observations as pairs (x i , y i ) where, for of a given model (treatment) over production (control), x i is the estimated improvement given an estimator and offline dataset, and y i is the actual observed improvement as measured by human-judged labels. for our study, our dataset consists of 19 pairs collected from 5 experiments within a 6-month span impacting ranking for a particular marketplace. our offline estimate is measured on a ranking evaluation dataset of queryproduct pairs by measuring the improvement in exact probability (output of our semantic predictor) among top results based on the treatment's rank over the control's rank, averaged over all queries. similar estimation is done for our baseline model, which is a gbdt model trained similar to baseline in section 3. using this dataset of offline-online impact pairs, we measure pearson and kendall correlation coefficients. ## conclusion and future work to improve relevance quality in search ranking we applied a high-precision bert cross-encoder model for semantic similarity in search. we demonstrated three applications where offline-generated scores can be leveraged to improve the end task. this can be viewed as a complementary approach alongside efforts for developing online-efficient models, where the advantages include leveraging higher-precision models and with potentially less development overhead. a follow-up study including benchmarking on public datasets is left as future work.