Centrality-aware Product Retrieval and Ranking (2024)

Abstract: This paper addresses the challenge of improving user experience on e-commerce platforms by enhancing product ranking relevant to user’s search queries. Ambiguity and complexity of user queries often lead to a mismatch between user’s intent and retrieved product titles or documents. Recent approaches have proposed the use of Transformer-based models which need millions of annotated query-title pairs during the pre-training stage, and this data often does not take user intent into account. To tackle this, we curate samples from existing datasets at eBay, manually annotated with buyer-centric relevance scores, and centrality scores which reflect how well the product title matches the user’s intent. We introduce a User-intent Centrality Optimization (UCO) approach for existing models, which optimizes for the user intent in semantic product search. To that end, we propose a dual-loss based optimization to handle hard negatives, i.e., product titles that are semantically relevant but do not reflect the user’s intent. Our contributions include curating challenging evaluation sets and implementing UCO, resulting in significant improvements in product ranking efficiency, observed for different evaluation metrics. Our work aims to ensure that the most buyer-centric titles for a query are ranked higher, thereby, enhancing the user experience on e-commerce platforms.

## introduction achieving a user-focused experience on ecommerce platforms (ebay, walmart, amazon, etsy, jd) is enabled by ranking products relevant to the user's intent expressed via the search query. however, user queries often do not fully reflect the underlying intent behind the search terms used within the query. for example, ambiguous queries like 'iphone 13', or 'i5 pc 1tb 16gb 8gb gpu' can lead to many variants. to aggravate the challenge further, user queries can consist of lexical terms with alphanumeric characters, which do not reveal a semantic match within existing product titles. information retrieval (ir) systems depend upon semantic similarity/distance between words or phrases used in the search query and the product title. therefore, ranking the product titles based on only lexical or only semantic query-title match can be a particularly challenging problem, as detailed in the examples below: ambiguous queries some queries can be ambiguous and do not clearly reflect the user's intention. from the same example above, for a query like 'iphone 13', the user is most likely looking to buy the base variant or to check out other device variants. however, this intent is not clear from the query, and the system can even rank 'iphone 13 cover' among the top retrieved products. hence, a major challenge faced by search systems is to retrieve titles that are likely to be relevant to the user intent at high ranks, and push down negative titles such as 'iphone 13 cover' which have semantic proximity to positive titles within the embedding space of the computational model but may not reflect users underlying objective. repetition similar to the example above, the repetition of the exact string of words from a user's query, such as 'iphone 13', in both relevant and irrelevant titles often renders embeddings-based similarity approaches futile as the proximity of positive and negative titles in the embedding space may not be reflective of their relevancy. in such cases, human annotation towards user intent for a query-title pair is needed to establish a clear ranking among products retrieved by the model. alphanumeric queries queries such as 's2716dg' consist of alphanumeric characters where a letter or number can signify important detail for the product/model. for example, based on the naming convention of pc monitors, a single letter defines the type of panel in the product. in this case, the dell s2716dg is a 27-inch monitor with a tn panel, and changing the last letter to p would refer to a monitor with an ips panel. similarly, product colour or a specific spare part can be identified from such queries. unless the product title contains this alphanumeric sequence of characters, the semantic similarity between the query and a non-intended product can be high, thus misleading the system. in this paper, we investigate the challenges listed above and take a two-step approach to improve product retrieval and ranking. we curate samples from existing internal datasets at ebay consisting of user search queries paired with retrieved product titles on their platform. these datasets are humanannotated based on detailed guidelines to produce two buyer-centric relevance annotations. first, a widely used relevance ranking schema where querytitle pairs are provided a ranked class from among bad (1), fair (2), good (3), excellent (4) and perfect (5), where 'perfect' reflects an exact querytitle pair match, i.e., the annotator is very confident that the user found precisely what they were looking for, while 'bad' reflects no match between the product and the need expressed in the query @xcite . second, query-title pairs are annotated with a binary centrality score, obtained from majority voting over multiple human annotations, i.e., indicating whether the item reflects the need expressed in the query. the difference between centrality and relevance scoring is that the latter detects whether an item is an outlier, a surprising addition to the recall set, or the item centrally matches the expectations. figures 1 and 2 show two examples of the centrality annotation for the same query, "thomas sabo charm". figure 1 shows a product central to the query since, based on purchase data, this query typically reflects the user's need for a charm (a small ornament worn on a necklace or bracelet). on the other hand, the product in figure 2 is not central to the user's intent as it is a thomas sabo charm attached to a bracelet; the user intent is a charm, not a bracelet. although both titles are semantically related to the query, based on the degree of specificity expressed in the query, the product in figure 2 becomes less central to the user's intent and gets annotated with 0 as its centrality score whereas the product in figure 1 receives 1. we use an internal human-annotated figure 2: non-central title: thomas sabo charm club bracelet with detachable dragonfly charm dataset for this task. henceforth, we refer to it as internal graded relevance or igr dataset. we extract challenging evaluation sets from the igr dataset based on the challenges discussed above. our objective is to increase the retrieval and ranking efficiency of product search by training a model for query-title pairs that integrates the user intent in the similarity algorithm. given the search query, we propose using a user-intent centrality optimisation (uco) step for existing models which cater to the ranking of relevant products. further, we propose utilising a dual-loss based optimisation to address the query-title pairs which constitute hard negatives, i.e., query-title pairs where the product title is semantically relevant to the user's query but is annotated as non-central to the user intent, or has bad or only fair annotated relevancy. we hypothesise that there is an unwanted semantic proximity of such negative titles to their search queries in the model embeddings space. to improve search, we optimise the existing ranking model with our dual-loss-based optimisation approach, ensuring that the retrieval algorithm should have the most "typical" titles for a query ranked highly than other titles which may be relevant but are not typical. our contributions are 1) curating challenging evaluation sets that cater to this problem and 2) user-intent centrality optimisation (uco), which results in a stark improvement on all the evaluation sets. ## related work our work is based on a two-step approach to improve product ranking given a search query for retrieving items. existing literature on traditional candidate retrieval research focused on learning query rewrites @xcite as an indirect approach to bridge the vocabulary gap between queries and documents/titles. some approaches, including latent semantic indexing with matrix factorization @xcite , and with probabilistic models @xcite , and semantic hashing with an auto-encoder (salakhutdinov and hinton, 2009), have been proposed. most of these are unsupervised models based on word co-occurrence in documents/product titles. modern ir systems deploy semantic retrieval models as bi-encoders @xcite or siamese networks @xcite ) comprising two encoders. most existing studies focus on designing or pre-training encoders with different representation learning approaches @xcite @xcite @xcite . representative works, namely, the deep semantic similarity model (dssm) @xcite , and cdssm @xcite , are some of the earliest methods which utilise a deep neural network (dnn) using clickthrough data. subsequently, cnns @xcite @xcite and rnns @xcite have been utilised for semantic retrieval. recently, new models, including drrm @xcite and @xcite were developed to include traditional ir lexical matching (e.g., exact matching, term importance) within semantic retrieval performed by dnns. however, @xcite argues that most works proposed in this direction focus on the ranking stage, where the optimisation objectives differ from candidate title retrieval. to further improve the performance of semantic retrieval, transformer-based pre-trained language models (ptlms) like bert @xcite and ernie @xcite have been leveraged @xcite @xcite . using larger pre-trained models, semantic retrieval has observed a significant performance improvement and generalisation for retrieval but without a specific focus on ambiguous or alphanumeric queries, which is what we essentially address in this paper. further, interaction-based approaches @xcite @xcite @xcite have also been widely used for ir systems, which further go into semantic matching to model for querydocument/title interaction using dnns @xcite @xcite @xcite . most of these approaches focus on user personalisation needs, and often rely on hand-crafted rules. often, such approaches cannot cache the document embeddings offline for faster retrieval, and may be inefficient for retrieval @xcite . @xcite use the results of an online survey and search logs from a commercial product search engine to show that product search falls into categories like target finding, decision making and exploration. @xcite propose personal word-embeddings for personalized search (peps) which uses as additional layer trained on user embeddings and personal logs. while personalised embeddings and interactionbased approaches improve ranking performance for ambiguous user queries, our work focuses on dealing with similar challenges using a different approach infusing centrality-awareness. to be considered an impactful solution for the challenges at hand, we believe that product ranking approaches can be more generalised compared to personalised embeddings, improving the base retrieval with a focus on user intent. our approach utilises two existing loss functions that cater to the task and optimise the retrieval model, which can be used at both stages, retrieval and ranking. ## results and discussion considering various aspects like retrieval and ranking quality, we analyse model performance using a diverse set of metrics (explained in §4.1). we also perform an ablation test on the ebert model to identify the contribution of both loss functions, mnrl and ocl, and discuss the qualitative analysis below. ## conclusion and future work this work addresses product search queries that represent an important challenge for e-commerce platforms. the main challenge occurs when the retrieved titles are semantically relevant, but not central to the user-intent as is reflected by the specificity of the query. the challenge is even greater with ambiguous queries where the same query string is present in both relevant and irrelevant titles as well as when queries are alphanumeric. we address the semantic complexity of these challenging query-title pairs by fine-tuning existing internal models with a user-intent centrality optimisation (uco) step to infuse information about the typicality of query-title pairs. the retrieval model performance showed significant improvement with several hard example datasets with a dual-loss based optimisation approach, which pays attention to negative pairs that have a lower distance than the positive pairs with the largest distance. the dual-loss based optimisation helps in separating the irrelevant pairs of queries and titles while keeping the distance smaller for relevant query-title pairs. the improvement in ranking performance demonstrated by our approach helps identify and categorise what users intend to find online when they search the platform. in future, we aim to restructure queries in our hard-negative pairs to be less ambiguous. leveraging genai-based prompt engineering and explainability using approaches like chain-of-thought, we can investigate titles that indicate typical queries, aligning them closer to the user intent, and moving towards explainable product retrieval.