Generative Product Recommendations for Implicit Superlative Queries (2025)

Abstract: In recommender systems, users often seek the best products through indirect, vague, or under-specified queries such as “best shoes for trail running.” These queries, referred to as implicit superlative queries, pose a challenge for standard retrieval and ranking systems due to their lack of explicit attribute mentions and the need for identifying and reasoning over complex attributes. We investigate how Large Language Models (LLMs) can generate implicit attributes for ranking and reason over them to improve product recommendations for such queries. As a first step, we propose a novel four-point schema, called SUPERB, for annotating the best product candidates for superlative queries, paired with LLM-based product annotations. We then empirically evaluate several existing retrieval and ranking approaches on our newly created dataset, providing insights and discussing how to integrate these findings into real-world e-commerce production systems.

## introduction superlative queries are common in product search as users seek products with the highest degree of one or more attributes to satisfy their needs. while some superlative queries can be handled by existing retrieval systems @xcite through attribute-based filtering (e.g., "the largest m2 pro with 32 gb ram"), others can pose challenges to the existing solutions. specifically, in this paper, we study the problem of product ranking and recommendation for implicit superlative queries, where the desired product attributes are not explicitly stated. these queries often involve aspects that require common sense knowledge of the product @xcite . this problem is further compounded by users creating vague and under specified search queries, either due to a lack * work done at amazon. ## related work we now discuss related work to place our contributions in context. ## implicit superlative queries we now formalize the type of superlative queries that we seek to address. we define implicit superlative queries as those which (i) seek the highest degree of one or more attributes or features of a product; and (ii) are implicit in nature. these queries involve preferences which are generally popular, subjective, and not just based on quantifiable attributes. e.g., the superlative query "best toy for my 3-year nephew who loves the flintstones" -requires an implicit understanding that the user might be looking for a good quality toy which is well-rated and reviewed, reasonably priced, age appropriate, and relates to characters or properties of the show "the flintstones". addressing such implicit superlative queries would require (i) inferring hidden attributes, (ii) world knowledge or a general understanding of concepts, and (iii) being able to reason and compare across different related products and ensure that the necessary attributes are of the highest degree. ## the superb relevance scheme we design a novel four-category relevance taxonomy to rank, recommend, and evaluate the retrieved product candidates for superlative queries. we design such a fine-grained system for multiple reasons. fine-grained labels have been found to be more advantageous than simplistic binary choices @xcite . in addition, they facilitate nuanced evaluations and provide comprehensive feedback. for example, differentiating between overall best and almost best might be less obvious when purchasing standard office supplies, where basic functionality is adequate. however, this distinction becomes essential when selecting infant car seats, where the highest safety and technology standards are vital. ## dataset construction we now describe how we generate superlative queries and pair them with products labeled with annotations from our schema. ## methods we perform our analysis in a constrained setting where the item description is limited to 512 tokens in length. this is useful for low latency applications. we then use superb for evaluating the following ranking pipelines: (i) bm25: we use bm25 as our baseline. (ii) rm3: we also employ a pseudo-relevance feedback baseline rm3 @xcite . (iii) bm25/rm3 + listwise re-ranking: here, we re-rank the results of the first stage bm25 and rm3 through a listwise ranking approach. we force the model to generate a ranked list of product ids in the style of rankgpt @xcite . this can also be seen as a black-box counterpart of pointwise ranking approaches which provide confidence through logit probabilities. the confidence scores range between @xcite @xcite @xcite @xcite . we choose the claude-haiku (anthropic, 2024a) model for our experiments since it is beneficial to evaluate smaller models for production pipelines. we use the pyterrier (macdonald and tonellotto, 2020) library with the pyterrier-genrank @xcite plugin for designing the retrieval and re-ranking pipelines, and computing precision and ndcg metrics. ## results and analysis as shown in we also show the results for top-100 and top-200 items with long descriptions in ## conclusion this work studied superlative queries with implicit attributes, which are typically more complex compared to other query types since ranking products for them requires inferring attributes, placing other products in context, and using commonsense knowledge to determine the best ones. our analysis shows that llms can rank the best items, improve ranking when provided with initial ranked lists, and can also be sensitive to them. in addition, our methods are applicable to rank superlative queries in other item and document ranking settings. we present the superb, 4-point schema and propose pointwise, deliberated pointwise, pairwise, and listwise methods to label superlative queries over it and re-rank retrieved products, using an llm as the backbone. the listwise approach is preferable for lower budgets, while the deliberated point-wise approach can be preferred for better quality annotations. we believe that our study can drive further research on superlative search queries. our work highlights key considerations for deploying an llm-based product ranking system into production. while a listwise approach effectively ranks multiple items at once, it can be inefficient due to lengthy item descriptions. in contrast, a pointwise approach is faster, especially with parallel processing. sliding window methods and query reformulation are also viable alternatives. generating attributes and explanations clarifies label assignments, boosting user trust and satisfaction. addressing superlative queries in product recommendation systems is essential, particularly for the next generation of interactive shopping assistants @xcite and generative recommender systems @xcite . this becomes even more relevant as informationseeking and product search system grow closer together @xcite . these superlative queries capture user intent to find the best possible items, an aspect often overlooked in current systems. introducing superb allows for the development and assessment of recommendation pipelines capable of handling high-expectation queries, helping systems address this unmet need.