Identifying High Consideration E -Commerce Search Queries (2024)

Abstract: In e-commerce, high consideration search missions typically require careful and elaborate decision making, and involve a substantial research investment from customers. We consider the task of identifying High Consideration (HC) queries. Identifying such queries enables e-commerce sites to better serve user needs using targeted experiences such as curated QA widgets that help users reach purchase decisions. We explore the task by proposing an Engagement-based Query Ranking (EQR) approach, focusing on query ranking to indicate potential engagement levels with query-related shopping knowledge content during product search. Unlike previous studies on predicting trends, EQR prioritizes query-level features related to customer behavior, finance, and catalog information rather than popularity signals. We introduce an accurate and scalable method for EQR and present experimental results demonstrating its effectiveness. Offline experiments show strong ranking performance. Human evaluation shows a precision of 96% for HC queries identified by our model. The model was commercially deployed, and shown to outperform human-selected queries in terms of downstream customer impact, as measured through engagement.

## introduction the integration of information content in online shopping is increasingly gaining importance @xcite , but such content may be more useful for certain search missions than others. the effective identification of specific subsets of keywords @xcite @xcite is essential not only for driving organic traffic and revenue in e-commerce, but also for enhancing the overall customer experience. related to the tasks of keyword selection and targeting @xcite figure 1 : an end-to-end demonstration of how the question-answer pairs of a widget is curated. @xcite , creating or serving customized content for specific queries helps provide relevant content for the right population. for example, when customers search for "prime day deals" on amazon, a dedicated widget will appear above product search results. clicking on this widget will direct customers to a specialized page with customized information, thus enhancing their shopping experience for specific deals. identifying such queries is usually the initial step prior to content creation and targeting. rather than directing customers to a dedicated web page, such targeted content could also be co-displayed with product search results when customers are looking for products. for example, a question-answer (qa) widget with relevant shopping knowledge could be rendered to match the customer's query, as shown in figure 1a . such tailored content is generally most useful for purchases requiring exploration, comparison, and deep decision making @xcite . we refer to such searches as high-consideration (hc) queries since consumers require additional information to consider their decision, or refine their search @xcite . however, curating customized content is an expensive manual process, and not feasible for all shopping queries as the query space is in the hundreds of millions. in figure 1b , we illustrate the end-to-end process of creating curated content. initially, hc queries are selected, typically through a manual process guided by heuristics. this often involves human annotators reviewing the top queries ranked by search frequency and identifying potential hc queries based on subjective criteria. next, customized content is manually curated for each selected query. to serve a qa widget, corresponding question-answer pairs are then created. finally, these customized qa pairs are ingested into a database and retrieved for specific queries @xcite as in figure 1a . manual keyword selection by humans is the most straightforward approach. however, even with frequency-ranked query lists this is an expensive and low-yield process as most queries are judged to be of low consideration (e.g., consumables and minor purchases). therefore, a lot of human efforts can be wasted using the conventional method. to address this, our work focuses on the task of automatically identifying the small subset of such hc queries in this large space. identifying the most valuable queries (i.e., step 1 in figure 1b ) is crucial for maximizing the return on investment (roi) of human efforts dedicated to content curation (i.e., step 2 in figure 1b ). as discussed later, the cost associated with content creation (and other factors), is an important consideration in framing this as a ranking task rather than a classification task. we hypothesize that hc queries can be identified by a combination of behavioral cues, financial signals, and catalog features. to identify hc queries, we propose the novel task of engagement-based query ranking (eqr) to train a model leveraging these signals. to learn this ranking function, our approach relies on engagement with informational shopping content in search results (e.g. a qa element) as a proxy target for hc query labels (which are expensive to define). this engagement can be measured as the click-through rate (ctr) of the content displayed for a set of seed queries. as we will show, these targets can then be used to create a generalizable model to predict hc queries across all search traffic. this approach also allows for continuous learning by using engagement from new content created for queries selected by our model. our key contributions are summarized below: query understanding in e-commerce query understanding is important to optimize search results for e-commerce platforms. to improve the relevance of search results while preserving the recall, embedding-based methods (lin et al., 2018) have been proposed to map a query into a target product category. the first empirical study on ecommerce queries was conducted by sondhi et al. (2018) and they categorized e-commerce queries into five categories based on different search behaviors. chen et al. (2023) introduced an intent classifier to determine whether to display an faq entry for a given query. our work can be considered as an extension to chen et al. (2023) and focuses on identifying new queries where customers could benefit from the associated content from a qa component. once those queries are identified, we could expand the qa database accordingly to increase its coverage. query performance prediction in information retrieval, the task of query performance prediction (qpp) @xcite aims to predict the effectiveness of a query given a retrieval system without using human-labeled relevance judgments. qpp methods can generally be categorized into pre-retrieval and post-retrieval methods @xcite . pre-retrieval methods are designed to estimate query performance before the retrieval stage is reached, and they utilize various features such as query term characteristics and collection statistics to make their predictions @xcite . on the other hand, post-retrieval techniques @xcite focus on deriving predictions from the ranked list of results obtained through the retrieval process. unlike pre-retrieval methods, post-retrieval predictors have access to the actual search outcomes, which can provide valuable additional information for analysis. for example, query clarity @xcite evaluates the quality of search results by measuring the kl divergence between language models derived from the search results and those from the corpus. for the first time, @xcite performed query performance prediction in e-commerce domain. our method for identifying high consideration queries shares commonalities with qpp methods, which provide insights for a query without relying on human judgments. instead of focusing on retrieval quality for a given query, we care about the downstream business impact (e.g., click-through rate) of curated content for a selected query. we also model the task as a regression task to predict a target measure of queries @xcite @xcite . trending queries detection our work is also related to detection of trending queries. @xcite analyzed on real data and showed that a topic trending on twitter may subsequently emerge as a popular search query on google. @xcite proposed to predict trending queries with a classifier trained on features derived from the historical frequencies of queries. more recently, trending prediction has also been explored in ecommerce scenario. @xcite introduced a method to mine fashion trends represented by product attributes on e-commerce platforms. trendspotter was proposed by @xcite to forecast trending products. different from prior work, our goal is to identify high-consideration queries that could encourage user engagement with related content, rather than focusing on the queries themselves. a trending query may not necessarily fall into the high-consideration category due to specific business considerations. additionally, our method does not rely on surface features of queries and we aim to discover new hc queries. ## experimental setup dataset as illustrated in figure 1a , we collected the data associated with a qa component from amazon spanning a one-year period. during this time, 11,273 queries were manually selected to trigger the qa component and display curated content. we obtained the corresponding query-level user engagement data as our proxy targets. the data was divided into training, validation, and test sets, with respective proportions of 70%, 15%, and 15%. ## discussion and conclusion we introduce the task of engagement-based query ranking in order to select high consideration queries. we proposed three categories of features to train pointwise rankers to address this task. our experimental results show that our proposed method achieves better performance than the baselines. the human evaluation indicates that our method could serve as an effective tool to save resources spent on error-prone human annotations. one limitation of our work is the difficulty in accurately measuring the true recall of our model. future work could consider the combination of product category predictions for queries and conduct the selection of high consideration queries for each product category. another limitation of our work is that we do not consider optimizing the curated content for selected queries and rely on human experts to decide what content to be create. furthermore, we did not address the removal of nearly duplicated queries, which would require a separate processing pipeline. future work could focus on personalized content selection for hc queries, and leverage active learning techniques to optimize the contentlevel engagement metrics. another promising direction to explore is the integration of behavioral and financial features, along with an innovative llm-based optimization approach @xcite to improve overall performance.