In this paper, we address the problem of evaluating whether results served by an e-commerce search engine for a query are good or not. is is a critical question in evaluating any e-commerce search engine. While this question is traditionally answered using simple metrics like query click-through rate (CTR), we observe that in ecommerce search, such metrics can be misleading. Upon inspection, we nd cases where CTR is high but the results are poor and vice versa. Similar cases exist for other metrics like time to click which are o en also used for evaluating search engines. We aim to learn the quality of the results served by the search engine based on users' interactions with the results. Although this problem has been studied in the web search context, this is the rst study for e-commerce search, to the best of our knowledge. Despite certain commonalities with evaluating web search engines, there are several major di erences such as underlying reasons for search failure, and availability of rich user interaction data with products (e.g. adding a product to the cart). We study largescale user interaction logs from Flipkart's 1 search engine, analyze behavioral pa erns and build models to classify queries based on user behavior signals. We demonstrate the feasibility and e cacy of such models in accurately predicting query performance. Our classi er is able to achieve an average AUC of 0.75 on a held-out test set.
Kumar et al. REF observed that predicting the query performance in product search cannot be based only on standard metrics such as click-through-rate (CTR), since CTR might be high while the results are poor, in terms of relevance perspective.
Did We Get It Right? Predicting Query Performance in E-commerce Search
ArXiv
ArXiv
Computer Science