A mazon QAC : A Large-Scale, Naturalistic Query Autocomplete Dataset (2024)

Abstract: Query Autocomplete (QAC) is a critical feature in modern search engines, facilitating user interaction by predicting search queries based on input prefixes. Despite its widespread adoption, the absence of large-scale, realistic datasets has hindered advancements in QAC system development. This paper addresses this gap by introducing AmazonQAC, a new QAC dataset sourced from Amazon Search logs, comprising 395M samples. The dataset includes actual sequences of user-typed prefixes leading to final search terms, as well as session IDs and timestamps that support modeling the context-dependent aspects of QAC. We assess Prefix Trees, semantic retrieval, and Large Language Models (LLMs) with and without finetuning. We find that finetuned LLMs perform best, particularly when incorporating contextual information. However, even our best system achieves only half of what we calculate is theoretically possible on our test data, which implies QAC is a challenging problem that is far from solved with existing systems. This contribution aims to stimulate further research on QAC systems to better serve user needs in diverse environments. We open-source this data on Hugging Face at https://huggingface.co/datasets/amazon/AmazonQAC.

## introduction query autocomplete (qac) is an important feature in nearly every modern search engine @xcite . as the user types out a search query, the qac system's aim is to provide a list of search term suggestions based on the partially typed query (the "prefix"). ideally, the qac system will provide the user's intended query, which they can select, thereby saving them the effort of typing out the full query. even where the user does not have a specific query in mind, qac suggestions can help them formulate search queries that lead them to the results they are seeking. however, despite the importance of qac, it is a comparatively under-explored task in research. the publicly available datasets tend to be derived from search query datasets (e.g. @xcite @xcite . however, these datasets do not contain the prefixes that users typed, so prefixes have to be synthetically constructed @xcite , greatly limiting the empirical value of these resources for qac. in fact, we were unable to find any publicly available large scale qac datasets beyond synthetically constructed ones from the aol data release in 2006. in general, the lack of large-scale realistic benchmarks has hampered research on qac; few tasks have as large a gap between their importance in real-world technologies and the amount of research devoted to them. in this paper we aim to facilitate more research on qac by releasing amazonqac, a qac dataset collected from amazon search logs with participation and support from amazon. amazonqac contains 395m anonymized examples, where an example consists of a submitted search term together with the sequence of prefixes that was typed to reach that search term, a session id, a timestamp, and other metadata ( we present the task description, analyze dataset statistics, describe evaluation metrics, and motivate an upperbound success@10 score of 69.8% on our held-out test set. we also evaluate several baseline approaches on amazonqac: prefix trees, semantic retrieval, and large language models (llms) with and without finetuning. we find that the qac problem is not just a simple case of prefix-search term memorization, as conventional wisdom might imply, but rather that it is a complex recommendation problem that is significantly influenced by the user's search context. our best baseline system is a finetuned llm that leverages session context, and it achieves success@10 of 37, which is half of our upperbound. this indicates that the qac problem is a difficult one not readily solved by current systems. we hope that releasing amazonqac will prompt further innovation in qac systems and that our baseline systems help guide these research efforts. ## baseline systems and results in order to provide researchers with qac baselines on our dataset, we train and benchmark a cross-section of different qac approaches with our dataset. our implemented systems may not be state-of-the-art, since we rather aim to provide a base number and insights into how different approaches to qac behave on our dataset. qac approaches can broadly be split into two categories: information-retrieval (ir) qac and generative qac. we explore representative models from both categories. our results are summarized in ## conclusion we introduced the amazonqac dataset to help address a critical need for realistic, large-scale datasets for query autocomplete (qac). ama-zonqac is derived from amazon search logs and contains 395m examples with rich metadata. our analysis of a range of baseline approaches suggests that qac is a challenging context dependent task that benefits from the generative capacity of modern llms. in particular, finetuning llms to perform the qac task and make use context leads to especially strong results. however, even the best of these systems falls well short of optimal performance on amazonqac, suggesting that there is plenty of room for further innovation. overall, we hope the availability of amazonqac helps catalyze further research and innovation in qac, driving the development of more intuitive and efficient search functionalities across digital services.