C ha I - T e A : A Benchmark for Evaluating Autocompletion of Interactions with LLM -based Chatbots (2025)

Abstract: The rise of LLMs has deflected a growing portion of human-computer interactions towards LLM-based chatbots.The remarkable abilities of these models allow users to interact using long, diverse natural language text covering a wide range of topics and styles. Phrasing these messages is a time and effort consuming task, calling for an autocomplete solution to assist users. We present **ChaI-TeA**: **Cha**t **I**n**te**raction **A**utocomplete; An autocomplete evaluation framework for LLM-based chatbot interactions. The framework includes a formal definition of the task, curated datasets and suitable metrics. We use it to evaluate 11 models on this task, finding that while current off-the-shelf models perform fairly, there is still much room for improvement, mainly in ranking of the generated suggestions. We provide insights for practitioners working on this task and open new research directions for researchers in the field. We release our framework to serve as a foundation for future research.

## introduction large language models (llms) have revolutionized many nlp applications @xcite . a prominent example is automatic chatbots; what used to be confined, topic-specific applications often requiring the user to use restricted language or choose from a closed list of interaction options, have been transformed. these applications, powered by llms, are now one-stop-shops successfully communicating in unbounded natural language while acting as experts on a wide variety of topics @xcite . due to their remarkable abilities, llm-based chatbots differ significantly from prior human-computer communication methods. interactions with these history current turn completion @xmath0. figure 1 : the chatbot interaction autocompletion task. given the conversation history and the current turn's prefix, task is to suggest suitable completions. autocomplete (ac) methods have been shown to be effective in saving users' time and reducing their cognitive load in many different use-cases, suggesting that such a solution might be of value for the llm-chatbot interaction use-case as well. the popular query autocomplete scenario @xcite focuses on search queries. classic solutions often rely on recurrence, making them irrelevant for the long unique natural language text found in chatbot interactions @xcite . later solutions include generative models @xcite , but still focus on short semi-structured queries. code autocomplete @xcite deals with structured language, and often relies on the ability to run the code and check its output in order to evaluate solutions. lastly, email (human-human) interactions @xcite , which bear a closer resemblance to human-chatbot interactions due to their natural language communication, also differ in several key aspects. these include the number of participants and their roles, the more formal writing style of emails and the nature of the topics discussed. in broader terms, human-human textual interactions (e.g., emails, but also texts from other kinds of messaging platforms) differ from human-chatbot interactions in the fact that humanchatbot interactions involve a human and a modelbased assistant, making them more instructional and knowledge-seeking. for example, the prompts "give me the latest updates of the war in ukraine as of the 31st of january." and "write a web scraping program in python capable of ..." are taken from the oasst dataset used in this work to demonstrate typical examples for a human-chatbot interaction, which are highly unlikely to be found in a humanhuman messaging platform. in this paper, we introduce the task of autocompleting user interactions with llm-based chatbots. we present chai-tea: chat interaction autocomplete; a framework for evaluating autocomplete solutions for llm-based chatbot interactions. it includes a formal definition of the task, suitable datasets tailored for autocomplete, suitable metrics, and baseline results. we go on to highlight some valuable insights. first, we explore how performance can be traded off for lower latency, a key factor in autocomplete solutions. second, we show that models can exploit distant history to suggest completions. third, it is beneficial to enable completions of various lengths (as opposed to only single words or full turns). we highlight a key factor in improving these solutions: we find that models tend to generate completion suggestions well, but are not as good at ranking these generated suggestions. given that users can ingest a small amount of suggestions at each turn, ranking is an important component in an offered solution. therefore, we advocate for future research in the field to focus on this aspect. ## task definition the chatbot interaction completion task focuses on completing user turns in user-chatbot interactions. similarly to @xcite , we model it as a sequential task; completions are suggested at each typing step (i.e., after a user types a character). formally, at each step t, an autocomplete solution (denoted by ac) is given a context c containing all previous conversation turns, originating from both the user and the chatbot, and the prefix of the current user turn denoted as p t . the autocomplete solution should then return a set of k completions, c t 1 , ..., c t k , possibly of varying lengths. each completion step can be described as: ac(c, p t ) = {c t 1 , c t 2 , ..., c t k } after receiving the set of completions, the user can either accept a completion or continue typing. if a completion c t i is accepted, the prefix is updated such that p t+1 = p t + c t i . then, whether the user selected a completion or continued typing, a new completion step is initiated, until reaching the end of the user's turn. a single completion step is illustrated in figure 1 , and full turns completions can be found in the appendix in ## further analysis latency-performance trade-off. given the practical importance of latency in ac solutions, we explore how performance can be traded off for reduced latency. to illustrate this trade-off, we varied the previously mentioned hyperparameters n c and n t , as well as the context length given to the model. we capped the conversation history concatenated with the turn prefix at different lengths, to determine whether giving the model access to the entire conversation context is both helpful and worthy of the extra latency costs. suggestions are offered between words, meaning that once the user begins typing the next word they become irrelevant. hence, we find it appropriate to use the mean time between typed words -718 ms, reported by (conijn, 2020) -as a benchmark. results per latency budget, presented in for each latency budget, we report the configuration with the highest saved@100 score that fits the budget. ## conclusions in this work, we showcase the task of autocompleting user interactions with llm-based chat-bots. we formally define the task and design an evaluation framework, and use it to test 9 different models. results show that while lms are able to perform fairly, there is room for a tailored solution to improve upon them, especially in the ranking of completion candidates. we show that models can exploit distant history, that enabling completions of different lengths is beneficial and that reducing latency for this task should be done by reducing context length and length of completions as opposed to generating less completions. we hope our framework will encourage further work in this area, which we believe holds great potential value for users across various llm chat-bot applications.