Hybrid- RACA : Hybrid Retrieval-Augmented Composition Assistance for Real-time Text Prediction (2024)

Abstract: Large language models (LLMs) enhanced with retrieval augmentation has shown great performance in many applications. However, the computational demands for these models pose a challenge when applying them to real-time tasks, such as composition assistance. To address this, we propose Hybrid Retrieval-Augmented Composition Assistance (Hybrid-RACA), a novel system for real-time text prediction that efficiently combines a cloud-based LLM with a smaller client-side model through retrieval augmented memory. This integration enables the client model to generate better responses, benefiting from the LLM’s capabilities and cloud-based data. Meanwhile, via a novel asynchronous memory update mechanism, the client model can deliver real-time completions to user inputs without the need to wait for responses from the cloud. Our experiments on five datasets demonstrate that Hybrid-RACA offers strong performance while maintaining low latency.

## introduction large language models have become powerful tools in language processing and they are widely adopted across applications. when augmented with retrieved documents @xcite , these models can generate more relevant and useful responses. however, the large size of these models and the additional retrieval step introduce significant computational overhead. this leads to increased latency and higher operational costs, limiting their effectiveness in real-time applications, such as composition assistance. real-time composition assistance tools are designed to swiftly suggest next words or sentences to help users write faster. these systems must operate within tight latency budgets, and they are frequently triggered as the user types. to minimize latency (including model inference latency * these authors contributed equally to this work. and communication to the cloud) and to reduce costs, these models are usually deployed on users' edge devices. this imposes strict constraints on the model's size and capabilities, limiting the effectiveness of composition assistance. while recent advancements have enabled models such as llama @xcite to run on smaller devices foot_0 , they still fall short in terms of achieving real-time responses. for real-time tasks, we encounter a dilemma: llms offer superior performance but they are slow and expensive to run, whereas client models are agile and efficient but limited in performance. hybrid computing between client and cloud models is a promising approach to bridge the gap between the challenges of latency and model performance. however, in existing hybrid computing patterns, such as model routing and split computing @xcite , client and cloud models usually function with synchronized communication. this means that whenever the cloud model is utilized, the system must wait for the cloud model to complete its processing before producing the output. therefore, simply applying existing hybrid patterns to cloud-based llms will not resolve the issue of latency and cost. besides, existing hybrid patterns usually overlook cloud-based data, which could be essential for effective composition assistance, such as accessing relevant documents in companies' cloud storage. to address these challenges, we propose a novel hybrid retrieval-augmented composition assistance (hybrid-raca) system (see figure 1 ). this system leverages a cloud llm and cloud data to boost the performance of small language models on client devices through retrieval augmentation, while operating in an asynchronous manner. hybrid-raca consists of an augmentation coordinator and a small model for text prediction de- ## hybrid-raca we present our hybrid-raca system that leverages cloud-generated memory to enhance the utility of client-based language model while maintaining low latency for composition assistance. in hybrid-raca, the augmentation coordinator (client) monitors the writing context and sends an asynchronous request for an augmented memory from the cloud. the retriever on the cloud searches for relevant data upon request. subsequently, the memory generator (cloud) leverages an llm to construct a memory that includes all essential information from the retrieved data, optimizing its usefulness. finally, the memory is transmitted to the client and seamlessly integrated into the client model for offering real-time suggestions. algorithm 1 describes the inference workflow of hybrid-raca. in the following subsections, we discuss the details of the four main components. ## experiments in this section, we introduce the experimental setup (section 4.1) and present the evaluation results of hybrid-raca system on utility (section 4.2.1), inference latency (section 4.2.2) and effects of asynchronous memory update (section 4.2.3). ## conclusion in this paper, we propose hybrid-raca, a novel hybrid retrieval-augmented generation system for real-time composition assistance. by integrating llm-enhanced memory into our instruction-tuned client model with asynchronous update, we show with experiment results on multiple datasets that our hybrid approach enables substantial utility improvements over smaller language models while maintaining inference efficiency, making it a valuable solution for real-time tasks.