L lama PIE : Proactive In-Ear Conversation Assistants (2025)

Abstract: We introduce LlamaPIE, the first real-time proactive assistant designed to enhance human conversations through discreet, concise guidance delivered via hearable devices. Unlike traditional language models that require explicit user invocation, this assistant operates in the background, anticipating user needs without interrupting conversations. We address several challenges, including determining when to respond, crafting concise responses that enhance conversations, leveraging knowledge of the user for context-aware assistance, and real-time, on-device processing. To achieve this, we construct a semi-synthetic dialogue dataset and propose a two-model pipeline: a small model that decides when to respond and a larger model that generates the response. We evaluate our approach on real-world datasets, demonstrating its effectiveness in providing helpful, unobtrusive assistance. User studies with our assistant, implemented on Apple Silicon M2 hardware, show a strong preference for the proactive assistant over both a baseline with no assistance and a reactive AI assistant, highlighting the potential of LlamaPIE to enhance live conversations.

## introduction user interaction with language models (lms) has primarily followed a turn-based dialogue format, where users actively request responses from lm assistants. this approach requires users to shift their attention and carefully phrase prompts to obtain useful answers. thus, usability is restricted to scenarios where users can pause their activities to engage with the conversational assistant. here, we explore a compelling alternative: what if an ai system could proactively assist users without explicit invocation? we introduce llamapie: proactive assistants designed to augment human-tohuman communication through discreet, contextaware guidance delivered via hearable devices. 1 these real-time assistants operate as mostly silent co-pilots, providing occasional and unobtrusive feedback to the wearer by anticipating user needs. unlike traditional dialogue assistants, they operate in the background and allow users to remain focused on their human interactions. such proactive companions may enhance human interactions in a variety of contexts, including negotiations @xcite , interviews @xcite , customer support @xcite , and cross-cultural communication @xcite . they may support neurodiverse individuals, such as those with autism or social anxiety, by assisting in the interpretation of social cues @xcite , or serve as personal memory assistants for older adults experiencing progressive memory loss, providing contextually relevant information during conversations @xcite . creating proactive assistants presents several challenges. modern language models are typically designed to provide detailed responses, but in our case, assistance should be limited to just 1-3 words and offered only when it enhances the conversation without disrupting its flow. the assistant must also be context-aware, leveraging the wearer's memory of prior activities and interactions. achieving this enhancement to live conversations requires realtime streaming processing, while privacy considerations necessitate processing conversational data on edge devices rather than sending it to the cloud. finally, there does not exist large-scale annotated data of in situ human conversations with real-time assistance for model training. to collect data emulating this kind of assistance, we first construct a semi-synthetic dialogue dataset that incorporates in-ear assistance. each example consists of a user profile, memory, and a dialogue between the user and other speakers with our assistant aiding the user. special silence markers indicate timing information for speaker utterances. to meet on-device real-time constraints, we design llamapie as a two-model pipeline that separates the decision-making process into two stages. first, a small classifier determines "when to respond" in a streaming manner. then, when a response is needed, a larger "what to say" model generates a concise reply. while both models continuously process input, the large model generates tokens only when triggered by the small model, reducing computational overhead. we tune llama3.2-1b and llama3.1-8b on our dataset to create the small and large models, respectively. to demonstrate the generalizability of our models trained on synthetic data, we evaluate llamapie on audio recordings from the mit interview dataset @xcite . in automatic evaluation with llm-as-a-judge, we show that assistance provided is generally helpful and unobtrusive. to validate the llm-as-a-judge scores, we also perform annotations with 21 human scorers that show strong correlation between the human and lm scores. finally, we bring llamapie "to life" by integrating it into a speech-to-speech setup that the user wears on a wireless bone-conduction headset. the end-to-end system is implemented on apple silicon m2 hardware, supported by commodity mobile devices. all language and speech models operate in a streaming manner, with intermediate states continuously cached to maximize computational reuse and achieve real-time inference. in our study, 15 participants read a passage, then were tested on the reading in an interview setting. with the proactive assistant, user test accuracy rose from 37% to 87%. participants also found llamapie much less disruptive to the conversation than a reactive system, where users interacted with chatgpt through both its voice and text interfaces. our work demonstrates a vision for proactive lm assistance that centers human-to-human conversation, and highlights the potential for such interactions to serve users in ways that complement the currently widespread human-ai "chat" interface. ## llamapie we first formulate the problem of proactive, in-ear assistance for human conversations ( §2.1). then, we describe the dataset construction ( §2.2) and finally our two-model pipeline ( §2.3). ## evaluation we first discuss evaluation metrics §3.1, followed by results on the system's ability to decide when to ## user study: llamapie in real-time human conversations to evaluate llamapie's potential in assisting real users during live conversations, we use our pipeline on a mac m2 platform with 16 gb of memory. speech assistance is delivered via a shokz open-move bone-conduction headset, which do not obstruct the user's hearing of the conversation. we recruit 15 human participants (ages 20-40) to interact with our real-time on-device prototype and assess their experience. to simulate conversations where users may require assistance, we design mock interview and trivia scenarios. each scenario consists of eight topics, with five questions per topic-two easy and three difficult. each participant experiences three different conditions: control, where no assistance is provided; proactive (ours), where users receive assistance from llamapie through the headset; reactive (baseline), where users can access gpt-4o via a web ui during the conversation. in addition, five participants also experienced the following condition: reactive (short audio), where users access gpt-4o via its voice mode during the conversation. gpt-4o was prompted to respond concisely (1-3 words) and users received assistance through the headset. for each participant, we randomly assign a different topic to each of the three conditions. all participants begin with the control condition, while the order of the baseline lm and proactive assistance conditions is randomized. participants are given 3-5 minutes to read and memorize the background information for their assigned topic. the background information falls into two categories: following the reading period, participants engage in a casual conversation, during which they are asked one easy question and two difficult questions. a transcript of the spoken dialogue, along with timestamps, is recorded for analysis. ## conclusion we introduce the first proactive in-ear conversation assistant, designed to provide discreet, concise guidance. our proof-of-concept, real-time, on-device implementation, which centers humanto-human conversation, demonstrates the potential for such interactions to serve people in ways that are complementary to the currently widespread human-ai "chat" interface. our experimental findings show the promise of such tools and establish a starting point for future research.