On-Device LLM s for Home Assistant: Dual Role in Intent Detection and Response Generation (2025)

Abstract: This paper investigates whether Large Language Models (LLMs), fine-tuned on synthetic but domain-representative data, can perform the twofold task of (i) slot and intent detection and (ii) natural language response generation for a smart home assistant, while running solely on resource-limited, CPU-only edge hardware. We fine-tune LLMs to produce both JSON action calls and text responses. Our experiments show that 16-bit and 8-bit quantized variants preserve high accuracy on slot and intent detection and maintain strong semantic coherence in generated text, while the 4-bit model, while retaining generative fluency, suffers a noticeable drop in device-service classification accuracy. Further evaluations on noisy human (non-synthetic) prompts and out-of-domain intents confirm the models’ generalization ability, obtaining around 80–86% accuracy. While the average inference time is 5–6 seconds per query—acceptable for one-shot commands but suboptimal for multi-turn dialogue—our results affirm that an on-device LLM can effectively unify command interpretation and flexible response generation for home automation without relying on specialized hardware.

## introduction smart home technologies and iot devices have proliferated in recent years, with an expected rise from 16.6 billion to 18.8 billion connected devices by the end of 2024 @xcite . major providers like amazon, google, and apple typically handle speech recognition and intent detection on cloud servers, which raises user concerns about privacy, data ownership, and reliance on proprietary ecosystems @xcite . conventional solutions for home assistants often rely on specialized, domain-specific classifiers for slot and intent detection (sid), paired with templated system responses. while these approaches can be effi-* the authors contributed equally to this work. cient, they can also be rigid, sometimes requiring precisely phrased user inputs and yielding repetitive or unpersonalized answers. recent developments in on-device computing-coupled with improvements in model compression and quantization @xcite @xcite -have paved the way for smaller yet still capable language models to run on commodity hardware. these models offer privacy benefits and allow customizable local inference with reduced latency. however, deploying a capable model under strict memory and computational constraints remains challenging. large-scale transformer-based language models @xcite , and especially llms @xcite @xcite @xcite , have demonstrated remarkable proficiency in tasks ranging from question answering to text generation @xcite , yet typically demand substantial hardware resources, restricting them to cloud-based services or large compute clusters. this paper explores whether a smaller, finetuned llm can provide two capabilities essential to a home assistant-accurate recognition of what users want (i.e., slot and intent detection), and natural textual responses-while running entirely on an edge device with limited cpu and memory. by unifying these tasks into one end-to-end system, we eliminate the need for separate domain-specific classification modules and templated responses, focusing on efficiency, robust language understanding, and strict correctness in json action output. additionally, we move away from classic sid datasets and other general spoken language understanding benchmarks. instead, we investigate whether llms can be directly applied to digital assistant software. to this end, we take the open-source home assistant software foot_0 as our gold standard for evaluation, targeting real-world deviceservice pairs and actionable json outputs. contributions. our contributions can be summarized as follows: foot_1 1 we show that a 0.5b llm can be fine-tuned to already jointly handle sid and response generation with high accuracy. 2 by quantizing the model (from 16-bit to 8-bit and 4-bit), we quantify trade-offs between memory usage, accuracy, and generative fluency on cpu-only edge hardware. 3 we evaluate the approach on synthetic data, human queries, and out-of-domain tasks, confirming robust generalization. ## methodology our goal is to integrate two core functionalities of a home assistant into a single model: traditional classifiers only handle device-service classification and do not produce any text. for userfacing text, the baseline approach would rely on templated responses. ## discussion despite near-perfect performance on the synthetic test set, regarding speed, the 1.5b model yields consistent accuracy gains on ood data but also increases inference time by up to 2-3×. for single-turn commands, 5-6 seconds per query might be acceptable, but multi-turn dialogue would require faster or more efficient strategies. future work may explore parameter-efficient fine-tuning, context truncation, or advanced quantization (e.g., 8-bit + partial 4-bit layering) to reduce inference times. ## conclusion we present that llms can simultaneously perform sid and natural language response generation for a home automation domain. experiments on an 8gb ram, cpu-only environment show that 8-bit quantization largely preserves in-domain accuracy (up to 99%) and strong text fluency, while 4-bit introduces significant classification errors despite retaining good generative capability. we further demonstrate promising generalization to humanwritten prompts and out-of-domain tasks, with accuracy around 80-86%. however, per-query inference times of 5-6 seconds indicate that llmbased assistants, as implemented here, are not yet ideal for fast multi-turn dialogues on edge devices. future work can refine these models for faster, more memory-efficient inference, enabling privacypreserving yet flexible home automation assistants.