Offloaded Reasoning: Efficient Inference for Large Language Models via Modular Reasoning and Refinement (2025)

Abstract: Large language models (LLMs) demonstrate strong reasoning capabilities but are expensive to run at inference time, limiting their practical deployment. We propose Offloaded Reasoning (OR), a modular strategy where a lightweight model generates intermediate reasoning traces that are then used by a larger model to produce the final answer. We further introduce Offloaded Reasoning with Refinement (ORR), where the large model first edits or improves the reasoning trace before answering. Unlike token-level acceleration methods, OR and ORR operate at the reasoning level and require no retraining of the large model. Experiments on GSM8K and Math500 show that OR achieves up to 8x faster inference than full large-model reasoning with minimal accuracy loss, while ORR recovers or exceeds full accuracy at substantially lower cost. Our results highlight the potential of modular, delegation-based reasoning for building more efficient and adaptable LLM systems.

## introduction large language models (llms) have demonstrated strong performance on a wide range of complex reasoning tasks, such as arithmetic problem solving, logical inference, and open-domain question answering @xcite @xcite @xcite . however, the computational demands of such models during inference-especially at deployment scale-pose significant bottlenecks in resource-constrained applications. a central challenge is to reduce the inferencetime compute cost of llms without degrading their reasoning capabilities. recent approaches like speculative decoding @xcite address this by drafting token sequences using smaller * work conducted while at samsung r&d institute india -delhi figure 1 : workflow of offloaded reasoning (or) and offloaded reasoning with refinement (orr). in or, a lightweight model generates an initial reasoning trace, which is consumed by a larger model to produce the final answer. in orr, the large model additionally refines the initial reasoning before generating the answer, improving accuracy while maintaining efficiency. models and verifying them with larger ones. while effective for speeding up token generation, such methods operate purely at the token level and remain agnostic to the higher-level reasoning structure required for complex tasks. moreover, speculative decoding imposes a significant practical constraint-it requires the smaller draft model and the larger verifier model to share the same tokenizer and vocabulary. this restricts model pair selection and limits flexibility in deploying heterogeneous model combinations for efficiency. in this work, we propose offloaded reasoning (or), a simple yet powerful alternative that targets the reasoning process itself. instead of having the large model perform all computation, or offloads the generation of the intermediate reasoning trace (e.g., a chain-of-thought) to a smaller model. the large model then reads this offloaded reasoning and produces the final answer. this division allows the small model to shoulder the "cognitive burden" of reasoning while the large model focuses on synthesis, significantly reducing overall inference cost. to further improve quality and robustness, we introduce offloaded reasoning with refinement (orr). here, the large model not only uses but also refines the offloaded reasoning-correcting or extending it as needed-before producing the answer. this enables the large model to catch errors in the initial trace while still enjoying substantial compute savings. our approach is orthogonal to decoding-level acceleration and complements token-efficient generation. it builds on insights from chain-ofthought prompting, and modular reasoning, but focuses squarely on decoupling and optimizing the reasoning pipeline. we evaluate or and orr on two challenging mathematical reasoning benchmarks-gsm8k @xcite and @xcite . our main findings are: our results show that reasoning can be effectively delegated to smaller models, enabling significant inference savings without compromising quality. we argue that modular reasoning strategies like or and orr offer a promising direction for efficient and scalable nlp systems, and point to a future where reasoning becomes a reusable, composable primitive in llm pipelines. ## methodology let q denote an input question and a the final answer. in traditional llm inference, a large model m l directly maps q to a by internally generating a reasoning trace r l (e.g., a chain-of-thought), this process enables the large model to focus solely on synthesis (answer generation) while delegating the expensive reasoning process to a smaller model. the approach assumes that the small model is capable of generating sufficiently high-quality intermediate reasoning to scaffold the large model's answer. ## experiment settings we evaluate offloaded reasoning (or) and offloaded reasoning with refinement (orr) on two established mathematical reasoning benchmarks: gsm8k and math500. ## results and analysis our central goal is to demonstrate that modularizing the reasoning process via a small model m s can deliver efficiency gains without sacrificing (and sometimes improving) performance. ## related work the goal of reducing inference cost while maintaining or improving quality has been explored through multiple research avenues, including efficient decoding strategies, modular computation, and few-shot reasoning techniques. our approach draws from and contributes to each of these directions. ## conclusion offloaded reasoning with refinement uniquely combines modular reasoning, efficiency, and robustness in a unified framework. while prior works have addressed components of this pipeline, our contribution lies in orchestrating small and large models for reasoning and refinement, resulting in significant inference gains without compromising accuracy. this modularity also enables dynamic scaling, making it highly suitable for real-world deployments with variable compute budgets.