Faster and Better LLM s via Latency-Aware Test-Time Scaling (2025)

Abstract: Test-Time Scaling (TTS) has proven effective in improving the performance of Large Language Models (LLMs) during inference. However, existing research has overlooked the efficiency of TTS from a latency-sensitive perspective. Through a latency-aware evaluation of representative TTS methods, we demonstrate that a compute-optimal TTS does not always result in the lowest latency in scenarios where latency is critical. To address this gap and achieve latency-optimal TTS, we propose two key approaches by optimizing the concurrency configurations: (1) branch-wise parallelism, which leverages multiple concurrent inference branches, and (2) sequence-wise parallelism, enabled by speculative decoding. By integrating these two approaches and allocating computational resources properly to each, our latency-optimal TTS enables a 32B model to reach 82.3% accuracy on MATH-500 within 1 minute and a smaller 3B model to achieve 72.4% within 10 seconds. Our work emphasizes the importance of latency-aware TTS and demonstrates its ability to deliver both speed and accuracy in latency-sensitive scenarios.

## introduction test-time scaling (tts) is an effective approach to improve the performance of large language models (llms) at the cost of additional inferencetime computations @xcite . tts can be realized by two basic approaches: sequential scaling and parallel scaling. sequential scaling requires the model to produce an extended reasoning process in a single pass @xcite . in contrast, parallel scaling generates multiple solutions in parallel and selects the final answer, usually through majority voting @xcite . hybrid approaches can be constructed on top of the two basic ones @xcite . with the number of generated tokens (#tokens) as budget, many existing studies @xcite @xcite @xcite have examined compute-optimal strategies that enhance average performance gain per token, a metric we 2 10 2 11 2 12 2 13 2 14 2 refer to as token efficiency. however, in latencysensitive scenarios where small batch sizes are employed, e.g., personal computer, small-scale commercial deployment and edge device, "computeoptimal" does not necessarily translate to "latencyoptimal". this discrepancy is because the performance achieved within a limited time is determined by token efficiency as well as the throughput (i.e., average number of output tokens per second). as shown in figure 1 , for s1.1-32b @xcite model, although sequential scaling has a better token efficiency, achieving a similar performance with 9× fewer tokens compared to parallel scaling, it turns out that parallel scaling achieves 1.6× lower latency. in fact, under small batch sizes, the time of an llm autoregressive decoding step is dominated by the memory access of the parameters. therefore, a moderately increased number of parallel branches incurs little additional latency, allowing a much higher throughput almost for free, as shown in figure 3 . we then ask how we can achieve latency-optimal test-time scaling. one lesson from the observation above is that, in addition to optimizing token efficiency, attention must be given to improving the generation concurrency to chase a throughput. there are two approaches to improving concurrency: (1) branch-wise parallelism, which increases the number of parallel branches b, and (2) sequence-wise parallelism, which generates multiple successive tokens for a sequence in a single forward pass with speculative decoding @xcite . however, current research lacks a thorough analysis of how to allo-cate computational power to these two resourcecompeting approaches and to what extent they can be improved. to bridge the gap, we examine the impact of concurrency configuration for latency-aware test-time scaling in this study. specifically, we conduct experiments on representative datasets including math-500 @xcite , aime24 (aops, 2024), aime25 (aops, 2025), and gpqa-diamond @xcite across model sizes from 3b to 32b and under varied concurrency configurations. revealed by our experiments and analyses, the latency-optimal concurrency depends on the comparative advantage of token efficiency in sequential scaling and parallel scaling. for models with higher token efficiency with parallel scaling (e.g., llama-3.1-8b-instruct (meta, 2024)), one should prioritize branch-wise parallelism. otherwise (e.g., for qwq-32b @xcite ) sequence-wise parallelism takes priority. to determine the latency-optimal configuration, we propose a simple yet effective greedy search algorithm with less searching steps. under 1 minute latency constraint, an optimal concurrency configuration can provide up to 7.3% better performance than baselines where only a single parallelism is applied. our contributions can be summarized as follows: ## related work test-time scaling scaling the compute at inference time has been proven to be a prominent approach to improve llm performance. generally, test-time scaling methods fall into two main categories: sequential and parallel scaling. for the former, sequential scaling methods, represented by openai's o1 (openai, september 2024), deepseek-r1 @xcite , and qwen qwq @xcite , enforce the model to generate longer solutions with a detailed reasoning chain. such an ability can be incentivized with supervised training @xcite @xcite or reinforcement learning @xcite . another main category is parallel scaling. this method first generates multiple response candidates in parallel, then applies a selection criterion to identify the best output. recent research primarily uses token count as a metric to measure the budget of test-time scaling @xcite @xcite . recent work @xcite shows that majority voting outperforms verifier-based methods under low/medium token budgets. notably, low latency necessitates limited token usage, where majority voting is superior. thus, our work employs majority voting for parallel scaling. different from previous works, our work points out that computeoptimal does not necessarily translate to latencyoptimal. we introduce latency-aware test-time scaling and figure out the concurrency configuration to latency-optimal tts. speculative decoding speculative decoding uses a small draft model to generate several draft tokens autoregressively, and the target model to verify them in parallel while ensuring a lossless acceleration @xcite . many studies focus on improving acceleration. eagle-series @xcite , medusa @xcite , and hydra @xcite employ features from the target model for better draft acceptance rate. hass @xcite mitigates the inconsistency between training and inference by simulating the multi-step draft generation in the training phase. hierarchical drafting @xcite improves the acceptance rate by regarding the hierarchical drafting strategy based on temporal locality. moesd @xcite shows that at medium batch sizes, moe models with speculative decoding can surpass dense models. our work demonstrates the feasibility of latency-aware tts optimization through speculative decoding, presenting it as one viable approach to latency-optimal tts. ## rethinking test-time scaling with latency previous works measure the budget of llm testtime scaling by #tokens. however, it provides an incomplete evaluation, particularly when considering real-world deployment constraints. our investigation reveals the necessity to consider latency-aware tts in latency-sensitive scenarios. ## conclusion in this paper, we propose to rethink test-time scaling in latency-sensitive scenarios. we show that compute-optimal does not always result in latencyoptimal under such conditions due to memorybound constraint. to address this, we propose to improve tts on generation concurrency to maximize throughput from a unified view. specifically, we present two approaches: (1) branch-wise parallelism via multiple branches and (2) sequence-wise parallelism by speculative decoding, along with their combinations. furthermore, we investigate the concurrency allocation strategy to balance these approaches for latency-optimal tts. a simple yet effective greedy search algorithm is proposed to determine the optimal configuration. experimental results show that latency-optimal tts enables 32b model to achieve 82.3% accuracy on math-500 within 1 minute, while 3b model attains notable accuracy within just 10 seconds, showing significant improvements in both speed and accuracy.