Automatic Transmission for LLM Tiers: Optimizing Cost and Accuracy in Large Language Models (2025)

Abstract: LLM providers typically offer multiple LLM tiers, varying in performance and price. As NLP tasks become more complex and modularized, selecting the suitable LLM tier for each subtask is a key challenge to balance between cost and performance. To address the problem, we introduce LLM Automatic Transmission (LLM-AT) framework that automatically selects LLM tiers without training. LLM-AT consists of Starter, Generator, and Judge. The starter selects the initial LLM tier expected to solve the given question, the generator produces a response using the LLM of the selected tier, and the judge evaluates the validity of the response. If the response is invalid, LLM-AT iteratively upgrades to a higher-tier model, generates a new response, and re-evaluates until a valid response is obtained. Additionally, we propose accuracy estimator, which enables the suitable initial LLM tier selection without training. Given an input question, accuracy estimator estimates the expected accuracy of each LLM tier by computing the valid response rate across top-k similar queries from past inference records. Experiments demonstrate that LLM-AT achieves superior performance while reducing costs, making it a practical solution for real-world applications.

## introduction large language models (llms) have become powerful tools for a wide range of natural language tasks. due to the high computational demands for serving large-scale llms, llm providers such as openai typically offer multiple tiers of proprietary models (e.g, 4o-mini, 4o, o1-mini, and o1). higher-tier models exhibit superior performance but come at a higher cost, while lower-tier models are cheaper but fall short in accuracy. as nlp tasks are more complex @xcite @xcite , tasks † major in bio artificial intelligence are solved by dividing them into multiple subtasks, each requiring different levels of reasoning or generation capabilities. for instance, the tree of thoughts approach @xcite often solves a single complex task with more than tens of llm calls. this introduces a new challenge: selecting the suitable llm tier for each substask to optimize the trade-off between accuracy and cost. although using the top-tier model for all subtasks guarantees high-quality results, it can be prohibitively expensive when a complex task requires a lot of llm calls. conversely, relying solely on lowertier models reduces costs but risks performance degradation. recent studies @xcite @xcite have explored trainingbased approaches to select the suitable llm, but have several limitations. first, these methods require extensive data labeling to assess whether each llm generates accurate responses to a given question, making the process time-consuming and costly. second, when new llms are released, retraining is necessary to integrate new llms into the selection process. third, these methods become unreliable when applied to test domains that differ from the training data distribution, limiting their generalizability in real-world applications. to address this problem, we propose the llm automatic transmission (llm-at), a novel framework that dynamically selects the suitable llm tier without training. llm-at consists of three main components: generator, judge, and starter. the generator produces a response for a given question using the initial llm tier selected by the starter. the judge then evaluates the validity of the response. if the response is invalid, llm-at upgrades to the higher tier llms and repeats the process until a valid response is obtained. figure 1 shows an example of the llm-at process. using gpt-4o selected as the initial tier by the starter, the generator answers a given question. however, the judge determines that the response is invalid, llm-at upgrades to the higher-tier model, o1-mini, which then produces the valid output. starting with a low llm tier for a complex question can cause unnecessary tier escalations, leading to increased costs and execution time. therefore, we introduce accuracy estimator that estimates the probability of each llm tier successfully answering a given question by leveraging real-time accumulated inference records, referred to as history, without requiring correctness-labeled data. this enables the starter to select the cheapest llm tier expected to generate a correct response while reducing additional llm calls. our main contributions are summarized as follows: llm routing. several studies have explored learned llm routing methods to select the most suitable llm for a given question. chen et al. (2024a) employs a cascading mechanism, where llms are sequentially invoked using a stop function that evaluates response quality. however, since this method always starts with the cheapest model from a fixed sequential permutation, complex queries necessitate multiple model invocations, leading to increased cost and latency. ding et al. (2024); ong et al. (2024) formulate routing as a binary classification problem, directing simple queries to smaller models and complex ones to larger models. however, these approaches fail to account for the diverse performance levels of proprietary llms, which are rapidly emerging, leading to suboptimal routing decisions and reduced flexibility in real-world applications. on the other hand, lu et al. (2024); chen et al. (2024b); maurya et al. (2024) explore routing as an alternative to ensembling multiple llms. thus, these methods focus on select the best-performing llm on a given question, disregarding inference cost. iterative refinement in llms. several studies have explored iterative refinement techniques to enhance the accuracy and reliability of llmgenerated responses. these approaches enable models to evaluate their own outputs, incorporate feedback, and iteratively improve their responses without additional training. madaan et al. (2023) introduce a method where llms generate selffeedback on their initial responses and refine them through multiple iterations, improving factual consistency and coherence. gou et al. (2024) propose an iterative verify-then-correct framework that enhances model reliability by using external tool-assisted critiques. yao et al. (2023) presents a prompting strategy that incorporates reasoning traces, allowing llms to dynamically track, update, and adjust their responses. existing works show that llms can engage in self-evaluation and iterative refinement, leveraging both internal reasoning and self-feedback to enhance response quality over multiple iterations. ## llm-at framework most llm providers offer various tiers of llms with different price points and performance levels. in general, higher-tier models provide better performance but come with higher usage costs. therefore, this study aims to optimize the tradeoff between performance and cost given the llm tiers. specifically, our goal is to achieve performance comparable to the top-tier model while minimizing overall costs, including monetary expenses and execution time, when handling inputs of varying difficulty levels. additionally, we assume an environment where past interactions can be stored and utilized. it is a realistic scenario widely adopted in actual llm-based chatbots such as chatgpt (openai, 2022), claude @xcite and deepseek (deepseek, 2025) etc. ## conclusion we introduce llm-at, a novel framework that automatically selects llm tiers to maximize performance while reducing cost and execution time. we also propose accuracy estimator, which estimates the accuracy of each llm tier based on past inference records without answer labels, enabling the selection of the most appropriate starting tier expected to solve a given question. our approach requires no additional training or supervised fine-tuning, making it readily applicable in real-world scenarios. we conduct extensive experiments on datasets ranging from simple to graduatelevel questions, demonstrating that llm-at optimizes the trade-offs between accuracy and cost as well as accuracy and execution time.