S ky LLM : Cross- LLM - API s Federation for Cost-effective Query Processing (2025)

Abstract: Large language models (LLMs) have demonstrated exceptional capabilities across a wide range of tasks, from text generation to complex problem-solving. LLM APIs provide easy access to these models by streamlining deployment and usage. Combining LLMs with complementary strengths has been shown to yield substantial performance gains over a monolithic LLM. However, invoking a fixed set of LLM APIs for each query incurs higher API costs and increased inference latency. To address these limitations, we propose SkyLLM, a system composed of a set of estimators and an API selector, which federates multiple LLM APIs and dynamically assigns a non-empty subset of these APIs to each query prior to inference under cost and latency budgets. The selected subset consists of either a single LLM or multiple LLMs. A single LLM efficiently handles simple queries at low cost, whereas multiple LLMs are employed for more complex queries to overcome performance limitations. We evaluate SkyLLM against individual LLMs and representative ensemble LLM methods from the literature. SkyLLM achieves the highest accuracy under a high budget. It can also be cost-effective, matching the most accurate individual LLM while cutting costs by 67.8%.

## introduction large language models (llms) have become the dominant force in natural language processing (nlp) in recent years. their impressive capabilities have spanned a wide range of applications e.g., question answering, code generation, text summarization, and open-ended conversation. this swift progression has been propelled by breakthroughs in model frameworks, particularly the transformer framework @xcite , in addition to enhancements in scaling up data and training infrastructure @xcite . in many cases, the massive size of modern llms makes local deployment both computationally and financially prohibitive, especially as many stateof-the-art models remain closed-source 12 . llm apis provide a streamlined solution, enabling users to access these powerful models via cloud-based services. by paying a usage-based fee, users can submit queries and receive responses without the overhead of managing hardware or model weights, thereby lowering the barriers to llm adoption. the diversity in data, architectures, and hyperparameters causes different llms to exhibit varying strengths and weaknesses across tasks @xcite . consequently, ensembling multiple llms with complementary advantages can help produce higher-quality outputs. llm ensemble can be broadly categorized into three types: cascading, routing, and fusion. llm cascading sequentially invokes increasingly powerful models until a satisfactory response is obtained. llm routing directs each query to the most suitable llm before inference, typically using a trained router. llm fusion integrates outputs from multiple models, during or after inference. despite their potential advantages, each of these ensemble approaches has inherent limitations. cascading and routing methods ultimately depend on the final output of a single llm, thereby constraining their accuracy to the performance of the best model in the pool. moreover, since models are invoked sequentially, llm cascading increases inference time, which is a critical factor for user experience, particularly in real-time applications such as chatbots and voice assistants. while previous studies have shown that combining multiple llms can significantly enhance accuracy compared to individual models @xcite , fusion-based methods generally invoke a fixed set of llms for each query @xcite , resulting in higher api costs and longer inference times. to address these limitations, we propose a dynamic ensemble approach. given a set of llms, we dynamically assign a non-empty subset of these models to each query, based on query-specific characteristics such as subject and difficulty. this subset consists of a single llm or multiple llms. for more complex queries, selecting multiple llms can improve accuracy at the cost of increased api expense and latency. conversely, for simpler queries, opting for a single, well-suited llm achieves cost efficiency and low latency. this flexible strategy aims to balance performance gains against the overhead of higher costs and longer inference time. however, the dynamic ensemble framework introduces several challenges. first, the combinatorial nature of choosing subsets expands the search space exponentially as the number of models increases, especially when each query may require a distinct subset. second, because the subset must be selected prior to inference, accurately predicting its performance (e.g., cost, latency, accuracy) is non-trivial. moreover, we explicitly treat cost and latency as constraints in the selection process to achieve a balanced trade-off, which further complicates the selection process. in this paper, we introduce skyllm, a unified system that dynamically assigns a non-empty subset of federated llm apis to each query prior to inference. skyllm is designed to maximize accuracy while adhering to cost and latency constraints. to implement skyllm, we leverage a set of estimators (cost, latency, and accuracy), derived from an open dataset containing responses and metadata (e.g., inference time and token count) from a diverse set of llms across a large collection of training queries, to assess the expected performance of each subset. skyllm then formulates the selection task as a multi-dimensional group knapsack problem. to reduce the exponential search space, skyllm employs pareto dominance pruning to eliminate strictly inferior subsets and then applies dynamic programming to determine the optimal subset of federated llm apis for each query. we evaluate skyllm against individual component llms and representative llm ensemble methods from the literature. under a high budget, skyllm achieves the highest accuracy. furthermore, it demonstrates cost-effectiveness, attaining accuracy comparable to the most accurate individ-ual llm while reducing costs by 67.8%. in summary, our contributions are as follows: for proprietary models, such as those offered by openai, access is exclusively provided through official apis, ensuring compatibility with the latest updates and provider policies. in contrast, open-source models are accessible via third-party providers such as together ai foot_2 and deepinfra foot_3 , which offer apis that streamline model deployment and usage. while api-based access eliminates the operational overhead of llm deployment, it introduces challenges related to api costs and inference latency. skyllm addresses these challenges by dynamically selecting subsets of federated llm apis tailored to cost and latency constraints, effectively balancing accuracy with reduced overhead. ## implementation we implement the skyllm system in python. this section details the implementation of each component, including the llms, estimators, and api selector. llms. we adopt llms from openai (gpt-4o, gpt-4o-mini, and gpt-3.5-turbo) and qwen (qwen-max, qwen-plus, and qwen-turbo). both openai and qwen apis return log probabilities for a limited number of top-ranked tokens at each generated token position: openai provides up to 20 tokens, while qwen provides up to 5. we average if the desired answer is not among these tokens, its probability is set to 0. estimators. for the accuracy estimator, we employ the sentence transformer all-minilm-l6-v2 from hugging face foot_7 to map each query to a 384dimensional dense vector. to efficiently search for similar queries in dataset t , we use faiss foot_8 to retrieve i similar queries for each query. in the cost estimator, model-specific api pricing is obtained from respective providers. the number of prompt tokens is estimated using tiktoken foot_9 . we apply skyllm to the mmlu dataset @xcite . since mmlu consists of multiple-choice questions, the number of output tokens is fixed at one. in the latency estimator, inference time is predicted using a multilayer perceptron (mlp) regressor consisting of a two-layer feedforward neural network with 50 neurons per hidden layer. the model takes the model id, query embedding, number of prompt tokens, and number of output tokens as input features. after evaluating our estimators on the mmlu dataset, we observe that the accuracy estimator achieves a cross-entropy loss of 0.44. the average cost per query is @xmath03.1 × 10 -10 , indicating highly precise cost predictions. meanwhile, since the maximum number of output tokens is restricted to one, the overall inference time remains relatively short, with a mean inference time of 0.54 s. consequently, even modest network fluctuations result in a comparatively large estimation error, leading to an mse of 0.16 s for the latency estimator. api selector. the api selector first enumerates all 2 n -1 possible subsets for each query, estimates their cost, latency, and accuracy, and applies pareto dominance pruning to discard inferior subsets. after pruning, we solve the multi-dimensional group knapsack problem via dynamic programming (dp). since dp uses cost and latency as discrete indices, we convert floating-point cost and latency values to integers by multiplying by enlargement factors (α c and α t ) and applying the ceiling function. larger α c and α t values provide finer granularity but increase dp's time and space complexity, as the budgets are also enlarged. to mitigate excessive memory usage, we employ lz4 11 , a high-speed lossless compression algorithm that reduces memory consumption during runtime. once dp completes, we retrieve the chosen subsets, invoke apis, and ensemble outputs for each query, ultimately returning the final answer. ## conclusion in this paper, we investigate the query handling problem under cost and latency constraints. we propose skyllm, a system composed of a set of estimators and an api selector, which federates multiple llm apis and dynamically assigns a nonempty subset of these apis to each query prior to inference. the selected subset consists of either a single llm or multiple llms. a single llm efficiently handles simple queries at low cost, whereas multiple llms are employed for more complex queries to overcome performance limitations. we evaluate skyllm against individual component llms and representative llm ensemble methods from the literature. skyllm achieves the highest accuracy under a high budget. it can also be costeffective, matching the most accurate individual llm while cutting costs by 67.8%.