Representing LLM s in Prompt Semantic Task Space (2025)

Abstract: Large language models (LLMs) achieve impressive results over various tasks, and ever-expanding public repositories contain an abundance of pre-trained models. Therefore, identifying the best-performing LLM for a given task is a significant challenge. Previous works have suggested learning LLM representations to address this. However, these approaches present limited scalability and require costly retraining to encompass additional models and datasets. Moreover, the produced representation utilizes distinct spaces that cannot be easily interpreted. This work presents an efficient, training-free approach to representing LLMs as linear operators within the prompts’ semantic task space, thus providing a highly interpretable representation of the models’ application. Our method utilizes closed-form computation of geometrical properties and ensures exceptional scalability and real-time adaptability to dynamically expanding repositories. We demonstrate our approach on success prediction and model selection tasks, achieving competitive or state-of-the-art results with notable performance in out-of-sample scenarios.

## introduction llms have recently emerged with remarkable capabilities, revolutionizing multiple diverse fields, including medical information processing @xcite , software engineering @xcite @xcite , and scientific research @xcite @xcite . moreover, open-source llms are widespread and have fueled a rapidly growing ecosystem of publicly-available models and benchmarks used to assess their capabilities. currently, the most prominent platform hosting these resources is hugging face @xcite , serving as a centralized repository for nearly one million pre-trained models and hundreds of thousands of benchmarks. such open-access repositories facilitates llms' large-scale deployment and promotes their continuous innovation across diverse applications. the demand for llm-based applications is everincreasing, and new, diverse models with improved capabilities are constantly being produced. however, this rapid growth and diversity present a substantial challenge: identifying the best-performing models. this challenge entails recognizing the most suitable model for producing a response to specific queries, or on average over queries in a given dataset. hand-selecting these models would require careful analysis and clear annotation of their properties, which are not widely available. a common approach is to rely on benchmark results to select suitable models @xcite . a benchmark consists of a dataset and an evaluation metric designed to assess specific model capabilities, such as domain expertise @xcite , reasoning skills @xcite @xcite , agentic abilities @xcite , or safety @xcite @xcite . while such benchmarking provides an initial assessment of models, it often involves a complex and time-consuming process to select a suitable model for a given prompt. a primary challenge with conventional benchmarks lies in their reporting of aggregated performance scores, derived from a static corpus of domain-specific prompts. such global metrics can be unreliable for guiding selection over queries that substantially diverge from the evaluation samples. moreover, model proficiency often varies considerably across different prompts even within the same domain @xcite . typical benchmark outputs tend to obscure these instancelevel performance nuances, thereby failing to provide the granular detail essential for precise, queryspecific model selection. furthermore, the per-ceived capabilities of llms can be skewed by inherent biases and sensitivities within benchmarks, resulting from particular prompt structures @xcite or the methodologies behind leaderboard rankings @xcite . beyond these limitations in granularity, relying on isolated benchmarks does not assess the collective insights available from the broader landscape of evaluation tools. performance prediction methods aim to predict models' performance on unseen prompts and tasks based on prior information. while such methods similarly utilize benchmarks, they differ in explicitly estimating models' performance over given queries, and aim to be applicable in scenarios where obtaining queries' labels is expensive or impractical. these approaches are particularly relevant to out-of-sample (oos) settings, where the queries originate from datasets entirely unseen by the performance prediction estimators. such settings introduce an additional layer of complexity, as generalization to unknown datasets is highly challenging. a recent work has suggested the approach of llm embeddings for performance prediction and subsequent model selection @xcite . this approach aims to represent both llms and prompts in a joint space. the performance estimation is then computed via the corresponding embeddings of the model and query. however, current approaches utilize a distinct representation space that depends on their training data, thereby requiring costly retraining to include additional benchmark results. in this context, we denote the setting of real-time success prediction and subsequent model selection as aiming to apply to newly published models with minimal delay. our work builds on the promising direction of using llm embeddings for performance prediction. we aim to represent llms as linear operators within the prompts' semantic task space, thus providing a highly interpretable representation of the models' application. we consider models' application on queries as semantic-space translations from input to output. we then utilize a closed-form computation to represent the difference between the model's induced and the desired translation, which produces the corresponding label. our approach is training-free, requires negligible computational resources, and can be adapted to additional benchmarks in real-time. moreover, we produce task-oriented embeddings with clear semantic interpretations relevant to diverse downstream tasks and oos scenarios. below we outline our main contributions: ## related work previous llm performance prediction works present various settings and approaches. some works aim to analyze the behavior and performance of given models rather than directly predict their performance over given queries. a common setting discusses the assessment of model performance via per-sample output analysis, where approaches leverage confidence scores @xcite , self-correction capabilities @xcite , or transferability estimation @xcite @xcite . another seeks to predict the broader capabilities of models through statistical analysis @xcite , or by deriving scaling laws from pretraining data @xcite . although such approaches provide important perspectives, they are typically not applicable to the scope of this work. another set of methods trains auxiliary models to predict performance, for example, by training an assessor model which uses an llm's results on fixed set of few reference prompts alongside target prompt intrinsic features to minimize evaluation costs @xcite , or by applying collaborative filtering to learn latent model and task factors from historical performance metadata @xcite @xcite . these methods primarily contribute a trained predictive model designed to operate with specific inputs for the llm in question. our work, however, has a different underlying framework and is focused on deriving pre-computed, explicit vector representations for each llm within an established library, using its comprehensive performance profile on source datasets. the line of research most pertinent to our objective of creating explicit model representations from performance data involves learning joint embeddings for llms and prompts. em-bedllm @xcite stands out as a key contribution in this domain. it employs an encoderdecoder architecture to learn informative representations from a large dataset of model-prompt interactions. while this work presents a significant step towards real-time performance prediction and subsequent model selection, it requires costly retraining to encompass additional models and benchmarks. moreover, embedllm's representations utilize an arbitrary space that lacks semantic grounding. our approach builds on the promising direction of using llm embeddings for performance prediction, but aims to be dynamically expanding while representing llms within the prompts' semantic task space. ## method we now detail our approach to creating linear, interpretable, and scalable representations for llms. we enable efficient performance prediction and model selection by embedding each llm as a vector aligned with the prompts it successfully computes. formally, our goal is to derive a linear representation e(m) @xmath0. this embedding e(m) i is conceptualized as a vector in the d promptdimensional prompt embedding space. specifically, e(m) i represents the "success hyperplane normal", namely a normal to a model-specific hyperplane that ideally separates between prompts where model m i succeeds from those where it fails. the orientation of e(m) i thus signifies a direction in the prompt space associated with success for that particular model. consequently, the success of model m i on a prompt q is estimated by: here, e(q) ∈ r dprompt is the vector embedding for a given target prompt q, generated using the same pre-trained sentence transformer as the source prompts. conceptually, an llm's success on a given prompt is a highly complex function of that prompt's semantic embedding, f (e(q)). our method does not attempt to model f in its entirety. instead, we seek the best linear operator, represented by the vector e(m) i , that approximates the average outcome of this function with respect to the success-failure dichotomy. this approach is predicated on the well-established property of high-dimensional embedding spaces where semantic relationships can be represented as linear vector operations, a principle first established for word vectors @xcite and since extended to produce robust sentence-level semantic representations @xcite . the strong empirical success of our method suggests that this first-order linear approximation is sufficient to capture the most significant variance in model performance, offering a favorable trade-off between model fidelity and the exceptional scalability our approach provides. due to the linearity of our approach, the aggregate success score for a model on a benchmark can be efficiently computed by averaging the embeddings of the benchmark's prompts to form a single benchmark vector, then taking its dot product with the model's embedding. ## experiments this section presents a comprehensive empirical evaluation of our method over the tasks of realtime success prediction and subsequent model selection. we first present the experimental setting in section 4.1, and continue to discuss the results in section 4.2. in addition, we present an ablation study of our method in supplementary appendix a. our evaluation aims to answer three key research questions: ## limitations this work proposes representing llms as embeddings in semantic task spaces, where the embedding encodes their corresponding performance. the models' embedding then corresponds to successfully answered queries and enables an efficient and explainable search of well-performing models. we can then consider the semantic vectors of models as representing their corresponding semantic translation between inputs and outputs. however, our suggested representation only regards a single task and its corresponding information, which may be insufficient to represent the complex semantic translation of llms. moreover, our representation does not consider the models' architectures or inference complexity in any way, and future work could extend it to also consider model efficiency in addition to performance.