LLM Performance Predictors are good initializers for Architecture Search (2024)

Abstract: In this work, we utilize Large Language Models (LLMs) for a novel use case: constructing Performance Predictors (PP) that estimate the performance of specific deep neural network architectures on downstream tasks. We create PP prompts for LLMs, comprising (i) role descriptions, (ii) instructions for the LLM, (iii) hyperparameter definitions, and (iv) demonstrations presenting sample architectures with efficiency metrics and ‘training from scratch’ performance. In machine translation (MT) tasks, GPT-4 with our PP prompts (LLM-PP) achieves a SoTA mean absolute error and a slight degradation in rank correlation coefficient compared to baseline predictors. Additionally, we demonstrate that predictions from LLM-PP can be distilled to a compact regression model (LLM-Distill-PP), which surprisingly retains much of the performance of LLM-PP. This presents a cost-effective alternative for resource-intensive performance estimation. Specifically, for Neural Architecture Search (NAS), we introduce a Hybrid-Search algorithm (HS-NAS) employing LLM-Distill-PP for the initial search stages and reverting to the baseline predictor later. HS-NAS performs similarly to SoTA NAS, reducing search hours by approximately 50%, and in some cases, improving latency, GFLOPs, and model size. The code can be found at: https://github.com/UBC-NLP/llmas.

## introduction large language models (llms) have diverse applications, encompassing both open-ended tasks (e.g., brainstorming and chat) and closed-ended tasks (e.g., summarization and question answering). this study explores a unique application of llms: constructing a performance predictor (llm-pp) for a deep neural network (dnn) architecture. the predictor takes the dnn architecture description, * work was done while ganesh was at ubc. typically hyperparameters (e.g., #layers, #attention heads), as input and predicts the performance (e.g., bleu score) for a specific downstream task. the aim is to create a performance predictor with low prediction errors compared to training from scratch. the hypothesis is that llms possess a 'general understanding' of dnn architectures, derived from relevant training data like dnn research papers and github repositories. the main objective of this work is to leverage this understanding to design accurate, efficient, and broadly applicable performance predictors, beneficial for tasks like neural architecture search (nas). how to design an accurate performance predictor (pp)? to answer this, we create pp prompts precisely specifying the task. these prompts include: (i) role: high-level description of the assigned llm role, (ii) instructions: detailed task instructions (e.g., downstream task, architecture, performance/efficiency metric) for the llm to follow, (iii) hyperparameters: definitions of architecturespecific hyperparameters, and (iv) demonstrations: supervised examples for the pp task with architecture descriptions and performance metrics (e.g., bleu score). using gpt-4 (openai, 2023a) as our primary llm and wmt datasets for machine translation (mt) tasks, we find that gpt-4 with our pp prompts (llm-pp) predicts architecture performance with a mean absolute error achieving the state-of-the-art (sota) and a slightly lower rank correlation coefficient compared to previous sota weight-sharing supernet-based performance predictors @xcite . using gpt-4 for llm-pp entails utilizing the gpt-4 api to score each architecture, rendering llm-pp prohibitively expensive for various use cases. one example is nas, where pp evaluates approximately 3,000 candidate architectures for each constraint (e.g., @xmath5. as of august 2023, gpt-4 pricing is 0.03@xmath0 for a single constraint on the target hardware. with varying constraint values (e.g., 100ms, 200ms), constraint types (e.g., latency, flops, memory), and target hardware (e.g., nvidia a100, raspberry pi), the cumulative cost can quickly become exorbitant (e.g., 1,800@xmath1 for a single downstream task, amortized across various constraint values, types, and target hardware. can llm-distill-pp speed up architecture search while preserving the efficiency and the quality of sota nas? to answer this, we apply using llm-distill-pp as the pp to design efficient mt architectures via sota nas methods like hat @xcite . we introduce the hybrid-search algorithm (hs-nas), where llm-distill-pp serves as the pp for the first 15 search iterations, and a weight-sharing supernet (sota performance predictor) takes over for the remaining 15 iterations. hs-nas achieves roughly 50% faster search than sota nas, maintaining or improving on the performance of architectures designed by sota nas. in some cases, it also yields reduced latency (∼2%), flops (∼1%), and model size (∼2%). our main contributions are as follows: additional background on related topics such as llms and distillation can be found in a.1. ## performance prediction problem informally, the performance prediction problem entails providing a dnn architecture description (usually hyperparameters like #layers, #attention heads) to the predictor, which then outputs the performance (e.g., bleu score) for a specified downstream task. an ideal predictor should minimize prediction errors compared to the performance achieved through training from scratch. formally, let t represent a downstream task, a t its search space of architectures, and y t ⊂ r the real space of performance scores. define d t as the data distribution over a t × y t . the performance predictor is denoted by f t : kendall rank correlation coefficient is another metric that computes the ranking correlation between a set of predictions and their corresponding tfs performances, formalized as kendall-tau([f t (a 1 ), . . . , f t (a m )], [p 1 , . . . , p m ]). examples for these metrics are discussed in section a.2. recently, @xcite emphasized the importance of both mae and kendall-tau metrics in evaluating performance predictor quality. for instance, a predictor with a 38% better mae and a 12% worse kendall-tau, compared to a base predictor, led nas to find an architecture with a 4% bleu improvement. conversely, a predictor with a 5% worse mae and a 6% higher kendall-tau resulted in a nas architecture with a 0.1% bleu improvement. hence, better mae and better kendall-tau are positively correlated with higher-quality architecture. ## baseline performance predictors in nas for nlp literature, the sota method for constructing performance predictors (f t ) involves training a weight-sharing supernet model on task t . simply put, a weight-sharing supernet model is the largest model in the search space, capable of parameterizing all architectures via weight sharing. the parameters for a specific architecture are obtained by extracting the relevant rows and columns from the supernet model's weight matrix. typically, the supernet is trained by iteratively sampling an architecture from the search space and training the extracted weights for that architecture. formalizing the supernet's training objective: denote the training data distribution as x train . represent the training sample and label as x and y, where x, y ∼ x train . a rand is a uniformly sampled architecture from the search space a t . a large and a small denote the largest and smallest architectures in a t . the subnet with architecture a is denoted by s a , parameterized by the supernet model weights w . the training objective of the supernet using sandwich sampling @xcite is given by hardware-aware transformers @xcite employs single-path one-shot (spos) optimization @xcite , focusing on optimizing only a rand at each training step. mixture-of-supernets @xcite ) (mos) utilizes mixture-of-experts (moe) @xcite to enhance the supernet's capacity, with the router specializing weights for each architecture. mos comes in two variants: layer-wise mos and neuron-wise mos, differing in the degree of freedom for weight generation. both variants of mos employ sandwich sampling for supernet training. ## llm performance predictor (llm-pp) llm demonstrates a "general understanding" of dnn architectures, likely acquired through training on relevant data sources like research papers and github repositories. testing these architecture understanding capabilities involves prompting llm to generate hyperparameter definitions and design principles for architecture search @xcite . these llm capabilities contribute to effective performance prediction by aiding the mapping of dnn architectures to their performances. to this end, we propose the llm-based performance predictor (llm-pp), which involves you are a performance estimator for machine translation task, where you will estimate the bleu score for the test architecture. you should follow these instructions: 1. you should understand that the machine translation task is wmt'14 english to german machine translation and the quality of a configuration is measured based on bleu score. 2. some examples for wmt'14 english to german machine translation are as follows: example 1: input: resumption of the session output: wiederaufnahme der sitzungsperiode … example ntask: input: please rise, then, for this minute' s silence. output: ich bitte sie, sich zu einer schweigeminute zu erheben. 3. you should understand that the backbone architecture is from ''attention is all you need'' @xcite paper, which is a transformer based encoder-decoder architecture. we use the same hyperparameters and optimization algorithms. 4. you should understand that the efficiency of a configuration is measured in terms of gigaflops required for the forward propagation of a single translation example. 5. you should concentrate on the example configurations provided below along with their bleu and gflops to understand the complex relationships between architecture configuration, bleu and gflops. hyperparameter definition: 'encoder-embed-dim-subtransformer' corresponds to encoder embedding dimension 'encoder-layer-num-subtransformer' corresponds to number of encoder layers 'encoder-ffn-embed-dim-all-subtransformer' correspond to embedding dimension of each ffn layer in encoder … example 1: encoder-embed-dim-subtransformer: 512 encoder-layer-num-subtransformer: 6 encoder-ffn-embed-dim-all-subtransformer: @xcite 1024] … bleu: 24.30 gflops: 2.7 ... example narch: … test architecture: encoder-embed-dim-subtransformer: 640 encoder-layer-num-subtransformer: 6 encoder-ffn-embed-dim-all-subtransformer: [2048, 1024, 1024, 1024, 2048, 1024] … bleu: role instruction hyperparameters demonstrations test figure 1: prompt template to prompt llm to generate performance predictions for wmt'14 en-de task. the expanded version of the prompt template can be seen in appendix a.3. the role specifies the llm's role, describing the downstream task (e.g., machine translation) and the performance metric (e.g., bleu). the instructions provide five detailed instructions covering the downstream task, dnn architecture, and model efficiency metrics. the first two focus on the task specifics, specifying the task type (e.g., machine translation), dataset (e.g., wmt'14 en-de), performance metric (e.g., bleu), and inputs/outputs (e.g., source/target language) for n task examples from the dataset. the third instruction details the dnn architecture, including backbone (e.g., transformer), type (e.g., encoder-decoder), and a reference to the original dnn paper. the fourth instruction outlines efficiency metrics details (e.g., gflops), included in the demonstrations. the final instruction directs the llm to consider complex relationships between architecture configuration, performance, and efficiency metric. the third component, hyperparameters, defines architecturespecific hyperparameters. demonstrations is the final component containing n arch supervised examples, each representing an architecture from the search space with hyperparameter values, efficiency score, and tfs performance score. the design process of the llm-pp prompt is discussed in a.4. ## distillation of llm-pp to illustrate the cost, let's consider the example of nas run by hat @xcite for a latency constraint on a given hardware, involving the evaluation of approximately 3,000 candidate archi-tectures. as of august 2023, the pricing for gpt-4 is 0.03@xmath2 ( 0.03 * 3000 3 ). the total cost depends on the number of constraint types (e.g., latency, memory, flops), values (e.g., 100ms, 200ms), and hardware options (e.g., nvidia a100, raspberry pi). for instance, with three constraint types, five values for each constraint, and four target hardwares, the estimated cost could soar to approximately 1, 800@xmath3 ( 0.03 * 3000 3 ). this one-time cost of llm-distill-pp is amortized across different constraint types, values, and hardwares (e.g., 60 search runs), leading to a substantial 98.3% reduction in cost (from 1, 800@xmath4). llm-distill-pp achieves a superior efficiency-accuracy tradeoff, offering comparable accuracy to llm-pp but with significantly faster prediction times (0.01s vs. 11.9s), as detailed in a.10.1. setup. llm-distill-pp's architecture encoding details can be found in a.6.3. the hyperparameters of its regression model, borrowed from hat's latency predictor, include 3 hidden layers, 400 as the hidden dimension, 128 as the batch size, 1e-5 as the learning rate, and 5000 as the number of training steps. distillation from llm-pp uses only 3000 architecture examples for each downstream task. results. llm-distill-pp's results are summarized in the third major row of ## llm-distill-pp for architecture search given llm-distill-pp's ability to achieve highperformance prediction quality in a cost-effective manner, we explore its application in a real-world task: nas. in nas, performance predictors typically rank candidate architectures to identify highperforming ones. as discussed in section 2, existing nas research in nlp primarily uses weight-sharing supernets as performance predictors. therefore, we address the research question: can llm-distill-pp accelerate architecture search while maintaining the efficiency and quality of sota nas? to answer this question, we introduce the hybrid-search algorithm for nas (hs-nas). the core idea of hs-nas is to employ llm-distill-pp for a subset of search iterations, utilizing the supernet for the remaining iterations. this approach is applied to the evolutionary search algorithm proposed in hat. llm-distill-pp will be used as performance predictor for all the search iterations in between llm-start-iteration and llm-end-iteration. in rest of the iterations, supernet will be used as performance predictor. algorithm 1 hybrid-search algorithm for neural architecture search (hs-nas). changes to hat's search algorithm are in red color. the expanded algorithm can be found in a.9. input: llm-distill-pp model: llm-distill-pp, weight-sharing supernet: supernet, latency predictor: latency-predictor, #search iterations: num-iterations, population size: population-size, latency constraint: latency-constraint, llm-distill-pp start iteration: llm-start-iteration, llm-distill-pp end iteration: llm-end-iteration, ... output: best-architecture 1: popu ← population-size rand. samples from search space // create init. population 2: for iter ← 1 to num-iterations do 3: // gen. parents by picking top cand. arch. 4: if llm-start-@xmath6: parents ← top 'num-parents' arch. from popu by llm-distill-pp 6: else 7: parents ← top 'num-parents' arch. from popu by supernet 8: mut-@xmath7: cross-@xmath8: for all our search experiments, we use llm-distill-pp gpt-4 as llm-distill-pp due to its superior performance over the chatgpt counterpart (see the third major row in ## conclusion this work shows that llms can be employed to create accurate and cost-effective performance predictors, providing insights into enhancing nas. this contribution adds to the expanding field of llms in nas, suggesting future research directions in adapting llms for both candidate architecture generation and joint performance prediction.