LPZ ero: Language Model Zero-cost Proxy Search from Zero (2024)

Abstract: Despite the outstanding performance, Neural Architecture Search (NAS) is criticized for massive computation. Recently, Zero-shot NAS has emerged as a promising approach by exploiting Zero-cost (ZC) proxies, which markedly reduce computational demands. Despite this, existing ZC proxies heavily rely on expert knowledge and incur significant trial-and-error costs. Particularly in NLP tasks, most existing ZC proxies fail to surpass the performance of the naive baseline. To address these challenges, we introduce a novel framework, LPZero, which is the first to automatically design zero-cost (ZC) proxies for various tasks, achieving higher ranking consistency than human-designed proxies. Specifically, we model the ZC proxy as a symbolic equation and incorporate a unified proxy search space that encompasses existing ZC proxies, which are composed of a predefined set of mathematical symbols. To heuristically search for the best ZC proxy, LPZero incorporates genetic programming to find the optimal symbolic composition. We propose a Predictive-Pruning Strategy (PPS), which preemptively eliminates unpromising proxies, thereby mitigating the risk of proxy degradation. Extensive experiments on FlexiBERT, GPT-2, and LLaMA-7B demonstrate LPZero’s superior ranking ability and performance on downstream tasks compared to current approaches.

## introduction traditional neural network design @xcite , heavily dependent on expert knowledge and experience @xcite , is both timeintensive and prone to trial-and-error. neural architecture search (nas) emerged to automate and refine this process by identifying optimal architectures from a set of possibilities using various strategies. however, early nas methods @xcite require extensive computation, which limits their wide accessibility. for ## related work zero-shot nas has gained prominence as a computation-efficient alternative to previous nas methods @xcite @xcite . it can estimate the performance of candidate architectures without extensive training. existing zc proxies rely heavily on experts and handcrafted heuristics. for instance, nwot @xcite leverages the local jacobian values across various images to construct an indicator for the model's capability. zennas @xcite assesses candidate architectures by employing the gradient norm of input images. zerocost nas @xcite introduces pruning-based metrics as zc proxies, which encompass indicators including gradnorm @xcite , snip @xcite and synflow @xcite , etc. these proxies evaluate the significance of network parameters and aggregate layer-wise values to estimate the overall performance. the above proxies mainly focus on convolution-based networks, recent efforts (serianni and kalita, 2023) first apply zc proxies to transformer-based networks, including rnn and bert, and propose the flexibert benchmark. litetransformersearch @xcite proposes to employ the number of decoder parameters as zc proxies on the gpt-2 benchmark. automatic design for zc proxies. several studies explore how to search for zc proxies automatically, notably eznas @xcite and emq @xcite . eznas introduces a proxy search space dedicated to convolutionbased networks, achieving commendable performance across various benchmarks @xcite . however, its effectiveness is notably diminished when applied to transformer-based networks. on the other hand, emq @xcite develops a specialized proxy search space tailored for mixed-precision quantization proxies for convolution-based networks. our lpzero framework can be applied to transformer-based architectures, particularly language models, and shows superior and more promising performance. nas for llms. large language models (llms), such as llama @xcite , are becoming increasingly large, with model sizes ranging from 7b to 70b @xcite . this rapid growth in model scale poses challenges for directly applying supernet-based nas methods @xcite to llms. to address this issue, recent works, including lonas (munoz et al., 2024) and llama-nas (sarah et al., 2024), leverage elastic low-rank adaptation (lora) (hu et al., 2022) to transform a pre-trained llm into a supernet. this approach enables the practical application of nas techniques to llms by reducing the search space and computational requirements. in this paper, we further enhance the efficiency of the sub-network search process by employing lpzero as a cost-effective performance estimator. ## experiments in this section, we first detail the experimental setup and implementation details of lpzero. then, we present the ranking correlation evaluation on flexibert and gpt-2 benchmark. subsequently, we assess lpzero's performance by examining the ranking correlation in the flexibert and gpt-2 benchmarks. after that, we report the performance on commonsense tasks for llama-7b model. finally, we conduct an ablation study to evaluate the impact of our genetic programming framework, the rule-based pruning strategy (rps), and other components including the number of unary operations and the initial population size. ## conclusion in this paper, we present lpzero, an innovative approach for discovering proxies for language models without training or expert intervention. our lpzero encompasses the design of a comprehensive proxy search space, spanning existing zc proxies. with genetic programming, we efficiently unearth promising zc proxies within this space. to expedite the search, we propose a rule-based pruning strategy, eliminating less promising proxies early in the process. to ascertain the efficacy of lpzero, we conducted experiments on the flexib-ert and gpt-2 benchmarks to evaluate the ranking consistency of the searched proxy, demonstrating the superior ranking capabilities of lpzero. furthermore, we assessed lpzero's performance on commonsense reasoning tasks, where it exhibited commendable results.