Pruning General Large Language Models into Customized Expert Models (2025)

Abstract: Large Language Models (LLMs) have transformed natural language processing, yet their substantial model sizes often demand significant computational resources. To preserve computing resources and accelerate inference speed, it is crucial to prune redundant parameters, especially for experienced users who often need expert models tailored to specific downstream scenarios. However, current pruning methods primarily focus on maintaining models’ general capabilities, either requiring extensive post-training or performing poorly due to coarse-grained pruning. In this work, we design a \underline{Cus} tom \underline{Prun} ing method ( \texttt{Cus-Prun} ) to prune a large general model into a smaller lightweight expert model, which is positioned along the “language”, “domain” and “task” dimensions. By identifying and pruning irrelevant neurons of each dimension, \texttt{Cus-Prun} creates expert models without any post-training. Our experiments demonstrate that \texttt{Cus-Prun} consistently outperforms other methods, achieving minimal loss in both expert and general capabilities across various models from different model families and sizes.

## introduction large language models (llms) @xcite @xcite have revolutionized the field of natural language processing (nlp), emerging as powerful tools with widespread applications across various languages @xcite @xcite , domains @xcite @xcite , and tasks @xcite @xcite . however, the impressive performance of llms often comes at the cost of immense model sizes, mostly containing billions of parameters and thus demand significant computing resources @xcite . to address this issue, researchers have recently proposed various model pruning methods for llms. these methods aim to reduce model parameters while maintaining the overall performance through techniques such as removal of unimportant structures @xcite @xcite , matrix approximation @xcite , and extensive post-training after pruning @xcite . most existing pruning methods focus on preserving the general capabilities of the model, often evaluated by broad-spectrum benchmarks like mmlu @xcite . while aiming for overall versatility, they may not align well with real-world user needs for a pruned small model, which are usually more specific and targeted. for instance, a user might require a question-answering model tailored specifically for the education domain in german. such specialized request in fact aligns well with the fundamental motivation behind pruning: to create a smaller model by eliminating unnecessary parameters. in this context, the notion of "unnecessary" parameters becomes more precise-referring to those parameters that are irrelevant to the particular use case. by selectively pruning these redundant parameters, one can construct a smaller, expert model that is better aligned with specific requirements. however, current pruning techniques focusing on preserving the general capabilities of llms often employ coarse-grained approaches such as removing entire layers or modules @xcite @xcite . therefore, it may remove parameters that are critical for specialized downstream scenarios, and thus sometimes require extensive post-training to recover the pruned capabilities @xcite . on the other hand, the preserved certain capabilities may not be useful or relevant to our desired application scenario. this misalignment with practical requirements, where users need compact ready-to-deploy expert models without retraining, motivates us to design a more fine-grained and expert model targeting approach. in this work, we introduce a novel custom pruning (cus-prun) method, designed to prune a large general-purpose model into a small specialized expert model tailored for specific scenarios. to achieve broad adaptability, we define the expert model by positioning the target user's needs along three key dimensions: language (e.g., english, chinese, german), domain (e.g., e-commerce, education), and task (e.g., qa, summarization). inspired by existing studies that certain neurons are responsible for certain functions @xcite @xcite , cus-prun identifies and preserves critical neurons that are more relevant to particular languages, domains, or tasks, while pruning less relevant ones, ultimately leading to a smaller expert models. specifically, as illustrated in figure 1 , cus-prun first identifies irrelevant neurons for each dimension by assessing the impact of their removal on the generated output when processing corresponding corpus, which could be easily constructed from the relevant plain text documents. next, the expert model is constructed by pruning irrelevant neurons across all dimensions. furthermore, cus-prun's flexibility allows it to focus on one, two, or all three dimensions (language, domain, task) as needed, making it adaptable to a wide range of real-world applications where specialized llms are required. importantly, by performing fine-grained pruning at the neuron level, the method could also retain most of the essential neurons within the model backbone, thereby preserving most general capabilities. we conduct comprehensive experiments to evaluate the performance of cus-prun across various scenarios. experimental results demonstrate that it consistently outperforms other pruning methods in all settings. for three-dimensional specific expert models, cus-prun demonstrates remarkable pruning effectiveness across different model families and sizes, such as mistral-nemo-12b, llama3-8b, llama2-13b, and llama3-70b, while maintaining strong expert and general capabilities. the method is evaluated across multilingual, multidomain, and multitask datasets, as well as representative compound nlp benchmarks, and consistently outperforms state-of-the-art pruning methods. moreover, cus-prun is highly adaptable, working effectively across a wide range of pruning ratios, even up to nearly half the parameters, without compromising its superior performance. for more focused applications, such as two-or one-dimensional specific expert models (e.g., language-domain specific or languagespecific models), cus-prun continues to significantly outperform other pruning methods, showcasing its versatility and effectiveness in diverse specialized settings. ## custom pruning (cus-prun) an expert model could be generally positioned from three dimensions: "language" (@xmath0, "domain" (@xmath1, and "task" (@xmath2, which can be represented as llm exp := (l, d, t ) ∈ l×d×t. specifically, the language dimension encompasses various languages such as english, spanish, and thai. the domain dimension covers different fields like finance, legal, and medical. the task dimension includes various applications such as questionanswering, data-to-text, and summarization. in this section, we propose a custom pruning method named cus-prun to derive smaller expert models with flexible customization granularity. ## preliminary evaluation in this section, to simulate realistic user-defined requirements where all three dimensions (language, domain, task) are explicitly specified, we conduct preliminary experiments on developing expert models with fine-grained pruning. this setting not only validates our approach under the most demanding conditions but also serves as a prototype for later experiments with one-or two-dimensional customizations. experiment design we consider three scenarios, each named according to the language-domain-task pattern: korean-legal-summarization @xcite , english-medical-mcq (garcía-ferrero et al., 2024), and chinese-e-commerce-sentiment analysis @xcite . these scenarios were selected because each represents a unique combination of language, domain, and task, effectively simulating the diverse needs a user might specify in practice. for each scenario, we curate the corresponding corpus for each dimension. this curation can be done through manual collection or by automatically retrieving relevant documents online. in this preliminary study, without loss of generality, we employ a strong proprietary model foot_1 to generate a corpus containing 50 documents for each dimension. detailed prompts can be found in appendix a.1. the generated documents could then be used to determine the relevance of neurons for each dimension of each scenario. experiment setup we use llama3-8b @xcite as the original dense model and follow the typical setting to set the pruning ratio as 25%. performance is evaluated using rouge-l @xcite for korean-legal-summary and accuracy score for another two tasks. for comparison, we adopt slicegpt (ashkboos et al., 2024) as a baseline, which replaces each weight matrix with a smaller proxy matrix without considering the specific use-case requirements. ## conclusion llms deliver impressive capabilities but incur high computational costs. efficient pruning of redundant parameters is vital for conserving resources and improving inference speed, especially for specialized models. our method, cus-prun, generates smaller expert models without post-training by pruning irrelevant neurons across "language", "domain", and "task" dimensions. this finer-grained approach outperforms existing techniques on threedimensional models and can adapt to realistic scenarios, such as language-domain or languagespecific models.