D a M o C : Efficiently Selecting the Optimal Large Language Model for Fine-tuning Domain Tasks Based on Data and Model Compression (2025)

Abstract: Large language models (LLMs) excel in general tasks but struggle with domain-specific ones, requiring fine-tuning with specific data. With many open-source LLMs available, selecting the best model for fine-tuning downstream tasks is challenging, primarily focusing on how to quickly identify the optimal LLM. We introduce a Data and Model Compression Framework (DaMoC) that addresses this challenge by: 1) Data Level: A systematic categorization of data filtering methodologies for LLMs is first established, classifying them into three distinct paradigms: (1) distribution-aware methods, (2) quality-aware methods, and (3) hybrid approaches considering both dimensions. Further, we enhance the density of key tokens in the text achieving token compression. Subsequently, we use an LLM to iterative rewrite the text to optimize its expression. 2) Model Level: We use layer similarity scores to assess each layer’s importance and remove those with lower importance. Then, we introduce a sparse merging paradigm to preserve as much of the original model’s capability as possible. Extensive experiments on four datasets, medical Q&A, financial Q&A, general Q&A, and reading comprehension, show that we can select the optimal LLM while saving approximately 20-fold in training time.

## introduction in recent years, llms have rapidly developed, showcasing strong general capabilities after being pre-trained on large-scale datasets, and can handle diverse tasks through few-shot prediction or n-context learning @xcite . however, in specific fields such as law and medicine, the performance of llms is not entirely satisfactory @xcite . to improve the model's performance in these areas, users typically * these authors contributed equally to this work † corresponding author (wyinggui@gmail.com). ## motivation inspired by in-context learning @xcite , we explore whether it is possible to choose the most suitable llm for domain fine-tuning through a few-shot approach without additional training. our experiments involve selected examples from the training data, structured as follows: instruct + example 1 +...+example n +question+answer. it was observed from when fine-tuning llms with domain-specific data, the challenge lies in how to efficiently identify the llms with the highest accuracy. ## efficient llms select framework to quickly and accurately select the most suitable llm for fine-tuning downstream tasks, we propose damoc to address this challenge. damoc encompasses three strategies: 1) data filtering methods, 2) token compression methods, and 3) model pruning methods. in the following sections, we will provide a detailed description. ## related work we classify data filtering approaches into three primary categories. (i). distribution-aware methods prioritize data diversity. for instance, graphcut @xcite seeks to enhance the diversity be-tween samples and the selected dataset while minimizing the distance between samples and the remaining dataset. (ii) quality-aware methods leverage quality assessment models to evaluate the quality of data. for example, superfiltering(sf) @xcite computes the ifd scores of data, subsequently applying filters based on these scores. (iii) hybrid approaches incorporating both dimensions to refine data filtering. an illustrative example is provided by @xcite , which initially evaluates data quality scores and subsequently applies clustering methods to achieve diversity classification of instructions. token compression is predominantly utilized to curtail the textual length of prompts in models. for instance, longllmlingua @xcite designed to address the challenges of noise and positional bias in extended contexts by enhancing the large language model's sensitivity to critical information pertinent to the query. model pruning aims to reduce the size of models by removing parameters deemed non-essential, with a particular focus on pre-trained models. for instance, flap @xcite evaluates the variation of each input feature concerning a baseline value to estimate the consequences of removing columns of weights. ## conclusion we introduce the damoc to address the problem of selecting the optimal llm for fine-tuning. the damoc addresses this challenge from three aspects: 1) data filtering: we investigate the impact of data filtering methods on the selection of the optimal model. 2) token compression: employ perplexity to compress tokens and use iterative rewriting to enhance the expressiveness. 3) model pruning: we combine layer importance and sparse merging to prune the model's layers. the results demonstrate that damoc can select the optimal llm while saving up to 20-fold the training time.