A uto ML Meets Hugging Face: Domain-Aware Pretrained Model Selection for Text Classification (2025)

Abstract: The effectiveness of embedding methods is crucial for optimizing text classification performance in Automated Machine Learning (AutoML). However, selecting the most suitable pre-trained model for a given task remains challenging. This study introduces the Corpus-Driven Domain Mapping (CDDM) pipeline, which utilizes a domain-annotated corpus of pre-fine-tuned models from the Hugging Face Model Hub to improve model selection. Integrating these models into AutoML systems significantly boosts classification performance across multiple datasets compared to baseline methods. Despite some domain recognition inaccuracies, results demonstrate CDDM’s potential to enhance model selection, streamline AutoML workflows, and reduce computational costs.

## introduction the advent of large language models (llms) has significantly advanced natural language processing (nlp), offering powerful tools for tasks such as text classification, summarization, and translation @xcite . fine-tuning these models for specific tasks has traditionally been the standard approach to achieving optimal performance. however, fine-tuning is resource-intensive, requiring substantial computational power and time, which may not be feasible for all practitioners @xcite . simultaneously, automl automates tasks like feature and model selection, offering a streamlined approach to machine learning @xcite . integrating llms into automl can boost nlp performance by leveraging their rich linguistic representations @xcite . a practical alternative to fine-tuning is utilizing pre-fine-tuned llms available in repositories such as hugging face. these models have been trained on specific tasks or domains and offer ready-to-use llms that can be incorporated into automl classifiers. this approach can improve performance while mitigating the resource constraints associated with fine-tuning. despite their potential, pre-fine-tuned llms from repositories like hugging face remain underexplored as text representation methods in au-toml. this study bridges this gap by developing an interface to a domain-annotated corpus of prefine-tuned models and evaluating their impact on classification performance across seven diverse text classification tasks. this study enhances automl-based text classification by introducing a structured corpus of prefine-tuned models annotated with domain-specific metadata to optimize model selection. by systematically mapping models to tasks based on domain alignment, we demonstrate substantial performance gains while reducing computational overhead. the findings highlight a scalable and resource-efficient approach for integrating pretrained representations into automl frameworks, making advanced nlp capabilities more accessible. text representations in automl: automl frameworks like auto-pytorch aim to automate feature extraction, model selection, and hyperparameter tuning @xcite . despite this, they often rely on basic text representations like one-hot encoding. recent research highlights the benefits of integrating advanced embeddings into automl systems. for instance, @xcite demonstrated the effectiveness of fine-tuned bert embeddings for binary classification in auto-pytorch. however, leveraging pre-fine-tuned llms for au-toml remains underexplored. ## methodology in order to achieve our two goals of interfacing and selecting pre-fine-tuned models, we implement the following two phases. ## experiment the experimental workflow evaluated the utility of pre-fine-tuned language models from hugging face for diverse text classification tasks. the process involved multiple steps, including collecting 3 model metadata, domain recognition, dataset preparation, model selection, and evaluation. below, we detail each step of the experimental setup. ## results and discussion the results of our evaluation, presented in ## conclusion and future works this study introduced a corpus of pre-fine-tuned models from hugging face enriched with domain annotations and dataset descriptions, demonstrating its utility for enhancing text classification tasks. the experimental results highlight how this resource improves model performance and streamlines integration into automated pipelines, reducing the need for fine-tuning. in conclusion, the hugging face corpus represents a critical step toward scalable and efficient nlp solutions. refinements in domain recognition and alignment hold the potential to revolutionize the use of pre-fine-tuned models in automl, advancing text classification and broader nlp tasks. future work will focus on improving domain recognition accuracy through advanced methods such as supervised learning or knowledge graphbased approaches. additionally, it will evaluate the corpus with a more diverse range of datasets, including low-resource languages. 5 future work will optimize text representation methods for specific datasets to enhance the proposed corpus's utility in automl systems. we will develop a multi-model evaluation framework that aligns three semantically similar pre-fine-tuned models from the corpus to each dataset based on domain similarity scores and zero-shot classification results. these models will be assessed using automl techniques supported by auto-pytorch, enabling efficient performance evaluation through automated hyperparameter optimization and model selection. by employing multi-fidelity optimization methods like successive halving and hyperband, we aim to identify the most effective model early in training, reducing computational costs. this method balances model performance with efficiency while preserving the domain-specific strengths of our corpus.