A uto T rain: No-code training for state-of-the-art models (2024)

Abstract: With the advancements in open-source models, training(or finetuning) models on custom datasets has become a crucial part of developing solutions which are tailored to specific industrial or open-source applications. Yet, there is no single tool which simplifies the process of training across different types of modalities or tasks.We introduce AutoTrain(aka AutoTrain Advanced)—an open-source, no code tool/library which can be used to train (or finetune) models for different kinds of tasks such as: large language model (LLM) finetuning, text classification/regression, token classification, sequence-to-sequence task, finetuning of sentence transformers, visual language model (VLM) finetuning, image classification/regression and even classification and regression tasks on tabular data. AutoTrain Advanced is an open-source library providing best practices for training models on custom datasets. The library is available at https://github.com/huggingface/autotrain-advanced. AutoTrain can be used in fully local mode or on cloud machines and works with tens of thousands of models shared on Hugging Face Hub and their variations.

## introduction with recent advancements in open-source and openaccess state-of-the-art models, the need for standardized yet customizable training of models on downstream tasks has become crucial. however, a universal open-source solution for a diverse range of tasks is still lacking. to address this challenge, we introduce autotrain (also known as auto-train advanced). autotrain is an open-source solution which offers model training for different kinds of tasks such as: large language model (llm) finetuning, text classification/scoring, token classification, training custom embedding models using sentence transformers @xcite , finetuning for visual language models (vlms), computer vision tasks such as image classification/scoring, object detection and even tabular regression and classification tasks. at the time of writing this paper, a total of 22 tasks: 16 text-based, 4 image-based and 2 tabular based have been implemented. the idea behind creating autotrain is to allow a simple interface for training models on custom datasets without requiring extensive knowledge of coding. autotrain is intended for not just nocoders but also for experienced data scientists and machine learning practioners. instead of writing complex scripts, one can focus on gathering and preparing your data and let autotrain handle the training part. autotrain ui is shown in figure 1 . when talking about model training, there are several problems which arise: complexity of hyperparameter tuning: finding the right parameters for tuning models can only be done by significant experimentations and expertise. improperly tuning the hyperparameters can result in overfitting or underfitting. ## related work in recent years, many automl solutions have been developed to automate the training process of machine learning models. some notable solutions include: automl solutions autosklearn @xcite , which is an open-source automl toolkit built on top of the popular scikit-learn library. au-tosklearn uses bayesian optimization to automate the process of model selection and hyperparameter tuning. autocompete @xcite , which won the codalab automl gpu challenges, builds a framework for tacking machine learning competitions. the code is, however, not open source. axolotl (cloud, 2024) is a cli tool for finetuning llms. autokeras @xcite , developed on top of keras offers functionalities for various tasks such as image classification, text classification, and regression many other closed-source solutions have also been developed by google, microsoft, and amazon. however, all these solutions have some limitations. they are either not open-source, or if they are, they can only handle a limited number of tasks. many of these solutions are also not no-code, making them inaccessible to non-coders. with autotrain, we provide a single interface to deal with many different data format, task, and model combinations, which depending on user's choices is also closely connnected to hugging face hub which enables download, inference and sharing of models with the entire world. moreover, au-totrain supports almost all kinds models which are compatible with hugging face transformers @xcite library, making it a unique solution to support hundreds of thousands of models for finetuning, including the models which require custom code. ## library: autotrain advanced the autotrain advanced python library provides a command line interface (cli), a graphical user interface (gui/ui) and python sdk to enable training on custom datasets. the datasets can be uploaded/used in different formats such as zip files, csvs or jsonls. we provide documentation and walkthroughs on training models for different task and dataset combinations with example hyperparameters, evaluation results and usage of the trained models. the library is licensed as apache 2.0 license and is available on github, foot_0 making it easy for anyone to adopt and contribute. the design of the library has been made keeping in mind both professionals and amateurs who would like to finetune model but don't know where to start and don't want to invest time setting up a separate environment for each of their finetuning tasks. the library lies on the shoulders of giants such as transformers @xcite , hugging face datasets @xcite , accelerate @xcite , diffusers @xcite , peft @xcite , trl @xcite and other libraries created by hugging face. autotrain uses @xcite as the main backend for training the models. for tabular datasets, models from (van der @xcite and @xcite are used as preferred models. ## conclusion in this paper, we introduce autotrain (aka auto-train advanced), which is an open source, no-code solution for training (or finetuning) machine learning models on a variety of tasks. autotrain addresses common challenges in the model training process, such as dataset processing, hyperparameter tuning, model validation, distributed training, monitoring, and maintenance. by automating these tasks, autotrain ensures that users can efficiently build high-performing models without needing extensive coding knowledge or experience. additionally, autotrain supports a diverse range of tasks, including llm finetuning, text classification, image classification, and regression, and even tabular data classification/regression, thus, making it a versatile tool for various applications.