Extreme Fine-tuning: A Novel and Fast Fine-tuning Approach for Text Classification (2024)

Abstract: Although fine-tuning a pre-trained model with a conventional approach has shown to be effective in various downstream tasks, previous work has used only backpropagation to fine-tune the model, which causes a massive amount of computational resources and time. We propose Extreme Fine-Tuning (EFT), a novel approach for fine-tuning a pre-trained model effectively and efficiently. EFT uses backpropagation for a brief fine-tuning and an iterative extreme learning machine for training a classifier. We applied EFT to four text classification datasets, MELD, IEMOCAP, IMDb, and AG News, and compared its performance with state-of-the-art (SOTA) approaches. The results indicate that EFT noticeably outperformed the other approaches in training-time measurement with comparable model performance. We will release our code at https://github.com/up-33/extreme-fine-tuning.

## introduction artificial neural networks (anns) have been successfully applied to many tasks such as natural language processing (nlp) @xcite @xcite and computer vision @xcite . one of the essential components under their hood is backpropagation (bp), a gradient-descent-based learning algorithm. the bp became the conventional approach to train an ann model due to its ability to learn sophisticated patterns from a large amount of data @xcite . however, training a model on large-scale data from scratch requires massive computational resources @xcite . pre-trained models (ptms), e.g., bidirectional encoder representations from transformers (bert) @xcite , have been proposed to acquire a huge amount of general knowledge from largescale data. adding a fully connected (fc) layer as the last layer of the pre-trained model enables finetuning to specific tasks. in fine-tuning, the layer can be modified for specific tasks. its parameters are adjusted to minimize task-specific loss for accurate prediction @xcite . however, it usually applies iterative weight updates through the bp, which consumes unnecessary computational resources, particularly for large and deep anns @xcite . apart from training the fc layer through the bp, an extreme learning machine (elm), an ann training framework, was proposed by @xcite to accelerate the training of a single hidden layer feedforward neural network (slfn). while the bp offers flexibility, the elm provides simpler and more computationally efficient solutions. it calculates weights based on another arbitrary weights, making it faster than the bp @xcite . variants such as a constrained elm (celm) @xcite and an iterative elm (i-elm) (jiaramaneepinit and watchareeruetai, 2018) have been also developed to improve performance and reduce memory consumption. to the best of our knowledge, most studies have used only the bp to transfer or fine-tune a ptm to a specific task @xcite @xcite @xcite @xcite . this makes the training process consume an enormous amount of computational resources and time. thus, we propose extreme fine-tuning (eft), a novel finetuning approach that keeps up model performance and improves training efficiency. eft utilizes the bp and i-elm to speed up fine-tuning, improving training performance comparing with other finetuning approaches. we conducted experiments for measuring model performance and training efficiency through text classification tasks. our contributions are as follows: 1) we propose eft that speeds up the conventional way to fine-tune ptms by adopting i-elm to replace a bp-based fc layer for text classification tasks. 2) applying eft to four datasets for text classification produced comparable results compared with the previous studies @xcite @xcite @xcite . nonetheless, eft requires noticeably less training time. ## conclusion we proposed eft, a novel approach for fine-tuning a pre-trained model effectively and efficiently. we showed our eft demonstrates shorter training time with competitive performance than the current sota models. these results highlight the potential of eft as promising options for various nlp tasks, offering a favorable balance between model performance and efficiency. the potential of eft is not limited to text classification but also extends to other classification and even non-classification tasks, such as generation tasks. it would be interesting to delve into the capability of eft in such other fields.