Does Fine-tuning a Classifier Help in Low-budget Scenarios? Not Much (2024)

Abstract: In recent years, the two-step approach for text classification based on pre-training plus fine-tuning has led to significant improvements in classification performance. In this paper, we study the low-budget scenario, and we ask whether it is justified to allocate the additional resources needed for fine-tuning complex models. To do so, we isolate the gains obtained from pre-training from those obtained from fine-tuning. We find out that, when the gains from pre-training are factored out, the performance attained by using complex transformer models leads to marginal improvements over simpler models. Therefore, in this scenario, utilizing simpler classifiers on top of pre-trained representations proves to be a viable alternative.

## introduction in the past few years, a dominating paradigm has emerged in text classification, primarily centered on a two-step approach: inducing pre-trained weights, followed by task fine-tuning using a transformer model with supervised labeled data @xcite . the new approach has led to significant improvements over previous classification strategies based on simpler linear models trained on sparse bag-of-words feature representations. the improvements observed in performance are often attributed to the induced representation @xcite @xcite . it is not surprising that leveraging contextual continuous word embeddings can lead to improvements by mitigating the sparsity issues of classical bag-of-words representations. at the same time, we expect that richer transformer architectures would enhance classification performance during fine-tuning. however, if the representation is already strong enough, is it justified to allocate additional resources for fine-tuning to achieve satisfactory results? when the same architecture is shared for both pre-training and fine-tuning @xcite , it becomes challenging to disentangle the relative influence of the representation and the classifier. to isolate the performance of each component, we propose an empirical study where we train both simple linear models and complex transformer models, with and without pretrained representations, and test their performance in high and low annotation budget scenarios. we specifically focus on investigating the previous question within the context of a low annotation budget scenario, where the availability of labeled data for fine-tuning is limited. our empirical study shows that: ## related work while transformer @xcite architectures are known to benefit from large amounts of training data for optimal performance (ezen-can, 2020; @xcite , the pre-training plus fine-tuning approach has also shown promising results in low annotation budget scenarios @xcite @xcite . fine-tuning is thought to adjust the pre-trained representations in order to simplify the downstream task @xcite . however, the fine-tuning step itself can be unstable @xcite and sensitive to weight initialization @xcite . these issues are particularly pronounced in low-budget scenarios @xcite . to address these challenges, researchers have explored techniques such as parameter reduction @xcite @xcite or modifications to the fine-tuning procedure @xcite . other authors have explored the possibility of using pre-trained representations directly with simpler classifiers @xcite the importance of representation choice has lately received a significant amount of attention from the active learning (al) community @xcite @xcite @xcite @xcite . most of the research in al attempts to quantify what representation is best when training the initial model for active learning, which is usually referred to as the cold start problem @xcite . the importance of word embeddings has also been studied in the context of highly imbalanced data scenarios @xcite @xcite . the main difference between our work and previous literature is that in prior studies, the fine-tuning process involved the simultaneous updates of both the pre-trained weights and the classifier, without considering their relative importance. having established the relevance of the representation, especially in few-shot learning scenarios, we aim to investigate whether fine-tuning complex architectures in classification tasks is justified. ## the role of the classifier in low-budget scenarios to conduct our study, we aim to compare the performance of a transformer-based model and a simple classifier, trained with and without pre-trained representations. the main focus of our investigation will be on scenarios with a limited annotation budget, by utilizing learning curves. each point in ## results figure 1 shows our main results on analyzing the performance of models under the low-budget annotation setting. to summarize the learning curve results, we also compute a single performance score for each model: the area under the learning curve (alc). this provides us with a more robust metric to compare the different models for a dataset 1 . we observe that in the low-budget scenario when pre-trained representations are used, the choice of model seems to be of little importance. both the complex transformer model and a simple linear max-entropy model perform similarly. in addition, when only very few labels are available (first curve points in figure 1 ), the simpler model seems to outperform the more complex one. maxent demonstrates a more stable behavior the biggest difference in performance in the lowbudget scenario comes from the representation and not the architecture. in fact, without pre-trained representations, the more complex models perform significantly worse than simpler models. pre-trained representations seem to be capturing some properties of the input space that can be exploited by all models. we suppose that since pre-training implicitly induces a distance space over words, models using pre-trained representations generalize more easily to unseen words. this would explain why pre-trained representations are especially helpful in the low-annotation budget scenario since generalization to unseen words is critical in this case. ## conclusion in this paper, we studied classifiers in a low-budget scenario, analyzing the impact of fine-tuning on performance by separating the benefits derived from pre-training weights from those of architectural fine-tuning. based on our findings, we recommend testing simple models that incorporate pre-trained representations before investing resources in fine-tuning complex models. in fact, when labeled data is scarce, the role of the representations is crucial, and the use of pre-trained representations enhances performance across all models, regardless of their complexity. as a result, the choice of classifier becomes irrelevant in this context compared to the quality of the representations. the marginal performance gains offered by more sophisticated architectures may not justify the additional computational resource demands.