Cluster & Tune: B oost Cold Start Performance in Text Classification (2022)

Abstract: In real-world scenarios, a text classification task often begins with a cold start, when labeled data is scarce. In such cases, the common practice of fine-tuning pre-trained models, such as BERT, for a target classification task, is prone to produce poor performance. We suggest a method to boost the performance of such models by adding an intermediate unsupervised classification task, between the pre-training and fine-tuning phases. As such an intermediate task, we perform clustering and train the pre-trained model on predicting the cluster labels. We test this hypothesis on various data sets, and show that this additional classification phase can significantly improve performance, mainly for topical classification tasks, when the number of labeled instances available for fine-tuning is only a couple of dozen to a few hundred.

## introduction the standard paradigm for text classification relies on supervised learning, where it is well known that the size and quality of the labeled data strongly impact the performance @xcite . hence, developing a text classifier in practice typically requires making the most of a relatively small set of annotated examples. the emergence of transformer-based pre-trained language models such as bert @xcite has reshaped the nlp landscape, leading to significant advances in the performance of most nlp tasks, text classification included (e.g., @xcite . these models typically rely on pretraining with massive and heterogeneous corpora on a general masked language modeling (mlm) task, i.e., predicting a word that is masked in the original text. later on, the obtained model is fine-tuned to the actual task of interest, termed here the target task, * these authors contributed equally to this work. our work focuses on a challenging yet common scenario, where unlabeled data is available but labeled data is scarce. in many real-world scenarios, obtaining even a couple of hundred of labeled examples per class is challenging. commonly, a target class has a relatively low prior in the examined data, making it a formidable goal to collect enough positive examples for it @xcite . moreover, sometimes data cannot be labeled via crowd-annotation platforms due to its confidentiality (be it for data privacy reasons or for protecting intellectual property) or since the labeling task requires special expertise. on top of this, often the number of categories to be considered is relatively large, e.g., 50, thus making even a modest demand of 200 labeled examples per class a task of labeling 10k instances, which is inapplicable in many practical cases (for an extreme example, cf. @xcite . in such limited real-world settings, fine-tuning a large pretrained model often yields far from optimal performance. to overcome this, one may take a gradual approach composed of various phases. one possibility is to further pretrain the model with the self-supervised mlm task over unlabeled data taken from the target task domain @xcite . alternatively, one can train the pretrained model using a supervised intermediate task which is different in nature from the target-task, and for which labeled data is more readily available @xcite @xcite . each of these steps is expected to provide a better starting point for the final finetuning phase, performed over the scarce labeled data available for the target task, aiming to end up with improved performance. following these lines, here we propose a strat-egy that exploits unsupervised text clustering as the intermediate task towards fine-tuning a pretrained model for text classification. our work is inspired by the use of clustering to obtain labels in computer vision @xcite . specifically, we use an efficient clustering technique, that relies on simple bag of words (bow) representations, to partition the unlabeled training data into relatively homogeneous clusters of text instances. next, we treat these clusters as labeled data for an intermediate text classification task, and train the pre-trained model -with or without additional mlm pretraining -with respect to this multi-class problem, prior to the final fine-tuning over the actual target-task labels. extensive experimental results demonstrate the practical value of this strategy on a variety of benchmark data. we further analyze the results to gain insights as to why and when this approach would be most valuable, and conclude that it is most prominently when the training data available for the target task is relatively small and the classification task is of a topical nature. finally, we propose future directions. we release code for reproducing our method. foot_0 ## intermediate training using unsupervised clustering a pre-trained model is typically developed in consecutive phases. henceforth, we will refer to bert as the canonical example of such models. first, the model is pretrained over massive general corpora with the mlm task. 2 we denote the obtained model simply as bert. second, bert is finetuned in a supervised manner with the available labeled examples for the target task at hand. this standard flow is represented via path-1 in fig. 1 . an additional phase can be added between these two, referred to next as intermediate training, or inter-training in short. in this phase, the model is exposed to the corpus of the target task, or a corpus of the same domain, but still has no access to labeled examples for this task. a common example of such an intermediate phase is to continue to intertrain bert using the self-supervised mlm task over the corpus or the domain of interest, sometimes referred to as further or adaptive pre-training (e.g., @xcite . this flow is represented via path-2 in fig. 1 , and the resulting model is denoted bert it:mlm , standing for intermediate task: mlm. a key contribution of this paper is to propose a new type of intermediate task, which is designed to be aligned with a text classification target task, and is straightforward to use in practice. the underlying intuition is that inter-training the model over a related text classification task would be more beneficial compared to mlm inter-training, which focuses on different textual entities, namely predicting the identity of a single token. specifically, we suggest unsupervised clustering for generating pseudo-labels for inter-training. as the clustering partition presumably captures information about salient features in the corpus, feeding this information into the model could lead to representations that are better geared to perform the target task. these pseudo-labels can be viewed as weak labels, but importantly they are not tailored nor require a specific design per target task. instead, we suggest generating pseudo-labels in a way independent of the target classification task. the respective flow is represented via path-3 in fig. 1 . in this flow, we first cluster to partition the training data into n c clusters. next, we use the obtained partition as 'labeled' data in a text classification task, where the classes are defined via the n c clusters, and intertrain bert to predict the cluster label. in line with mlm, inter-training includes a classifier layer on top of bert, which is discarded before the fine-tuning stage. the resulting inter-trained model is denoted bert it:clust . finally, path-4 in fig. 1 represents a sequential composition of paths 2 and 3. in this flow, we first intertrain bert with the mlm task. next, the obtained model is further intertrained to predict the n c clusters, as in path-3. the model resulting from this hybrid approach is denoted bert it:mlm+clust . importantly, following path-3 or path-4 requires no additional labeled data, and involves an a-priori clustering of training instances that naturally gives rise to an alternative or an additional inter-training task. as we show in the following sections, despite its simplicity, this strategy provides a significant boost in performance, especially when labeled data for the final fine-tuning is in short supply. 3 experiments ## results figure 2 depicts the classification accuracy for the different settings for varying labeling budgets, using sib for clustering-based inter-training. over the topical datasets, bert it:clust and bert it:mlm+clust clearly outperform bert and bert it:mlm in the small labeled data regime, where the gain is most prominent for the smallest labeled data examined -when only 64 labeled examples are available -and gradually diminishes as more labeled samples are added. we performed paired t-tests to compare bert it:clust with bert and bert it:mlm , pooling together all datasets and repetitions for a given both bert it:mlm and bert it:clust expose the model to the target corpus. the performance gains of bert it:clust over bert it:mlm suggest that inter-training on top of the clustering carries an additional benefit. in addition, these intertraining approaches are complementary -as seen in fig. 2 , bert it:mlm+clust outperforms both bert it:clust and bert it:mlm (at the cost of some added runtime). taken together, our results suggest that in topical datasets, where labeled data is scarce, the pseudolabels generated via clustering can be leveraged to provide a better starting point for a pre-trained model towards its fine-tuning for the target task. ## related work in our work, we transfer a pretrained model to a new domain with little data. transfer learning studies how to transfer models across domains. it suggests methods such as pivoting @xcite , weak supervision @xcite , data augmentation (anaby-tavor et al., 2020) and adversarial transfer @xcite . in computer vision, pretrained models are often learnt by image clustering @xcite . in nlp, however, clustering was mainly used for nontransfer scenarios. ball (2019) relies on pretrained embeddings to cluster labeled and unlabeled data. then, they fill the missing labels to augment the training data. clustering itself was improved by combining small amounts of data @xcite . pretrained models improved state-of-the-art in many downstream tasks @xcite and they are especially needed and useful in low resource and limited labeled data settings @xcite @xcite . there are many suggestions to improve such models, including larger models @xcite , changes in the pretraining tasks and architecture @xcite , augmenting pretraining @xcite , or improving the transfer itself @xcite @xcite . two findings on pretraining support our hypothesis on the intermediate task, namely that classification surpasses mlm. some pretraining tasks are better than others @xcite and supervised classification as additional pre-training improves performance @xcite @xcite . all these works aim to improve the performance upon transfer, making it more suitable for any new domain. in contrast, we focus on improvement given the domain. with a transferred model, one can further improve performance with domain-specific information. for example, utilizing metadata @xcite , training on weakly-supervised data @xcite or multitasking on related tasks concurrently @xcite . given no domain-specific information, it was suggested to further pretrain on unlabeled data from the domain @xcite @xcite @xcite . this, however, is sometimes unhelpful or even hurts results @xcite . transferring a model and retraining with paucity of labels is often termed few-shot learning. few shot learning is used for many language-related tasks such as named entity recognition @xcite , relation classification @xcite , and parsing @xcite . there have also been suggestions other than fine-tuning the model. @xcite suggests ranking examples' similarity with siamese networks. @xcite rely on memory and attention to find neighboring examples and @xcite search for prototypes to compare to. @xcite don't define in advance how to compare the examples. instead, they meta-learn how to train the few shot learner. these works addressed the image classification domain, but they supply general methods which are used, improved and adapted on language domains @xcite . in conclusion, separate successful practices foreshadow our findings: clustering drives pretraining on images; supervised classification aids pre-training; and training on unlabeled domain examples is helpful with mlm. ## conclusions we presented a simple approach for improving pretrained models for text classification. specifically, we show that inter-training bert over pseudolabels generated via unsupervised clustering creates a better starting point for the final fine-tuning over the target task. our analyses suggest that bert can leverage these pseudo-labels, namely that there exists a beneficial interplay between the proposed inter-training and the later fine-tuning stage. our results show that this approach yields a significant boost in accuracy, mainly over topical data and when labeled data is scarce. note that the method does require the existence of an unlabeled corpus, in the order of several thousand examples. we opted here for a practically oriented approach, which we do not claim to be optimal. rather, the success of this approach suggests various directions for future work. in particular, several theoretical questions arise, such as what else determines the success of the approach in a given dataset; understanding the potential synergistic effect of bow-based clustering for intertraining; could more suitable partitions be acquired by exploiting additional embedding space and/or more clustering techniques; co-training @xcite methods, and more. on the practical side, while in this work we fixed the inter-training to be over 50 clusters and for a single epoch, future work can improve performance by tuning such hyper-parameters. in addition, one may consider using the labeled data available for fine-tuning as anchors for the intermediate clustering step, which we have not explored here. another point to consider is the nature of the inter-training task. here, we examined a multiclass setup where bert is trained to predict one out of n c cluster labels. alternatively, one may consider a binary inter-training task, where bert is trained to determine whether two samples are drawn from the same cluster or not. finally, the focus of the present work was on improving bert performance for text classification. in principle, inter-training bert over clustering results may be valuable for additional downstream target tasks, that are similar in spirit to standard text classification. examples include key-point analysis @xcite and textual entailment @xcite . the potential value of our approach in such cases is left for future work.