Leveraging the Structure of Pre-trained Embeddings to Minimize Annotation Effort (2024)

Abstract: Most current state-of-the-art approaches for text classification are based on fine-tuning the representations computed by large language models (LLMs). This strategy has led to significant improvements in classification performance and contributed to a reduction of the amount of labeled data required for training a model. However, for some challenging classification tasks, providing enough annotations to ensure a reliable classification continues to be the main bottleneck. This is especially true in settings of highly imbalanced class distributions. This paper proposes to tackle this bottleneck by exploiting the structural properties of pre-trained embeddings. We develop a label propagation method that uses pre-trained embeddings to spread information from the labeled samples to nearby samples in the induced space, ensuring the optimal use of annotations. Our approach is simple and relatively low-cost since it only requires computing some distances in the embedded space. We conduct experiments on different text classification datasets showing that the proposed method is efficient and significantly outperforms both self-training and random walk label propagation strategies.

## introduction currently, the dominant state-of-the-art approach for text classification is based on fine-tuning the representations computed by large language models (llms). this approach has led to significant improvements in classification performance and contributed to a reduction of the amount of supervised labeled data required for training a model. however, for some challenging classification tasks, providing enough annotations to ensure a reliable classification continues to be the main bottleneck in terms of cost and time. we would like to train models with minimal annotation effort. in this paper, we focus on the problem of learning a text classifier under annotation budget con-straints and with no prior trained model. the only assumption we make is access to a large unlabeled dataset. this is sometimes referred to as few-shot learning or the cold-start problem. notice that our setting is different from the classical active learning scenario, where one assumes several training iterations and the algorithm alternates between sampling data and learning a new model. our focus is on the initial setting and we exploit a semisupervised strategy. we are interested in classification scenarios with highly imbalanced class distributions. class imbalance is a frequent phenomenon in real nlp applications because target categories tend to be skewed (spam detection, hate speech). class imbalance poses an additional challenge when learning under budget constraints. this is because, even if we only require a small sample of the target class, to obtain such a sample we might still require a significant number of annotations. for instance, in a scenario with a 10% target class probability, obtaining 10 representative samples would, on average, require annotating 100 samples. self-training @xcite is a popular strategy for learning textual classifiers with tight annotation budgets and it is widely used by nlp practitioners @xcite . this semi-supervised learning strategy starts by training a model with a labeled seed set. its predictions on a large unlabeled dataset are then used to train a new model. typically this process is repeated until the whole unlabeled dataset has been labeled. instead of training a model with its predictions, an alternative approach is to exploit the properties of the input space directly. recent literature on analyzing textual representations @xcite @xcite shows that pre-trained embeddings exhibit some structural properties that make them especially well-suited for text classifica-tion. @xcite showed that the best representations are those in which we can find latent clusters that are well aligned with the classification task. the paper concludes with the suggestion that the structure of the embedded space could be further exploited to improve the performance of classifiers when the budget for supervised annotations is tight. motivated by this study, we propose a strategy to leverage the structure of the embedded space to improve classifier performance under tight annotation budgets. if the latent structure of a representation is well aligned with a task, we should be able to exploit this with an appropriate label propagation strategy. with this in mind, we develop a label propagation strategy specifically designed for the imbalanced class scenario. additionally, a label propagation strategy might be more successful if the initial labeled set is diverse. label propagation and diversity sampling are complementary ideas of how to exploit the structure of the embedded space. diversity sampling prioritizes the selection of representative samples, while label propagation maximizes the utility of the chosen samples, therefore it is natural to combine them. in this paper, we combine both techniques and examine how the choice of initial samples impacts overall performance under annotation budget constraints. our experiments on text classification show that our approach is very effective for fine-tuning classifiers under tight annotation budgets, especially in imbalanced scenarios. our technique outperforms both self-training and label propagation strategies. furthermore, although the setting is different, we compare our approach against a classical active learning baseline and show that it leads to significant reductions in the annotations required to achieve a given classification performance. overall, the results show that our proposed propagation strategy leads to significant performance improvements independent of the initial seed sampling strategy. however, combining label propagation and diversity sampling does lead to further improvements with very low annotation budgets. in summary, our main contributions are: ## leveraging pre-trained embeddings for learning with an annotation budget in this section, we present our proposed semisupervised approach, which leverages pre-trained embeddings to learn textual classifiers under tight annotation budgets. ## related work semi-supervised learning techniques that leverage unlabeled data to improve the performance of models trained with tight annotation budgets have a long history in nlp and are still widely used to mitigate the data annotation bottleneck. we can distinguish two main approaches: those based on self-training and those based on graph-based regularization. self-training is a popular semi-supervised strat-egy for text classification and has been explored by several recent works @xcite @xcite , with a focus on zeroshot or few-shot scenarios @xcite @xcite , light architectures @xcite , or the multilingual setup @xcite . one of the main known limitations of selftraining is that it can suffer from confirmation bias which happens when the model repeatedly over-fits and assigns incorrect pseudo-labels. since these pseudo-labels are used to retrain the model, this repeated over-fitting can make the model diverge from the true class distribution. this problem is especially critical with over-parameterized models such as transformers because they tend to overfit. to avoid the confirmation bias problem an alternative approach is to consider strategies that directly exploit the structure of the unlabeled space. graph-based regularization approaches for semisupervised classification have been proposed in this context. these techniques first compute a similarity graph among unlabeled data points, using a representation space. then soft label propagation is implemented implicitly by modifying the training loss function, using unlabeled data as a form of regularization. this is achieved by including a term in the optimization function that biases the classifier to provide labels for unlabeled points that are similar according to the graph. we term it soft label propagation because the regularization penalty implicitly imposes some regularities on the label assignments of unlabeled points. one of the first graph-based semi-supervised approaches for text classification was introduced in @xcite , before the emergence of pre-trained llms. more recently, saraiva et al. (2021) used a graph-based method for toxic comment detection. similar to our work, they also use pre-trained embeddings to build the similarity graph over the unlabeled data points. however, their approach differs significantly from ours, as they use the graph to regularize the loss function of simpler classifiers like decision trees and support vector machines. another line of work has studied the use of graph-based semi-supervised techniques in multilabel classification @xcite . in this case, the objective is somehow different since the graph regularization is used to infer missing labels in the label set associated with a document. unlabeled data might as well be multi-modal (e.g. images and text). @xcite proposed a semisupervised graph-based method for this setting. one potential disadvantage of graph-based methods is that they tend to be classifier-specific, that is, they are specifically designed with one classification function in mind. another disadvantage is that they can be computationally expensive because typically to train a model the loss function is modified to include constraints between all (or a large subset) of pairs of unlabeled points. to mitigate the inefficiency and lack of generality of graph-based methods, an alternative is to consider explicit label propagation techniques. these methods generate a set of pseudo-labeled samples to be used by any training algorithm. different from graph-based methods, the model is regularized by giving it additional pseudo-labeled data instead of via loss constraints. this approach can have the advantages of more costly graph-based approaches while being simpler, more efficient, and general (applicable to any classification model). in this paper, we decided to follow this approach and develop an efficient and effective label propagation strategy for text classification. interestingly, d'sa et al. (2020) studied the application of random walk label propagation over pre-trained embeddings and reached a negative conclusion. in contrast, we found our proposed label propagation technique to be very effective. as we have shown, not all label propagation techniques are equally effective. in particular, we have seen how random walk label propagation leads to noisy pseudo-labels. to summarize, while there has been a significant amount of work in self-training and graph-based methods for text classification, to the best of our knowledge, we are the first ones to show that a simple and efficient semi-supervised explicit label propagation strategy in the pre-trained embedded space can lead to significant and consistent improvements when learning classifiers under tight annotation budgets. ## conclusion and future work this study has illustrated the effectiveness of implementing label propagation within the embedded spaces derived from llms. the clustering alignment property of the llm embedded space explains the method's success. one of the key distinctions that set our propagation strategy apart from self-training is its inherent resistance to overfitting in the fine-tuning phase. this is because label propagation primarily exploits the embedded space, making it less prone to the pitfalls associated with over-fitting. our findings suggest that the specific strategy employed for seed selection is less critical than previously assumed. overall, our paper demonstrates that a simple and efficient strategy can be effective for training classifiers in scenarios with imbalanced class distribution and limited annotated data. this opens up possibilities for improving semi-supervised learning strategies in scenarios closer to real-world applications. future work will explore the combination of label propagation with active learning techniques, such as uncertainty sampling. we intend to investigate the role of seed selection in this context, as it may still yield performance gains when combined with active strategies.