Evaluating Pre-Trained Sentence- BERT with Class Embeddings in Active Learning for Multi-Label Text Classification (2022)

Abstract: The Transformer Language Model is a powerful tool that has been shown to excel at various NLP tasks and has become the de-facto standard solution thanks to its versatility. In this study, we employ pre-trained document embeddings in an Active Learning task to group samples with the same labels in the embedding space on a legal document corpus. We find that the calculated class embeddings are not close to the respective samples and consequently do not partition the embedding space in a meaningful way. In addition, we explore using the class embeddings as an Active Learning strategy with dramatically reduced results compared to all baselines.

## introduction while text classification models have become more and more powerful, the need for sufficient data to train ever growing neural networks is also increasing massively. when dealing with domain-specifc data, such as legal or medical in particular, finding a fitting dataset with detailed annotations can be exceedingly difficult. creating such a dataset is likely to be a massive undertaking due to the difficult annotation process which often requires domain experts to work through enormous amounts of data. active learning serves as a way to speed up this process by selecting informative samples to be annotated. however, active learning strategies are often very specific to target domains @xcite and strategies tailored specifically for pre-trained transformer language models are often experimental and not thoroughly explored @xcite . in this work, we present an active learning strategy that employs class embeddings which are generated from pre-trained sentence embeddings to predict the classes of unlabeled samples. while the intuition of the approach is sound, we find that the class embeddings do not generalize from the samples they were calculated on. our experiment focuses on powerful pre-trained, transformer sentence-embeddings which are prevalent in both research and industrial application. we demonstrate that such embeddings struggle to find good separations between the multi-class, multi-label texts in the training set on two domain-specific datasets. our work details the class embedding approach, illustrates the reduced performance on two domain-specific, multi-label datasets and analyses the vector space of the samples to gain an understanding of the methods failure. ## related work the effectiveness of al for text classification has been subject to extensive research @xcite , @xcite with specific solutions for deep models @xcite , @xcite and multi-label settings @xcite . our approach targets active learning for deep learning which poses new challenges @xcite and is still a topic in need of exploration @xcite . generating embeddings from words has been performed with trained vector models @xcite ) @xcite but has been moved to the contextual embedded information within large transformer language models such as bert @xcite . extracting embeddings across word boundaries from bert can be done in several ways, such as a grid-based approach (denk and reisswig, 2019), a "siamese" dual network architecture @xcite or unsupervised techniques @xcite . ## conclusion & future work we present class embeddings, which hinder the active learning (section 4.3) since the classes of new samples can not be correctly predicted (section 5.1). despite reasonable assumptions about the effectiveness of pre-trained embeddings (section 3.1) we find that class embeddings are not meaningful representatives of the dataset classes and that their ability to partition the dataset is limited (5.2). we encourage experimenting with this approach, as it is relatively inexpensive to compute. in addition to using common heuristics with bert, such as averaging the word embeddings, fine-tuning the sentence-embeddings on the dataset might make a difference and result in higher quality class embeddings. also, testing the approach on different datasets is crucial -in our work, improving upon random selection is difficult even for sophisticated active learning strategies. finally, we would like to motivate more application-oriented research (e.g. information retrieval, semantic similarity rankings etc...) into the inner workings of pre-trained contextual embeddings in order to improve understanding of the information they encode.