BERT -Flow- VAE : A Weakly-supervised Model for Multi-Label Text Classification (2022)

Abstract: Multi-label Text Classification (MLTC) is the task of categorizing documents into one or more topics. Considering the large volumes of data and varying domains of such tasks, fully supervised learning requires manually fully annotated datasets which is costly and time-consuming. In this paper, we propose BERT-Flow-VAE (BFV), a Weakly-Supervised Multi-Label Text Classification (WSMLTC) model that reduces the need for full supervision. This new model (1) produces BERT sentence embeddings and calibrates them using a flow model, (2) generates an initial topic-document matrix by averaging results of a seeded sparse topic model and a textual entailment model which only require surface name of topics and 4-6 seed words per topic, and (3) adopts a VAE framework to reconstruct the embeddings under the guidance of the topic-document matrix. Finally, (4) it uses the means produced by the encoder model in the VAE architecture as predictions for MLTC. Experimental results on 6 multi-label datasets show that BFV can substantially outperform other baseline WSMLTC models in key metrics and achieve approximately 84% performance of a fully-supervised model.

## introduction as vast numbers of written comments are posted daily on social media and e-commerce platforms, there is an increasing demand for methods that efficiently and effectively extract useful information from this unstructured text data. one of the methods to analyze this unstructured text data is to classify them into organized categories. this can be considered as a multi-label text classification (mltc) task since a single data may contain multiple non-mutually-exclusive topics (aspects). there are a range of relevant applications of this task such as categorizing movies by genres @xcite , multi-label sentiment analysis @xcite and multi-label toxicity identification @xcite . fully-supervised learning methods are undesirable for this task, because of the diversity of domains of application and cost of manual labelling @xcite . seeded topic models, such as seededlda and corex @xcite , where users can designate seed words as a prior to guide the models to find topics of interest, can be seen as a weakly-supervised multi-label text classification (wsmltc) method. nevertheless, as these models are mainly statistical models based on bagof-words representation, they fail to fully exploit key sentence elements such as context and word positions. in contrast, large pre-trained language models such as bert and gpt-3 @xcite produce contextualized embeddings for each word in a sentence, which has afforded them great success in the nlp field @xcite . recently, prompt-based few-shot learning (fsl) and zero-shot learning (zsl) methods @xcite @xcite ) that take advantage of the general knowledge of large pre-trained language models can also approach mltc tasks using only a few examples or topic surface names as a means of supervision. specifically, these models convert text classification to a textual entailment task by preparing a template such as 'this example is about _' as input, and then estimating the probability of the model filling the blank with certain topic names. however, this method does not work well for abstract topics and there is no agreed way to use multiple words for the entailment task. in this paper, we propose bert-flow-vae (bfv), a wsmltc model. it is based on the variational autoencoder (vae) @xcite framework to reconstruct the sentence embeddings obtained from distil-bert @xcite . inspired by the work of @xcite , we use a shallow @xcite model to map the sentence embeddings to a standard gaussian space before feeding them into the vae model. finally, we use the averaged results of a seeded sparse topic model and a zsl model to guide our model to build latent variables towards pre-specified topics as predictions for mltc. our contributions can be listed as follows: (1) we propose bfv, a wsmltc model based on vae framework, that can achieve comparable performance to a fully-supervised method on 6 multilabel datasets with only limited inputs (4 to 6 seed words per topic and surface name of topics). ( 2) we show that using a normalizing-flow model to calibrate sentence embeddings before feeding them into a vae model can improve the model's mltc performance, suggesting that pre-processing inputs is needed as it can better fit the overall objective of the vae framework. (3) we present that the topic classification performance of zsl method can be further improved by properly integrating predictions from a sparse seeded topic model, which complements the results from zsl method by naturally incorporating multiple words to define a topic and could play a role of regularization. weakly-supervised text classification recently, weakly-supervised text classification (wstc) has been rapidly developed @xcite . most of the works used pseudo labels/documents generation and self-training. particularly, @xcite proposed westclass which uses seed information such as label surface name and keywords to generate pseudo documents and refines itself via self-training. @xcite proposed conwea that uses contextualized embeddings to disambiguate user input seed words and generates pseudo labels for unlabeled documents based on these words to train a text classifier. cosine from @xcite receives weak supervision and generates pseudo labels to perform contrastive learning (with confidence reweighting) to train a classifier. some studies integrated simple rules as weak supervision signals: @xcite used rule-annotated weak labels to denoise labels, which then supervise a classifier to predict unseen samples; @xcite developed astra that utilizes task-specific unlabeled data, few labeled data, and domain-specific rules through a student model, a teacher model and self-training. however, these wstc methods were specifically designed for multi-class tasks and are not optimized for wsmltc tasks in which documents could belong to multiple classes simultaneously. prompt-based zero-shot learning mltc tasks can also be approached with very limited supervision by prompt-based few-shot learning (fsl) or zero-shot learning (zsl). for example, @xcite proposed a zsl method for text classification tasks by treating text classification as a textual entailment problem. this model treats an input text as a premise and prepares a corresponding hypothesis (template) such as 'this example is about _' for the entailment model. finally, it uses the probability of the model filling the blank with topic names as the topic predictions. however, the choice of template and word for the entailment task requires domain knowledge and is often sub-optimal @xcite . also, it is not straightforward to find multiple words as entailment for a topic. this may limit model's ability to understand abstract topics (e.g., 'evacuation' and 'infrastructure') where providing a single surface name is insufficient @xcite . although some automatic search strategies @xcite @xcite have been suggested, relevant research and applications are still under-explored. ## conclusion we presented bfv, a wsmltc model, that uses a vae framework to reconstruct bert-produced and flow-calibrated sentence embeddings under the guidance of the averaged results of corex and 0shot-tc. it can significantly outperform other wsmltc models in key metrics and achieve a approximately 84% performance of a fullysupervised model in terms of macro f1-score evaluated on 6 datasets. we found the improvements are mainly due to: (1) combining bert and vae framework, (2) mapping the sentence embeddings into a standard gaussian space to better fit the overall objective of the vae framework and (3) learning simultaneously from the results produced by a mixture of backends. as the input of surface name of topics and seed words are only used to produce t , bfv can be viewed as a post-processing model to refine an already made document-topic matrix. one limitation of the current bfv model is that it does not explicitly model the relationship between topics and words. thus, relevant tasks such as calculating topic coherence and selecting keywords for each topic cannot be done directly. another drawback of the bfv model is that it does not take into account the dependencies and hierarchies within topics. this may limit the model's performance for datasets in which labels are correlated. in the future, bfv could be improved by reconstructing embeddings of each words in a sentence rather than an embedding of a sentence as a whole. this can also potentially expand this method into an applicable generative model for real sentence generation. further work could also evaluate the performance of the model using embeddings made by language models other than distil-bert. here we compare our model with the following strong weakly-supervised multi-class models on mltc datasets: westclass @xcite : a wstc model that has been briefly introduced in the related-work section. it can receive inputs of topic surface name, keywords or limited amount of documents. we used the keywords as input to it. we replaced its last softmax layer with sigmoid layer and used the same threshold (0.5). x-class @xcite : this methods is based on aligning document representation and class representation. it uses class surface names as input and can generage pseudo labels to train a text classifier. we used its x-class-align version which uses a gaussian mixture model (gmm) to make final predictions. we modified its final to output unnormalized probabilities and used the threshold of 0.5. lotclass @xcite : it is based on masked category prediction (mcp) task and a subsequent self-training to perform wstc with only label surface names. we replaced its last softmax layer with sigmoid and used the same threshold of 0.5. we also tested transforming it into a binary relevance task by only using one label surface name each time, which results in similar performance and therefore is not reported. we used default hyper-parameters for all three models. we slightly modified keywords and topic surface names if the model has a different specification, otherwise we used the same keywords and topic surface names as used in our model. the results of comparison is presented in