Progressive Class Semantic Matching for Semi-supervised Text Classification (2022)

Abstract: Semi-supervised learning is a promising way to reduce the annotation cost for text-classification. Combining with pre-trained language models (PLMs), e.g., BERT, recent semi-supervised learning methods achieved impressive performance. In this work, we further investigate the marriage between semi-supervised learning and a pre-trained language model. Unlike existing approaches that utilize PLMs only for model parameter initialization, we explore the inherent topic matching capability inside PLMs for building a more powerful semi-supervised learning approach. Specifically, we propose a joint semi-supervised learning process that can progressively build a standard K -way classifier and a matching network for the input text and the Class Semantic Representation (CSR). The CSR will be initialized from the given labeled sentences and progressively updated through the training process. By means of extensive experiments, we show that our method can not only bring remarkable improvement to baselines, but also overall be more stable, and achieves state-of-the-art performance in semi-supervised text classification.

## introduction text classification is a fundamental task in natural language processing (nlp) and underpins various applications, e.g., spam detection @xcite , sentiment analysis @xcite ) and text summarization @xcite . supervised training of text classifiers often demands a large amount of annotation, which can be expensive for many applications. semi-supervised learning (ssl) provides an economical way for alleviating this burden since it can make use of easyaccessible unlabeled samples to build a reasonably performed classifier with a limited amount of labeled data. recently, ssl received increasing attention in both image classification @xcite @xcite ) and text classification @xcite @xcite areas. meanwhile, pre-trained language models (plms) @xcite @xcite are developing rapidly and achieve impressive performance in various nlp tasks @xcite ) including text classification @xcite . in the context of semi-supervised text classification, many existing methods achieve excellent performance by directly using a plm as a sentence encoder and further fine-tuning it with a semi-supervised learning process @xcite @xcite . in this paper, we further explore the usage of plms for ssl. we go beyond the strategy of using plms for encoder initialization and make full use of inner knowledge of plms. concretely, we identify that some plms, e.g., bert, have an inherent matching capability between sentence and class-related words thanks to its pre-training pretext task @xcite (as the examples shown in fig. 1 ). we further propose to strengthen this capability through ssl on labeled and unlabeled data. specifically, we develop a joint training process to update three components progressively, that is, a classifier that performs the standard k-way classification, a class semantic representation (csr) that represents the semantic of each category, and a matching classifier that matches the input sentence against the csr. those three components can help each other during the training process, i.e., the k-way classifier will receive more accurate pseudo-labels by jointly generating pseudo-labels with the matching classifier; the matching classifier will also upgrade its matching capability with the guidance of the k-way classifier. the csr will become more accurate and comprehensive with the improvement of the k-way classifier and matching classifier. this joint process leads to a more powerful semi-supervised learning algorithm for the text classification task. throughout our experimental evaluation, we demonstrate that the proposed method achieves the state-of-the-art performance on text data, especially when the number of labeled sentences becomes extremely low, i.e., 3 or 5. ## related work in this section, we briefly review the relevant research works. ## inherent matching capability of a plm in this section, we will demonstrate the inherent topic matching capability of bert which motivates our method. utilizing plms for a downstream task has become common since it often brings a significant performance boost @xcite . in the context of semisupervised learning, a plm is usually employed for initializing the network before performing semisupervised training. however, the value of a plm can go beyond a good initial model or feature extractor. in particular, a plm like bert has already learned certain topic matching capabilities thanks to its pretext tasks. for example, bert uses the next sentence prediction (nsp) as one of its pretext tasks. in this task, the network is asked to discern if two input sentences are two successive sentences in the original corpus. after training on this task, bert can implicitly acquire topic matching capability since two successive sentences in a paragraph usually share the same topic. fig. 1 shows a concrete investigation of the in-herent matching capability of bert. following the nsp task, we concatenate the sentence and class semantic-related words c k , e.g., "sports", via the format: "[cls] sentence [sep] c 1 • • • c k • • • c k [sep]". then we pass the input sequence to a pretrained bert and calculate the attention value of each token with respect to each class name. specifically, this attention value is calculated by averaging the last layer self-attention values across all heads between a token and the appended word c k . for better visualization, we use different color to show the class that leads to the largest attention value (indicated by the color brightness). from the visualization, we can see that bert can automatically match keywords corresponding to the respective class names. moreover, we find that if we replace the class names with words under the same topic, i.e., @xmath0, @xmath1, the words related to the ground-truth class can still be attended, as shown in fig. 1a and 1b . finally, we extract bert last-layer's feature corresponding to each class word c k and average features align with sentence tokens, and compare the cosine similarity between them. as histograms shown in fig. 1a and 1b , we can find that the correct class leads to the highest matching score, although not always by a large margin. ## progressive class-semantic matching to further strengthen the above topic matching capability and use it for classification, we propose to progressively build a sentence-class matching model through the framework of semisupervised learning. the idea is to construct a process that can jointly update three components: (1) a standard k-way classifier (2) a matching classifier which matches texts against class semantic representation (3) the class semantic representation (csr) itself. the update of each component will help other components and thus can iteratively bootstrap classification performance. we call our method as progressive classsemantic matching (pcm). ## experimental results in this section, we perform the experimental study of the pcm method on four text datasets. datasets following mixtext @xcite , we use four datasets, namely, ag news (zhang implementation details same as mixtext foot_3 , we use back-translation to perform data augmentation. two languages, german and russian, are chosen as the intermediate language. the back-translation texts on yahoo! answers are provided by mix-text, and we directly use them. for the other three datasets, we generate the back-translation data by ourselves (with fairseq toolkit @xcite ). we use the input format "[cls] sentence [sep]" for all the baseline methods. we empirically find this format leads to the overall best performance. meanwhile, this format actually brings performance improvement to both uda and mixtext methods. so we are comparing against stronger baselines in our paper. due to bert's length limit, we only kept the last 256 tokens for imdb and the first 256 tokens for the other datasets during training. we use the same learning-rate setting for all methods: 5e-6 for the bert encoder and 5e-4 for the classifier (i.e., a two-layer mlp with a 128 hidden size and tanh as its activation function). all our experiments were run on a geforce rtx 2080 ti gpu and each experiment takes around 5 hours. ## limitations and potential risks one underlying assumption about our findings is that we mainly consider bert-style pre-trained language models for semi-supervised text classification. the utilization of inherent knowledge of other language models (e.g., gpt @xcite and xlnet @xcite ) are not explored in this paper and is left for future work. pcm algorithm has been verified to be effectiveness on texts in english, whether other languages can achieve the same performance improvement is at risk and will be explored in the future. ## conclusion in this paper, we proposed a semi-supervised text classification approach by leveraging the inherent topic matching capability in pre-trained language models. the method progressively updates three components, a k-way classifier, the class semantic representation, and a matching classifier that matches input text against the class semantic representation. we show that the updating of the three components can benefit each other and achieve superior semi-supervised learning performance.