Multi-pretraining for Large-scale Text Classification (2020)

Abstract: Deep neural network-based pretraining methods have achieved impressive results in many natural language processing tasks including text classification. However, their applicability to large-scale text classification with numerous categories (e.g., several thousands) is yet to be well-studied, where the training data is insufficient and skewed in terms of categories. In addition, existing pretraining methods usually involve excessive computation and memory overheads. In this paper, we develop a novel multi-pretraining framework for large-scale text classification. This multi-pretraining framework includes both a self-supervised pretraining and a weakly supervised pretraining. We newly introduce an out-of-context words detection task on the unlabeled data as the self-supervised pretraining. It captures the topic-consistency of words used in sentences, which is proven to be useful for text classification. In addition, we propose a weakly supervised pretraining, where labels for text classification are obtained automatically from an existing approach. Experimental results clearly show that both pretraining approaches are effective for large-scale text classification task. The proposed scheme exhibits significant improvements as much as 3.8% in terms of macro-averaging F1-score over strong pretraining methods, while being computationally efficient.

## introduction large-scale text classification is a natural language processing (nlp) task and a long-standing yet challenging problem. it seeks to classify arbitrary texts into semantically relevant classes or categories. a wide range of applications exploit large-scale text classification, including web search personalization @xcite , contextual advertising @xcite , topical web search @xcite , and recommender systems @xcite . the success of these applications is highly dependent on the quality of text classification. many applications of large-scale text classification require a sufficiently large taxonomy of topical categories to capture various topics in arbitrary texts. in addition, it is necessary to collect a large amount of training data for each category in the taxonomy. deep neural models have demonstrated promising results in text classification tasks @xcite @xcite , owing to their strong expressive power and less requirement for feature engineering. however, the deeper and more complex the neural model, the more it is essential for them to be trained on substantial amount of training data. hence, pretraining methods have attracted significant attention @xcite ; these methods leverage large text corpora to train a model with better generalization properties @xcite . recently, several pretraining methods using bidirectional long short-term memory (lstm) networks @xcite or transformer @xcite have been highly successful in many nlp tasks @xcite @xcite . these methods first pretrain neural networks on large unlabeled text corpora, and then, finetune the pretrained networks on downstream tasks. although pretraining methods have achieved state-of-the-art status on many nlp tasks @xcite @xcite , their applicability to large-scale classification is yet to be well-studied, in which training data for each category is severely insufficient and distributed unevenly among classification categories. in addition, existing pretraining methods typically need very high capacity and long training time. to handle large-scale text classification, @xcite have alleviated the problem of the limited amount of training data by utilizing a multitask learning @xcite . the proposed multi-task framework converts the large-scale text classification task to a small-scale text classification task, and learns both tasks simultaneously. they achieve a new state-ofthe-art large-scale text classification. however, this scheme may not benefit from the generalization of pretraining because different scale tasks come from a single limited dataset. in this paper, we develop a novel multipretraining framework based on the convolutional neural network (cnn) to handle large-scale text classification. by the multi-pretraining, we mean that the framework simultaneously learns both a self-supervised and a weakly supervised pretraining. self-supervised learning methods, such as masked language modeling and permutation language modeling, have shown to be effective for improving neural models on many nlp tasks @xcite @xcite . however, they usually require excessive computation and training time. instead of the language model, we newly introduce an out-of-context (ooc) words detection task on the unlabeled data for a self-supervised pretraining. notably, the ooc words detection task is inspired by the success of learning to spot artifacts @xcite , one of self-supervised learning methods in computer vision. we hypothesize that the ooc words detection in nlp is analogous to the spotting artifacts in computer vision, thereby helping improve nlp tasks. it turns out that the task indeed learns useful features for text classification by detecting whether or not there are ooc words in a sentence. in order to generate ooc words, we randomly select a word from the sentence or paragraph foot_0 in corpora and replace it with a random word. in addition, we propose a weakly supervised pretraining @xcite , where labels are obtained automatically with an existing text classification method. to this end, we use the output of an explicit representation model @xcite based on bag-of-words or bagof-phrases as a weak supervision signal. notably, weakly supervised labels are highly abstract topics (e.g., sports, health, computers, business, etc.) rather than complete categories (e.g., sports/baseball/mlb/awards) in a large-scale tax-onomy. the proposed framework involves two steps. the first step is to pretrain the models on the unlabeled large corpora, which learns to spot whether ooc words are used, and a weakly supervised text classification task simultaneously. the second step is to finetune the same model on the labeled large-scale text classification dataset. we demonstrate the efficacy of our framework on large-scale text classification. the experimental results show that the proposed multi-pretraining scheme yields significantly improved results in large-scale text classification. in summary, our contributions are three-fold: the performance evaluation clearly shows that our approach outperforms a dozen of state-ofthe-art large-scale text classification methods and is competitive with excessively large pretraining methods. the remainder of this paper is organized as follows. section 2 describes the proposed multipretraining framework for large-scale text classification. we present the performance evaluation results and in-depth analysis in section 3 and 4, respectively. we discuss related work in section 5 and conclude in section 6. ## methodology in this section, we describe two pretraining methods, a self-supervised pretraining and a weakly supervised pretraining, for large-scale text classification. we propose a novel multi-pretraining to utilize the knowledge obtained from both pretrainings. in addition, we introduce finetuning strategies to maintain the useful knowledge or features. we adopt the cnn which has been very successful in text classification tasks @xcite . in particular, a carefully designed cnn outperforms other architectures in large-scale text classification @xcite with relatively small number of parameters. figure 1 shows our proposed framework. ## experiments we experiment with the large-scale text classification to verify the efficacy of the proposed multipretraining framework. ## analysis xlnet by a large margin for the limited training data, as shown in figure 2 . in addition, we observe that mp-cnn trained with only 1,250 labeled examples outperforms cnn trained with 5× more data. these results clearly demonstrate that the proposed multi-pretraining successfully improves the performance of large-scale text classification. we qualitatively examine the classified categories from sp-cnn and cnn to analyze why the ooc words detection task improves the performance of large-scale text classification. we select a magazine title "professional tennis player w on feminism and motherhood", which is related to both "sports" and "female". then, we analyze the changes of predicted score in the presence of ooc words at different time steps, which are obtained by sp-cnn. the output layer for large-scale text classification of sp-cnn returns sports/tennis/players up to the fifth word (i.e., on), while sports/tennis/players/female when the sixth word (i.e., feminism) is processed. in contrast, cnn keeps returning sports/tennis/players until the last word. interestingly, from figure 3 , we observe that sp-cnn captures "feminism" and "motherhood" are semantically different from the previous context. these results illustrate how sp-cnn effectively classifies the sentence into the highly specific categories, both "sports" and "female". we note that this is consistent with many sentences in the test dataset. ## related work for large-scale text classification, many techniques have been proposed to handle data sparsity. @xcite firstly addressed data sparsity on a hierarchical taxonomy. they adopted a statistical technique, called shrinkage, to estimate the parameters of data-sparse child categories with their data-rich ancestor categories. @xcite proposed a large-scale text classification method called merge-centroid (mc). mc utilizes enriched training data for each category based on webpages classified into their ancestor and/or descendants in the odp. in another line of work @xcite , they utilize a multi-task learning by treating different scales of text classification as related tasks. they have achieved a new state-of-the-art performance in large-scale text classification. however, these approaches cannot utilize the implicit knowledge from general-domain large corpora. neural network-based models have utilized pretrainings to overcome the problem of insufficient training data. recent neural network-based approaches have utilized the language model pretraining on large text corpora. @xcite pretrained a transformer decoder-based language model and finetuned it using task-aware transformations. it consequently accomplished large gains on natural language understanding tasks. yet another work @xcite used a masked language model based on a bidirectional transformer in pretraining. very recently, @xcite have suggested finetuning a parameter-efficient variant of transformer pretrained on the masked language model to benefit both the model size and performance. the ideas from transformer-xl @xcite were integrated into the permutation language model @xcite . another line of work @xcite introduced several techniques for finetuning a pretrained language model. to the best of our knowledge, our current work is one of only a few work that applies the pretrainings to large-scale text classification. ## conclusion in this paper, we have developed a novel cnnbased pretraining framework to handle large-scale text classification. specifically, we pretrain the proposed cnn-based model, which simultaneously learns both the ooc words detection and the text classification task on unlabeled corpora. we have verified the large-scale text classification performance of our methodology using real-world datasets. our experimental results confirm that our methodology significantly outperforms a dozen of strong baseline methods. we plan to apply our framework to another nlp tasks, such as sentiment analysis and keyphrase extraction.