Contextualized Weak Supervision for Text Classification (2020)

Abstract: Weakly supervised text classification based on a few user-provided seed words has recently attracted much attention from researchers. Existing methods mainly generate pseudo-labels in a context-free manner (e.g., string matching), therefore, the ambiguous, context-dependent nature of human language has been long overlooked. In this paper, we propose a novel framework ConWea, providing contextualized weak supervision for text classification. Specifically, we leverage contextualized representations of word occurrences and seed word information to automatically differentiate multiple interpretations of the same word, and thus create a contextualized corpus. This contextualized corpus is further utilized to train the classifier and expand seed words in an iterative manner. This process not only adds new contextualized, highly label-indicative keywords but also disambiguates initial seed words, making our weak supervision fully contextualized. Extensive experiments and case studies on real-world datasets demonstrate the necessity and significant advantages of using contextualized weak supervision, especially when the class labels are fine-grained.

## introduction weak supervision in text classification has recently attracted much attention from researchers, because it alleviates the burden of human experts on annotating massive documents, especially in specific domains. one of the popular forms of weak supervision is a small set of user-provided seed words for each class. typical seed-driven methods follow an iterative framework -generate pseudolabels using some heuristics, learn the mapping between documents and classes, and expand the seed set @xcite @xcite @xcite . most of, if not all, existing methods generate pseudo-labels in a context-free manner, therefore, the ambiguous, context-dependent nature of human languages has been long overlooked. suppose the user gives "penalty" as a seed word for the sports class, as shown in figure 1 . the word "penalty" has at least two different meanings: the penalty in sports-related documents and the fine or death penalty in law-related documents. if the pseudolabel of a document is decided based only on the frequency of seed words, some documents about law may be mislabelled as sports. more importantly, such errors will further introduce wrong seed words, thus being propagated and amplified over the iterations. in this paper, we introduce contextualized weak supervision to train a text classifier based on userprovided seed words. the "contextualized" here is reflected in two places: the corpus and seed words. every word occurrence in the corpus may be interpreted differently according to its context; every seed word, if ambiguous, must be resolved according to its user-specified class. in this way, we aim to improve the accuracy of the final text classifier. we propose a novel framework conwea, as illustrated in figure 1 . it leverages contextualized representation learning techniques, such as elmo @xcite and bert @xcite , together with user-provided seed information to first create a contextualized corpus. this contextualized corpus is further utilized to train the classifier and expand seed words in an iterative manner. during this process, contextualized seed words are introduced by expanding and disambiguating the initial seed words. specifically, for each word, we develop an unsupervised method to adaptively decide its number of interpretations, and accordingly, group all its occurrences based on their contextualized representations. we design a principled comparative ranking method to select highly labelindicative keywords from the contextualized corpus, leading to contextualized seed words. we will repeat the iterative classification and seed word expansion process until the convergence. to the best of our knowledge, this is the first work on contextualized weak supervision for text classification. it is also worth mentioning that our proposed framework is compatible with almost any contextualized representation learning models and text classification models. our contributions are summarized as follows: note that, all these words could be upgraded to phrases if phrase mining techniques @xcite were applied as preprocessing. in this paper, we stick to the words. framework overview. we propose a framework, conwea, enabling contextualized weak supervision. here, "contextualized" is reflected in two places: the corpus and seed words. therefore, we have developed two novel techniques accordingly to make both contextualizations happen. first, we leverage contextualized representation learning techniques @xcite to create a contextualized corpus. we choose bert @xcite as an example in our implementation to generate a contextualized vector of every word occurrence. we assume the user-provided seed words are of reasonable quality -the majority of the seed words are not ambiguous, and the majority of the occurrences of the seed words are about the semantics of the user-specified class. based on these two assumptions, we are able to develop an unsupervised method to automatically group word occurrences of the same word into an adaptive number of interpretations, harvesting the contextualized corpus. second, we design a principled comparative ranking method to select highly label-indicative keywords from the contextualized corpus, leading to contextualized seed words. specifically, we start with all possible interpretations of seed words and train a neural classifier. based on the predictions, we compare and contrast the documents belonging to different classes, and rank contextualized words based on how label-indicative, frequent, and unusual these words are. during this process, we eliminate the wrong interpretations of initial seed words and also add more highly label-indicative contextualized words. this entire process is visualized in figure 1 . we denote the number of iterations between classifier training and seed word expansion as t , which is the only hyper-parameter in our framework. we discuss these two novel techniques in detail in the following sections. to make our paper self-contained, we will also brief the pseudo-label generation and document classifiers. ## document contextualization we leverage contextualized representation techniques to create a contextualized corpus. the key objective of this contextualization is to disambiguate different occurrences of the same word into several interpretations. we treat every word separately, so in the rest of this section, we focus on a given word w. specifically, given a word w, we denote all its occurrences as w 1 , . . . , w n , where n is its total number of occurrences in the corpus. contextualized representation. first, we obtain a contextualized vector representation b w i for each w i . our proposed method is compatible with almost any contextualized representation learning model. we choose bert @xcite as an example in our implementation to generate a contextualized vector for each word occurrence. in this contextualized vector space, we use the cosine similarity to measure the similarity between two vectors. two word occurrences w i and w j of the same interpretation are expected to have a high cosine similarity between their vectors b w i and b w j . for the ease of computation, we normalize all contextualized representations into unit vectors. choice of clustering methods. we model the word occurrence disambiguation problem as a clustering problem. specifically, we propose to use the k-means algorithm (jain and dubes, 1988) to cluster all contextualized representations b w i into k clusters, where k is the number of interpretations. we prefer k-means because (1) the cosine similarity and euclidean distance are equivalent for unit vectors and ( 2 ) it is fast and we are clustering a significant number of times. automated parameter setting. we choose the value of k purely based on a similarity threshold τ . τ is introduced to decide whether two clusters belong to the same interpretation by checking if the cosine similarity between two cluster center vectors is greater than τ . intuitively, we should keep increasing k until there exist no two clusters with the same interpretation. therefore, we choose k to be the largest number such that the similarity between any two cluster centers is no more than τ . we pick τ based on user-provided seed information instead of hand-tuning, as mentioned, we make two "majority" assumptions: (1) for any seed word, the majority of its occurrences follow the intended interpretation by the user; and (2) the majority of the seed words are not ambiguousthey only have one interpretation. therefore, for each seed word s, we take the median of pairwise cosine similarities between its occurrences. algorithm 1: corpus contextualization input: word occurrences w 1 , w 2 , . . . , w n of the word w, seed words s 1 , s 2 , . . . , s m and their occurrences s i,j . output: contextualized word occurrences ŵ1 , ŵ2 , . . . , ŵn obtain b w i and b s i,j using bert. compute τ follow equation 3 . then, we take the median of these medians over all seed words as τ . mathematically, the nested median solution makes the choice of τ safe and robust to outliers. for example, consider the word "windows" in the 20newsgroup corpus. in fact, the word windows has two interpretations in the 20newsgroup corpus -one represents an opening in the wall and the other is an operating system. we first compute the pairwise similarities between all its occurrences and plot the histogram as shown in figure 2(a) . from this plot, we can see that its median value is about 0.7. we apply the same for all seed words and obtain τ following equation 3. τ is calculated to be 0.82. based on this value, we gradually increase k for "windows" and it ends up with @xmath0. we visualize its k-means clustering results using t-sne @xcite in figure 2(b) . similar results can be observed for the word penalty, as shown in figure 2(c ). these examples demonstrate how our document contextualization works for each word. in practice, to make it more efficient, one can subsample the occurrences instead of enumerating all pairs in a brute-force manner. contextualized corpus. the interpretation of each occurrence of w is decided by the cluster-id to which its contextualized representation belongs. specifically, given each occurrence w i , the word w is replaced by ŵi in the corpus as follows: by applying this to all words and their occurrences, the corpus is contextualized. the pseudo-code for corpus contextualization is shown in algorithm 1. ## pseudo-label and text classifier we generate pseudo-labels for unlabeled contextualized documents and train a classifier based on these pseudo-labels, similar to many other weakly supervised methods @xcite @xcite @xcite . these two parts are not the focus of this paper. we briefly introduce them to make the paper self-contained. pseudo-label generation. there are several ways to generate pseudo-labels from seed words. as proof-of-concept, we employ a simple but effective method based on counting. each document is assigned a label whose aggregated term frequency of seed words is maximum. let tf( ŵ, d) denote term-frequency of a contextualized word w in the contextualized document d and s c represents set of seed words of class c, the document d is assigned a label l(d) as follows: our framework is compatible with any text classification model. we use hierarchical attention networks (han) @xcite as an example in our implementation. han considers the hierarchical structure of documents (document -sentences -words) and includes an attention mechanism that finds the most important words and sentences in a document while taking the context into consideration. there are two levels of attention: word-level attention identifies the important words in a sentence and sentence level attention identifies the important sentences in a document. the overall architecture of han is shown in figure 3 . we train a han model on contextualized corpus with the generated pseudo-labels. the predicted labels are used in seed expansion and disambiguation. ## seed expansion and disambiguation seed expansion. given contextualized documents and their predicted class labels, we propose to rank contextualized words and add the top few words into the seed word sets. the core element of this process is the ranking function. an ideal seed word s of label l, is an unusual word that appears only in the documents belonging to label l with significant frequency. hence, for a given class c j and a word w, we measure its ranking score based on the following three aspects: here, different from f c j ,w defined earlier, f c j (w) is the frequency of word w in documents that are predicted as class c j . similar to previous work @xcite , we combine these three measures using the geometric mean, resulting in the ranking score r(c j , w) of a word w for a class c j . r(c j , w) = li(c j , w) × f(c j , w) × idf(w) ## experiments in this section, we evaluate our framework and many compared methods on coarse-and finegrained text classification tasks under the weakly supervised setting. ## related work we review the literature about (1) weak supervision for text classification methods, (2) contextualized representation learning techniques, (3) document classifiers, and (4) word sense disambiguation. ## conclusions and future work in this paper, we proposed conwea, a novel contextualized weakly supervised classification framework. our method leverages contextualized representation techniques and initial user-provided seed words to contextualize the corpus. this contextualized corpus is further used to resolve the interpretation of seed words through iterative seed word expansion and document classifier training. experimental results demonstrate that our model outperforms previous methods significantly, thereby signifying the superiority of contextualized weak supervision, especially when labels are fine-grained. in the future, we are interested in generalizing contextualized weak supervision to hierarchical text classification problems. currently, we perform coarse-and fine-grained classifications separately. there should be more useful information embedded in the tree-structure of the label hierarchy. also, extending our method for other types of textual data, such as short texts, multi-lingual data, and code-switched data is a potential direction.