MEGC lass: Extremely Weakly Supervised Text Classification via Mutually-Enhancing Text Granularities (2023)

Abstract: Text classification is essential for organizing unstructured text. Traditional methods rely on human annotations or, more recently, a set of class seed words for supervision, which can be costly, particularly for specialized or emerging domains. To address this, using class surface names alone as extremely weak supervision has been proposed. However, existing approaches treat different levels of text granularity (documents, sentences, or words) independently, disregarding inter-granularity class disagreements and the context identifiable exclusively through joint extraction. In order to tackle these issues, we introduce MEGClass, an extremely weakly-supervised text classification method that leverages Mutually-Enhancing Text Granularities. MEGClass utilizes coarse- and fine-grained context signals obtained by jointly considering a document’s most class-indicative words and sentences. This approach enables the learning of a contextualized document representation that captures the most discriminative class indicators. By preserving the heterogeneity of potential classes, MEGClass can select the most informative class-indicative documents as iterative feedback to enhance the initial word-based class representations and ultimately fine-tune a pre-trained text classifier. Extensive experiments on seven benchmark datasets demonstrate that MEGClass outperforms other weakly and extremely weakly supervised methods.

## introduction text classification is a fundamental task in natural language processing (nlp) that enables automatic labeling of massive text corpora, which is necessary in many downstream applications @xcite @xcite . prior works train text classifiers in a fully-supervised manner @xcite @xcite that requires a substantial amount of training data, which is expensive and time-consuming, especially in emerging domains that require the supervision of domain experts. recent works have explored weakly supervised text classification, where a few labeled documents or class seeds are provided to serve as weak supervision @xcite @xcite @xcite . these works typically compile a pseudo-training dataset from a given corpus by assigning a pseudolabel to each document based on its alignment to a specific class. the pseudo-training dataset aims to be a confident substitution for an expert-curated collection of class-indicative documents, ultimately being used to fine-tune a text classifier. the motivation of this work begins with the assumption that there are three types of documents with a ground truth class topic, as illustrated in figure 1: documents discussing (1) the class topic at the word, sentence, and document level, (2) the class topic at the document-level but with multiple other topics mentioned at the word or sentence level, or (3) multiple topics as well as the class topic at all levels. existing weakly supervised methods focus on word-level @xcite @xcite or documentlevel @xcite information independently and only ever associate one pseudo-label to a document, which may allow them to differentiate documents of type (1) from those of type (3). however, this notion is fundamentally unrealistic for real-world data, which often takes on the inter-granularity class disagreements exhibited in type (2), and hence encounters the risk of the word-and sentence-level topics overriding the true documentlevel class. for example, the education-news article of type (2) uses strong political terms to discuss the government being the subject matter of a teacher's lesson to students, which may mistakenly lead it being classified as "politics" instead of "education". in general, documents may also feature several indeterminate sentences that do not confidently indicate any sort of class alignment. the existing classification methods unfortunately do not address such inconsistency concerns among different levels of granularity. several approaches @xcite generate pseudo-labels based on the individual words or sentences instead of the entire document and evaluate their confidence in aligning to a single class. this may lead to two glaring issues: (1) many words and sentences are only considered high quality and class-indicative based on the context of their parent document, and (2) they could be representative of classes different from the document's fundamental topic (e.g., "checks and balances" and "judicial branches" in conflict with "education" in figure 1 ). on the other hand, another approach @xcite heavily relies on both word-level and document-level, but independently-resulting in both overconfident and vague document representations used for generating pseudo-labels. in this study, we attack this problem by utilizing all three levels of text granularity (words, sentences, and documents) in a manner that allows them to mutually enhance each other's significance in understanding which class a given document aligns with. furthermore, we consider a document's class distribution instead of a single class such that we rank documents with a narrower distribution higher and can correct any initial misclassifications (e.g., @xmath0. doing so will allow us to exploit three levels of insight into the document's local (word/sentence) and global (document) class distribution and jointly factor them into our understanding of the document and final pseudo-training dataset construction. in this study, we propose a novel framework megclass to perform extremely weakly supervised text classification using mutually enhancing text granularities. specifically, when only the class surface names are given, megclass first identi-fies class-indicative keywords and uses them to represent classes and sentences. megclass then automatically weighs the importance of each sentence in classification for more accurately estimating a document's class distribution. megclass further leverages a multi-head attention network to learn a document representation that reflects the critical information at multiple text granularities. finally, megclass takes the most confident document representations to enhance our initial wordbased class representations through our iterative feedback approach, helping our model better understand what an e.g., "education" document should look like at all levels of granularity. because our multi-granularity approach can handle all three document types when constructing the final pseudotraining dataset, we allow our fine-tuned final classifier to be more robust to challenging real-world documents. comparing with existing weakly and extremely weakly supervised methods, megclass achieves a stronger performance on most datasets, especially with longer documents and fine-grained classes. our contributions are summarized as follows: reproducibility: we release our data and source code foot_0 to facilitate further studies. ## conclusion in this work, we propose our novel method meg-class for extremely weakly supervised text classification by exploiting mutually-enhancing text granularities. megclass learns new document representations that reflect the strongest class indicators with respect to all three levels of text granularities. megclass identifies the top documents with a narrower class distribution (more confidently single-topic) to use as feedback for the next iteration, and it ultimately constructs a pseudo-dataset of documents for fine-tuning a pre-trained text classifier. we demonstrate megclass's strong performance and stability through extensive experiments on varying domains (news and reviews) and label spaces (topics, locations, and sentiments). due to megclass's novel use of mutually enhancing text granularities, further exploration can be done to utilize megclass for information extraction at varying text granularities. furthermore, more sophisticated nonlinear dimensionality reduction methods can be a potential aspect to explore. finally, handling class hierarchies may improve megclass's fine-grained classification abilities.