Towards Open-Domain Topic Classification (2022)

Abstract: We introduce an open-domain topic classification system that accepts user-defined taxonomy in real time. Users will be able to classify a text snippet with respect to any candidate labels they want, and get instant response from our web interface. To obtain such flexibility, we build the backend model in a zero-shot way. By training on a new dataset constructed from Wikipedia, our label-aware text classifier can effectively utilize implicit knowledge in the pretrained language model to handle labels it has never seen before. We evaluate our model across four datasets from various domains with different label sets. Experiments show that the model significantly improves over existing zero-shot baselines in open-domain scenarios, and performs competitively with weakly-supervised models trained on in-domain data.

## introduction text classification is a fundamental natural language processing problem, with one of its major applications in topic labeling @xcite . over the past decades, supervised classification models have achieved great success in closed-domain tasks with large-scale annotated datasets @xcite @xcite . however, they are no longer effective in open-domain scenarios where the taxonomy is unbounded. retraining the model for every new label set often incurs prohibitively high cost in the sense of both annotation and computation. by contrast, having one classifier that is flexible with unlimited labels can save such tremendous efforts while keeping the solution simple. therefore, in this work, we build a system for open-domain topic classification that can classify a given text snippet into any categories defined by users. at the core of our system is a zero-shot text classification model. while supervised models are typically insensitive to class names, a zero-shot model is usually label-aware, meaning that it can understand label semantics directly from the name or definition of the label, without accessing any annotated examples. our model te-wiki combines a textual entailment (te) formulation with wikipedia finetuning. specifically, we construct a new dataset that contains three million articlecategory pairs from wikipedia's subcategory graph, and finetune a pretrained language model (e.g. bert) to predict the entailment relations between articles and their associated categories. we simulate the diversity in open-domain classification with the wide coverage of wikipedia, while preserving label-awareness through an entailment framework. in our benchmarking experiments, te-wiki outperforms all previous zero-shot methods on four benchmarks from different domains. it also shows competitive performance against weaklysupervised models trained on in-domain data. by learning from wikipedia, our method does not require any data that is specifically collected from the evaluation domains. on the other hand, since our model is label-aware, it can flexibly classify text pieces into any labels outside wikipedia. finally, we compare our system against humans for further insights. we show that even humans are sometimes confused by ambiguous labels through a crowdsourcing study, which explains the performance gap between open-domain and supervised classification. the gap is reduced significantly when label meanings are clear and well aligned with the semantics of text. we also use an example to illustrate the negative effect of a bad label name. through the analysis, we demonstrate the importance of choosing proper label names in opendomain classification. ## related work open-domain zero-shot text classification was first studied in the nlp domain in @xcite (under the name "dataless classification") as a method that classifies solely based on a general knowledge source and does not require any indomain data, whether labeled or not. it was proposed to embed both the text and labels into the same semantic space, via explicit semantic analysis, or esa @xcite , and pick the label with the highest relevance score. this idea was further extended to hierarchical @xcite and cross-lingual @xcite text classification. later on, @xcite called this protocol "label fully unseen" and proposed an entailment approach to transfer knowledge from textual entailment to text classification. it formulates an n-class classification problem as n binary entailment problems by converting labels into hypotheses and the text into the premise, and selects the premise-hypothesis pair with highest entailment score. more recently, another concurrent work @xcite proposed to explore resources from wikipedia for zero-shot text classification, but with a different formulation. there are many other methods that also require less labeling than supervised classification, though in slightly different settings. for example, previous works have explored to generalize from a set of known classes (with annotation) to unknown classes (without annotation) using word embed-dings of label names @xcite @xcite , class correlation on knowledge graphs @xcite , or joint embeddings of documents and labels @xcite . besides, weakly supervised approaches @xcite learn from an unlabeled, but in-domain training set. given a set of predefined labels, a label-aware knowledge mining step is first applied to find class-specific indicators from the corpus, followed by another self training step to further enhance the model by propagating the knowledge to the whole corpus. however, none of these approaches are suitable for building an open-domain classification system. they either require domain-specific annotation or knowing test labels beforehand. ## system description we present details about our open-domain topic classification system, starting with an overview of our web interface, followed by the backend model. ## evaluation we evaluate all the backend models of our system on four classification benchmarks to compare their performance. we also compare them against weakly-supervised and supervised models to quantify how much we can achieve without any domainspecific training data. ## importance of label names since zero-shot classifiers understand a label by its name, the quality of label names can be a important performance bottleneck in designing open-domain text classification systems. to study this, we con- we summarize the results in we also calculate the accuracy on examples where all 5 workers agree, as in row 2&4 in ## conclusion we introduce a system for open-domain topic classification. the system allows users to define customized taxonomy and classify text with respect to that taxonomy at real time, without changing the underlying model. to build a powerful model, we propose to utilize wikipedia articles and categories and adopt an entailment framework for zero-shot learning. the resulting te-wiki outperforms all existing zero-shot baselines in open-domain evaluations. finally, we demonstrate the importance of choosing proper label names in open-domain topic classification through a crowdsourcing study.