Advancing Topical Text Classification: A Novel Distance-Based Method with Contextual Embeddings (2023)

Abstract: This study introduces a new method for distance-based unsupervised topical text classification using contextual embeddings. The method applies and tailors sentence embeddings for distance-based topical text classification. This is achieved by leveraging the semantic similarity between topic labels and text content, and reinforcing the relationship between them in a shared semantic space. The proposed method outperforms a wide range of existing sentence embeddings on average by 35%. Presenting an alternative to the commonly used transformer-based zero-shot general-purpose classifiers for multiclass text classification, the method demonstrates significant advantages in terms of computational efficiency and flexibility, while maintaining comparable or improved classification results.

## introduction topical text classification remains an important task in text classification because it allows users to explore, analyze and organize large text collections. however, the nature of topical text classification is subjective as the content and context of the text are often perceived differently based on the intended audience. to address this, methods that dynamically explore topics are necessary, one of which is unsupervised text classification. this approach allows classifying text collections based on a predefined list of topics for further analysis. @xcite outlined three primary techniques for unsupervised text classification: 1) evaluating the frequency of class labels in a text, 2) measuring the distance between class labels and text in a shared vector space, and 3) leveraging natural language inference with pre-trained classifiers to ascertain if a class label can be deduced from the text. with the advancement of transformer models, the latter method has gained increasing attention in the nlp community due to its successful outcomes @xcite . in this study we show that task-specific sentence embeddings trained on transformer models for distance-based topical text classification, can provide a flexible and efficient alternative to the aforementioned methods. in this study, we undertake a comprehensive examination of unsupervised topical text classification utilizing contextual embeddings, and propose a methodology for generating sentence embeddings that are more appropriate for this task. to achieve this objective, we first evaluate a diverse array of existing contextual embeddings and their derived sentence embeddings on seven datasets across a broad spectrum of genres and topics. subsequently, we explore the various options for training custom sentence embeddings, including the choice of training data, base models, and loss functions, with the aim of identifying the most suitable configuration for the given task. finally, we assess the benefits and limitations of our proposed method. the paper unfolds as follows: section 2 outlines the previous research on unsupervised text classification; section 3 presents the proposed method; section 4 explains experiment setup; section 5 presents evaluation results. ## related work unsupervised text classification, also referred to as dataless or zero-shot text classification, relies on semantic relatedness between class labels and documents for classification without requiring training data. @xcite pioneered this concept, employing explicit semantic analysis (esa) and wikipedia as an external knowledge base to encode class labels and document texts within a single semantic space and classifying them based on proximity. this approach was further extended by @xcite for hierarchical text classification and by @xcite for cross-lingual text classification. recently, the development of sentence embeddings introduced by @xcite , with improved text representation, added additional push for improvement of various nlp tasks such as information retrieval and semantic search. subsequent enhancements to sentence embeddings, like sgpt @xcite , showcased their promising potential. @xcite introduced lbl2transformervec, an enhancement of the previously introduced lbl2vec for unsupervised text classification using sentence embeddings. ## proposed method we formulate the problem of unsupervised topical text classification as follows: given a set of predefined topic categories, the objective is to classify texts based on the semantic relatedness between the topic name and the text content. taking into account large amounts of data involved, and rapid changes in the data and topical categories, this classification should be done as efficiently as possible. of the two major methods of unsupervised text classification, the distance-based method with neural word embeddings is more computationally efficient but the transformer-based zero-shot classifiers has been shown to be more accurate due to its ability to better capture text semantics. to combine the advantages of these two methods we propose replacing the often used neural word embeddings with transformer-based embeddings tailored for this task. for this purpose, we employ sentence embeddings introduced @xcite to embed both texts and topic names into a shared semantic space. however, instead of the typical training of sentence embeddings on text pairs that preserve the same level of abstraction and granularity, we propose training task-specific sentence embeddings on tag-text pairs, where tags serve as proxies for topics with a higher level of abstraction. to better demonstrate the distinctions between traditional and proposed methods, we provide examples of training data for both approaches. snli and ms marco datasets typically are used for training sentence embeddings: snli foot_0 . sentence 1: a senior is waiting at the window of a restaurant that serves sandwiches, sentence 2: a person waits to be served his food. ms marco 2 . query: when was the town of farragut tn incorporated, passage text: in january of 1980, residents decided to incorporate by an overwhelming margin. the town of farragut was incorporated on january 16, 1980, with the first board of mayor and alderman elected on april 1, 1980. our approach suggests leveraging resources similar to wikipedia categories and new york times descriptors for training task-specific sentence embeddings: wikipedia. text: sojunghwa sojunghwa is a century korean concept that means little china referring to the joseon dynasty after the qing dynasty conquered the han ming dynasty koreans thought that barbarians ruined the center of civilization of the world and so confucianist joseon korea had become the new center of the world replacing ming china hence the name little china tokugawa japan and vietnam also had a similar belief in themselves after the qing dynasty had taken over china based on sinocentrism the belief that china was the center of civilization in the world the chinese believed that korea then a tributary state was a highly civilized state meanwhile the koreans considered japanese and jurchen people to be barbarians or beasts under the distinction, general category: philosophy by region. nyt ldc. text: no one here knew diane o'dell's secret. she was, said people who live in this wide spot in on a narrow rural road, a pleasant if somewhat standoffish neighbor and an affectionate mother. "everybody in the area knows everybody," said john karpauitzs, who lives a few doors down from the gray, tumble-down house that ms. o'dell shared with her common-law husband and their five children. "she was quiet. she kept mostly to herself. not much else to say about her." there was nothing in her behavior, neighbors said, to indicate that she traveled with the corpses of three of her other children around the country for a decade. ms. o'dell, 49, was charged in sullivan county, n.y., on tuesday with murdering three babies she bore in the early 1980's in sullivan county... ## conclusion & future work to address these limitations, we introduce the concept of task-specific sentence embeddings that enforce the relationship between topic labels and text in a shared semantic space. this enhances their suitability for distance-based topical multiclass text classification. our method is model and training data agnostic and can be applied with various transformer-based models and trained on plain texts tagged with relevant topic labels. the results demonstrate comparable or improved performance compared to state-of-the-art transformerbased zero-shot general-purpose classifiers and offer additional benefits such as increased computational efficiency and greater flexibility in topical text classification. the promising avenues for future research in-volve addressing the limitations of shallow semantic representation of texts using sentence embeddings and extending the proposed method to enable multilabel topical text classification.