Self-supervised Topic Taxonomy Discovery in the Box Embedding Space (2024)

Abstract: Topic taxonomy discovery aims at uncovering topics of different abstraction levels and constructing hierarchical relations between them. Unfortunately, most prior work can hardly model semantic scopes of words and topics by holding the Euclidean embedding space assumption. What’s worse, they infer asymmetric hierarchical relations by symmetric distances between topic embeddings. As a result, existing methods suffer from problems of low-quality topics at high abstraction levels and inaccurate hierarchical relations. To alleviate these problems, this paper develops a Box embedding-based Topic Model (BoxTM) that maps words and topics into the box embedding space, where the asymmetric metric is defined to properly infer hierarchical relations among topics. Additionally, our BoxTM explicitly infers upper-level topics based on correlation between specific topics through recursive clustering on topic boxes. Finally, extensive experiments validate high-quality of the topic taxonomy learned by BoxTM.

## introduction taxonomy knowledge discovery, the process of extracting latent semantic hierarchies from text corpora, is a crucial yet challenging research field. for text mining applications, it can serve as the foundation of complex question answering @xcite and recommendation systems @xcite ). an important line of research focuses on learning word-level or entity-level taxonomies @xcite , but such products may encounter problems of low coverage, high redundancy, and limited information (zhang * corresponding author. @xcite . since a topic can cover the semantics of a set of coherent words, some works propose to use topics as the basic taxonomic units. taking the topic taxonomy of the arxiv website as an example, ''computer science'' is an academic discipline highlighted by general keywords of ''information'', @xcite it involves various sub-fields such as ''computation and language'' and ''computer vision'', which have specific keywords of ''language'' and ''image'', respectively. with this topic taxonomy, users can readily retrieve papers of interest and explore related research fields. early methods for topic taxonomy discovery @xcite @xcite take a probabilistic perspective originated from lda @xcite . in these approaches, each topic is a distribution across words. a document is generated by sampling topics in different levels, and then sampling words from the selected topics iteratively. as a more flexible and efficient solution compared with probabilistic models, the hierarchical neural topic models (hntms) that adopt deep generative models and neural variational inference (nvi) have been developed in recent years @xcite . with remarkable developments of text representation learning @xcite @xcite , mining topic taxonomy in the high-quality embedding space has become a promising idea. particularly, the latest hntms @xcite extend the embedded topic modeling (etm) @xcite ) method to topic taxonomy discovery. with the assumption that topics and their keywords are close in the embedding space, these models utilize dot products between topic and word embeddings to infer topic-word distributions. in parallel, some other methods conduct recursive clustering on word embeddings to construct topic taxonomy directly @xcite . such clustering-based methods often train the word embedding space on local contexts, which helps them capture accurate word semantics. unfortunately, they have difficulty in exploiting global statistics of word occurrences, such as bag-of-words and tf-idf representations. as a result, topics mined by these methods are highly coherent but may not be representative of the entire corpus. due to this flaw of clustering-based methods, hntms persist as the prevailing paradigm for topic taxonomy discovery. despite the impressive performance of existing hntms, they suffer from the following problems. (1) suboptimal representation: most of these methods are limited in modeling semantic scopes of words and topics at different abstraction levels using classic point embeddings @xcite . instead, geometric embeddings such as hyperbolic and box embeddings are more effective representations for structured data, including knowledge graphs and taxonomies @xcite . although hyper-miner @xcite attempts to uncover topic taxonomy within a geometric embedding space, it simply replaces point embeddings in traditional hntms with hyperbolic embeddings and lacks in-depth analysis. this makes hyperminer suffer from the following problems. (2) topic collapse: prior models struggle to learn high-quality topics, especially at higher abstraction levels. in particular, their top-level topics often degenerate into clusters of meaningless common words @xcite . (3) inaccurate hierarchy relations: many existing hntms rely on the symmetric distance metric (i.e., dot product) to infer the asymmetric hierarchy relations among topics. such approximation results in an inaccurate hierarchical topic structure. considering the above challenges, we propose to learn topic taxonomy in the box embedding space @xcite and develop a box embedding-based topic model (boxtm) 1 following the framework of nvi. figure 1 shows the differences of the topic taxonomy discovery processes in the point embedding space and the 1 the source code of our model is available in public at: https://github.com/luyy9apples/boxtm . box embedding space, which are adopted by most existing hntms and our boxtm, respectively. and the topic taxonomy discovery process in the hyperbolic embedding space is similar to that in the point embedding space. specifically, boxtm represents a topic or word as a hyperrectangle instead of a point, whose volume is proportional to the size of its semantic scope. in other words, the box embedding of a general topic covers a relatively larger region than that of a specific topic. additionally, we conduct recursive clustering on the box embeddings of the lower-level topics to extract the upper-level topics. this approach leverages the connection between descendant topics to precisely capture the semantics of the upper-level topics, which can address the topic collapse problem caused by unguided upper-level topic mining. intuitively, we employ symmetry and asymmetry distance metrics defined in the box embedding space respectively to capture similarity and hierarchy relations among topics. in summary, the main contributions of this paper are as follows: ## proposed method in this section, we introduce the proposed boxtm in detail. firstly, we propose the box embedding-based document generative process in section 4.1, which is the main framework of boxtm. in general, boxtm infers topic distributions via the symmetric affinities and semantic scopes of topics and words in the box embedding space. additionally, the hierarchical relations are modeled by the values of the asymmetric metric between topic boxes. subsequently, we introduce more detailed designs of boxtm, including a novel workflow of recursive topic clustering for upper-level topic mining (section 4.2) and two self-training tasks for modeling the semantic scopes of words and topics better (section 4.3). finally, we introduce the learning strategy of boxtm in section 4.4. ## conclusion this paper proposes a novel model called boxtm for self-supervised topic taxonomy discovery in the box embedding space. specifically, boxtm embeds both topics and words into the same box embedding space, where the symmetric and asymmetric metrics are defined to infer the complex relations among topics and words properly. additionally, instead of initializing topic embeddings randomly, boxtm uncovers upper-level topics via recursive clustering on topic boxes. while our boxtm has achieved state-of-the-art performance in multiple evaluation experiments, it also exhibits a limitation in efficiency. the point model, a variant of boxtm that replaces the box embeddings with point embeddings, is trained for 0.22 gpu (gtx 1080 ti) hour on the 20news dataset. due to the extra computation of box op-erations compared to dot product, boxtm costs about 1.0 hour, which reveals the research space for efficient computation of box embeddings.