Coordinated Topic Modeling (2022)

Abstract: We propose a new problem called coordinated topic modeling that imitates human behavior while describing a text corpus. It considers a set of well-defined topics like the axes of a semantic space with a reference representation. It then uses the axes to model a corpus for easily understandable representation. This new task helps represent a corpus more interpretably by reusing existing knowledge and benefits the corpora comparison task. We design ECTM, an embedding-based coordinated topic model that effectively uses the reference representation to capture the target corpus-specific aspects while maintaining each topic’s global semantics. In ECTM, we introduce the topic- and document-level supervision with a self-training mechanism to solve the problem. Finally, extensive experiments on multiple domains show the superiority of our model over other baselines.

## introduction we often ask questions about well-defined topics when we read articles (e.g., news/research articles). e.g., in news domain, such a question can be like "what is in entertainment today? (it's about oscar 2022)" or in academic domain, it can be like "what is trending in machine learning in 2016? (it's about deep learning)". this is a practical human trait while understanding information in a text corpus. rather than finding arbitrary topics, people often want to explore the text based on some welldefined topics. e.g., in news domain, such topics are business, politics, sports, etc. a well-defined topic is not merely a name (i.e., surface name); it generally also has a representation (e.g., word distribution) that can be obtained from a large text corpus, which we may call the reference representation. e.g., in such representation of the topic sports, words like play, game, and 1 code and data are available at https://github.com/ pritomsaha/coordinated-topic-modeling player are mainly dominated. similarly, for politics, these are vote, election, party, etc. however, a topic's representation generally deviates from its reference to other corpora. e.g., while describing a news corpus specific to the usa, it is likely to use words such as nfl, baseball or football to represent sports, while words like cricket, ipl or run are more relevant for india. the above scenario can be explained by the concept of the coordinate system in geometry. specifically, we can consider a set of well-defined topics as the axes or basis of semantic space with their reference representation β r . e.g., in figure 1 , two topics c 1 :sports and c 2 :politics represents two axes (i.e., reference axes) defined by their reference representations β r 1 and β r 2 respectively. defining a target corpus by these two topics is analogous to finding a representation in the space defined by the reference axes. figure 1 shows such target representation β u sa and β in dia as coordinates by reference axes for corpora specific to usa and india, respectively. in practice, the above interpretation of defining a corpus can be helpful from several perspectives. first, it has better interpretability than traditional document modeling approaches like topic models @xcite that discover some unknown topics. it defines a corpus over well-defined topics by finding their corpus-specific representations. thus, users do not need much ef-fort to understand unknown topics; instead, they easily grasp the corpus-specific aspects of given well-known topics. second, it facilitates the comparability between two corpora. it uses a predefined set of topics with their reference representation as the axes to describe a corpus in a semantic space. thus, two corpora represented by the same axes can be easily comparable by studying their relative position in the coordinates. third, it allows reusability by utilizing existing information to explore new knowledge. as the reference representation is already known information about some topics, resuing it to model a new corpus in a similar domain is intuitive and helpful. specifically, this is useful for small target corpora as traditional models are not typically effective in such cases. considering the above motivations, we thus formalize a new problem, coordinated topic modeling (ctm). it takes a target corpus d, a reference representation β r for k well-defined topics with their surface names c. as output, it aims to learn a target representation β to best define d. in many cases, we can obtain a set of well-defined topics with their representation. e.g., there may exist some large corpora in a particular domain annotated with topics, and there are effective existing methods @xcite to obtain the reference representations. specifically, we use labeled lda @xcite to get reference representation (more details on section 3.1) some previous attempts incorporate prior knowledge into topic models to impose more interpretability. one such work takes document-level supervision by providing topic labels of all or a subset of documents @xcite . while they improve the predictive ability of unsupervised topic models, they require a massive set of annotated documents to be effective. an alternative way is called topic-level supervision by providing seed words for each topic @xcite . although they make the topics converge toward the user's interest, the seed words should be only from the target corpus vocabulary, which may be impractical in many cases. finally, the categoryguided topic mining @xcite ) considers the topic names as the only supervision for mining user-desired discriminative topics. however, it also assumes that the name of each topic appears in the corpus. moreover, none of the above works does impose requirements of making topics compara-ble over multiple corpora (see appendix a for a detailed discussion on related work). from the above discussion, we identify the following two requirements that the solution of ctm needs to meet. (1) the target representation should be learned based on reference representation by capturing the target corpus-specific aspects; (2) it also needs to relate the documents in the corpus with the global semantics of each topic represented by its surface name so that it maintains general interpretability and comparability. although previous work considers mining topics from user guidance, none of those fulfill these two requirements. there are several challenges to fulfilling the above requirements. the first challenge is handling the vocabulary mismatch problem. as the vocabulary of β r and d can be the different, β r cannot be directly calibrated into the target β. to solve this problem, we propose a method called reference projection. here, the main idea is first to generate a proxy representation βr of the target β having the same vocabulary with reference β r , making it comparable for supervision. it then enforces βr and β r to be as close as possible, which indirectly allows the target β to be guided by given β r even if they have different vocabularies. moreover, another benefit of βr is that it enables directly comparing two corpora given the same topics set. the second challenge is providing surface names guidance. we only know the surface name of each given topic without further knowledge of how each document corresponds to them. therefore, we generate document-level supervision from the surface names using the textual entailment approach @xcite . it imitates how humans determine the topic(s) of a document by filling in a template (e.g., "this document is @xmath0. in this paper, we utilize a pre-trained textual entailment model @xcite with given surface names to generate document-level distribution matrix θ t . this θ t later guides ctm by relating the global semantics of surface names with given documents. the final challenge is combining two supervisions. now, we have topic-level supervision from prior projection and document-level supervision from the textual entailment model. to combine these two supervisions, we propose a method called embedding based coordinated topic model, ectm. it exploits the architecture of an embedded topic model (etm) @xcite . the main idea is to regularize the objective of etm using our two proposed supervisions. to further generalize ectm, we employ a mechanism similar to selftraining @xcite where we iteratively use the model's current output to update θ t . our contributions can be summarized as follows. first, we propose coordinated topic modeling, a new problem for modeling a text corpus using well-defined topics as reference. second, we develop an embedding-based framework for solving the problem. it uses given reference representation to effectively model a corpus while maintaining the semantics of given surface names. third, we propose methods for generating topic-level and document-level supervisions using an introduced projection mechanism and the textual entailment approach, respectively. we then combine these two supervisions into a unified model to solve the ctm problem. finally, we conduct a comprehensive set of experiments on multiple domains for different tasks, demonstrating our framework's superiority against strongly designed baselines. ## proposed methodology overview. to solve the ctm problem, we design a method ectm (sec. 2.2) based on an embedded topic model (etm) @xcite ) by imposing our requirements. first, we incorporate a topic-level supervision (sec. 2.2.1) from given reference representation β r by introducing a mechanism called reference projection. it generates a proxy representation of intended β with the same vocabulary dimension with β r , thus enabling supervision. second, we include a document-level supervision (sec. 2.2.2) from the given surface name of each topic by employing the textual entailment approach @xcite . finally, we combine two supervisions (sec. 2.2.3) by regularizing the etm's objective, to further generalize the model, we employ a self-training @xcite by iteratively using the model's current output to update the supervision. the rest of the section first reviews etm and then presents our ectm. ## experiments in this section, we employ empirical evaluations, which are designed mainly to answer the following research questions (rqs): rq1. how effective is the ectm quantitatively in terms of the quality of topics for the ctm problem and text classification performance? rq2. how does the ectm qualitatively perform in terms of interpretability and distinctiveness of generated topic words? rq3. how does ectm benefit the task of comparing multiple corpora? rq4. how does each part of ectm contribute to its performance? ## conclusion in this paper, we propose a new problem called coordinated topic modeling that uses a set of welldefined topics to describe a target corpus. it takes a reference representation of topics with their surface names and calibrates those representing the target corpus. the proposed problem describes corpora in a more interpretable and comparable way. to solve the problem, we design an embedding-based coordinated topic model that leverages topic-level and document-level supervision with a self-training mechanism. a set of empirical evaluations demonstrate the superiority of our approach over other strong baselines in multiple datasets.