Multi-Surrogate-Objective Optimization for Neural Topic Models (2025)

Abstract: Neural topic modeling has substantially improved topic quality and document topic distribution compared to traditional probabilistic methods. These models often incorporate multiple loss functions. However, the disparate magnitudes of these losses can make hyperparameter tuning for these loss functions challenging, potentially creating obstacles for simultaneous optimization. While gradient-based Multi-objective Optimization (MOO) algorithms offer a potential solution, they are typically applied to shared parameters in multi-task learning, hindering their broader adoption, particularly in Neural Topic Models (NTMs). Furthermore, our experiments reveal that naïve MOO applications on NTMs can yield suboptimal results, even underperforming compared to implementations without the MOO mechanism. This paper proposes a novel approach to integrate MOO algorithms, independent of hard-parameter sharing architectures, and effectively optimizes multiple NTMs loss functions. Comprehensive evaluations on widely used benchmark datasets demonstrate that our approach significantly enhances baseline topic model performance and outperforms direct MOO applications on NTMs.

## introduction traditional topic models @xcite @xcite have become widely used due to their ability to uncover hidden topics from unstructured data and their interpretability. although such models have achieved some success, they often suffer from inefficient and laborious parameter inference @xcite . to address these limitations, neural topic models (ntms) have emerged as a promising solution, achieving effectiveness in various domains such as content organization @xcite ## background denote @xmath0, based on a vocabulary containing v words. the objective of neural topic modeling targets to discover k latent topics. we have β = (β 1 , . . . , β k ) ∈ r v ×k is the topic-word distribution matrix of all k topics. given word embedding dimension l, we define @xmath1, . . . , w v ) ∈ r v ×l as the word embedding matrix and @xmath2, . . . , t k ) ∈ r k×l as the topic embedding matrix. recent state-of-the-art neural topic models @xcite @xcite no longer decompose β into two components: word embeddings w and topic embeddings t ; but instead represent β as: then the bow representation of a document x d is encoded through neural networks. the parameters of the gaussian distribution are computed with mean µ = h µ (x d , γ) and the diagonal covariance matrix is σ = diag(h σ (x d , γ)), where γ is the parameter of these inference networks. latent variable z is subsequently sampled from the posterior distribution q(z|x d ) = n (z|µ, σ) employing the reparameterization trick proposed by @xcite . the topic proportions are then derived by using the softmax function, resulting in θ = softmax(z). next, bow representation is reconstructed through topicword distribution matrix β and topic proportion θ from a multinomial distribution: xbow ∼ multi(softmax(βθ)). finally, the objective function for the topic model comprises two components: a reconstruction term and a regularization term, defined as follows: recent modern neural topic models frequently integrate multiple objective functions into their overall objective to enhance both the performance of topic quality and document representations. for instance, ecrtm @xcite adds the embedding clustering regularization objective l ecr to effectively mitigate topic collapsing. neuro-max @xcite adds three objectives: embedding clustering regularization l ecr , group topic regularization l gr , and information noise-contrastive estimation l infonce . to effectively manage these multiple objectives, gradient-based multi-objective optimization (moo) methods are employed. several notable frameworks, including mgda @xcite , pcgrad @xcite , imtl @xcite , fairgrad @xcite , and ex-cessmtl @xcite , propose gradient-based mtl methods aimed at finding solutions on the pareto front. let k represent the total number of tasks. we define δ share as the shared parameters while δ 1 , δ 2 , . . . , δ k denote task-specific parameters. these approaches commonly seek the update direction as a linear combination of individual task gradients: ## conclusion this paper introduces a novel approach to address the challenge of optimizing multiple loss functions in ntms. directly applying moo methods is often ineffective in this context. our proposed method, msoo, overcomes this limitation by employing surrogate loss functions with more balanced magnitudes, which consider not only individual ob-jectives but also the total loss. comprehensive experiments demonstrate that msoo significantly improves the performance of baseline topic models, outperforming direct applications of moo. we explore both static and adaptive strategies for weighting the surrogate losses, with the adaptive approach showing particularly strong potential.