Topic Modeling for Maternal Health Using R eddit (2021)

Abstract: This paper applies topic modeling to understand maternal health topics, concerns, and questions expressed in online communities on social networking sites. We examine Latent Dirichlet Analysis (LDA) and two state-of-the-art methods: neural topic model with knowledge distillation (KD) and Embedded Topic Model (ETM) on maternal health texts collected from Reddit. The models are evaluated on topic quality and topic inference, using both auto-evaluation metrics and human assessment. We analyze a disconnect between automatic metrics and human evaluations. While LDA performs the best overall with the auto-evaluation metrics NPMI and Coherence, Neural Topic Model with Knowledge Distillation is favorable by expert evaluation. We also create a new partially expert annotated gold-standard maternal health topic

## introduction evidence suggests that poor quality maternal and newborn care is responsible for nearly 60% of the estimated 5 million deaths each year globally @xcite . this situation spouses urgent demands of artificial intelligence. ai has potential applications in automating aspects of service delivery such as basic counselling, thereby reducing the burden on health systems. in this work, we hope to help improving healthcare counselling ai by understanding maternal health related content. what are people concerning about regarding maternal health? and among state-of-the-art techniques, which model is the best at extracting and interpreting those highly-professional topics? with the development of the online communities and social media, advanced text mining techniques are widely needed when applied to domains. it is necessary to understand a question in order to answer it, and the first step towards understanding the content of a document is to determine which topics that document addresses. therefore, many successful conversational agents employ topic models to encourage more coherent dialogue @xcite . topic models can provide legible and concise representations of both the entire corpus and individual documents because they represent topics as ranked word lists and documents regarding their probable topics. therefore, while mainly applied to mining the text content, topic models can also classify the topics for new documents by inferring the latent topic distribution. in classical topic models like latent dirichlet allocation (lda), distributions over the latent variables are estimated with variational inference algorithms (em) or gibbs sampling @xcite . however, recently, with the development of deep learning methods, now there are neural topic models that incorporate additional information and leverage the variational autoencoder (vae) framework for latent variable inference. particularly, pre-trained transformerbased language models (e.g., bert, @xcite ) is employed to fine-tune on a wide variety of nlp problems; some models combine the advantages of pre-trained transformers and topic models. this work contributes the following: ## related work and background there are a multitude of state-of-the-art topic models adapted from the lda model. neural topic models @xcite surpass traditional methods in architecture by using various forms of neural networks and can be applied to either labeled or unlabeled data. hoyle et al. (2020) further improves neural topic models by knowledge distillation (kd). another thread of innovations improve models in terms of semantic meaning of words, for example, dieng et al. (2020) incorporates word embedding to topic models. given that our dataset is unlabeled, this work focus on two state-of-the-art models, the kd model and the embedding topics model (etm), and compares their performances to lda. we briefly summarize the details of lda, kd and etm in this section. ## discussion measurement incompatibility there is an incompatibility between human assessment and the auto-evaluation metrics for topic quality. in terms of the auto-evaluation metric coherence, lda always performs the best, which is inconsistent with the human assessment. in terms of npmi, kd dominates in the 10-topic setting, but lda still performs better in the 25-topic, 50-topic, and 100topic settings. in terms of topic uniqueness, kd is the best in 10-topic, 25-topic setting, which is the same as human evaluation. topic repetitiveness although fixed values are set for the number of topics when training on the corpus, fewer topics could be identified from the generated word lists because of the overlapping among the lists. for example, when generating 10 topics, kd has 9 topics identified by the expert, while topic 3 and topic 5 are categorized as "vaginal health". for etm, there are more overlapped generated lists: topic "pregnancy", "vaginal health", "menstruation", and "labor and delivery" all include 2 generated lists. ## conclusion