Understanding Narratives from Demographic Survey Data: a Comparative Study with Multiple Neural Topic Models (2022)

Abstract: Fertility intentions as verbalized in surveys are a poor predictor of actual fertility outcomes, the number of children people have. This can partly be explained by the uncertainty people have in their intentions. Such uncertainties are hard to capture through traditional survey questions, although open-ended questions can be used to get insight into people’s subjective narratives of the future that determine their intentions. Analyzing such answers to open-ended questions can be done through Natural Language Processing techniques. Traditional topic models (e.g., LSA and LDA), however, often fail to do since they rely on co-occurrences, which are often rare in short survey responses. The aim of this study was to apply and evaluate topic models on demographic survey data. In this study, we applied neural topic models (e.g. BERTopic, CombinedTM) based on language models to responses from Dutch women on their fertility plans, and compared the topics and their coherence scores from each model to expert judgments. Our results show that neural models produce topics more in line with human interpretation compared to LDA. However, the coherence score could only partly reflect on this, depending on the corpus used for calculation. This research is important because, first, it helps us develop more informed strategies on model selection and evaluation for topic modeling on survey data; and second, it shows that the field of demography has much to gain from adopting NLP methods.

## introduction demographers are interested in the number of children people have or will have, also referred to as fertility. in trying to understand future fertility, researchers have studied fertility intentions, i.e. plans to have children in the future. the usefulness of measurements of fertility intentions are often debated among demographers due to the gap between intentions and fertility outcomes @xcite and a large portion of respondents being uncertain about their intentions @xcite . it is proposed that this is because fertility intentions are contextual and largely depend on subjective narratives @xcite . therefore, understanding these narratives might be the key for advancing theories on the fertility decision-making process. open-ended questions (oeqs) help researchers obtain "top-of-the-head" answers from respondents, and they have been employed in previous qualitative demographic studies (e.g., interviews with a small sample of respondents, sometime deliberately non-representative of the whole population) @xcite . however, to expand the analysis to a larger and generalizable sample of the population, an automatic process of extracting and quantifying themes from responses is needed as an initial exploratory data analysis. this objective can be met with topic modeling methods. latent dirichlet allocation (lda; @xcite ) is one of the most popular topic modeling algorithms, which is based on co-occurrence of words. lda's performance on short texts, such as online survey responses, may be compromised due to the small number of co-occurrences. to overcome this problem, many topic models that support incorporating prior language knowledge (e.g. word embeddings or language models) have been developed, such as sparse contextual hidden and observed language autoencoder (scholar; @xcite ). this model uses variational autoencoder (vae) to incorporate word2vec @xcite ) embeddings. a different example is bidirectional encoder representations from transformers (bert, @xcite ), one of the most prominent language models, that has also been incorporated in topic modeling tasks, e.g. in combined topic model (combinedtm; @xcite ) and bertopic @xcite . in this paper, our first aim is to compare the performance of multiple topic modelling algorithms on responses to open questions. such an analysis of survey responses is rare, and it is an open question how well these topic models do on short texts from relatively few respondents ( 400), a scale larger than usual qualitative studies. to achieve this, we implemented and evaluated four models (lda, scholar, combinedtm and bertopic), trained on the fertility response dataset to provide unsupervised topics. we then compare metrics of quality across these diverse methods through comparable implementations. building on the comparative study of @xcite , we further compare metrics and their difference to human annotations respectively. our study contributes to the literature by: 1) evaluating the performance of multiple topic models incorporating prior knowledge; 2) examining the correlation between metrics and human judgments; 3) modeling topics on dutch texts of online survey responses and 4) bringing novel text analysis methods to the field of demography. ## data the data used in this study is collected through the liss (longitudinal internet studies for the social sciences) panel administered by centerdata at tilburg university, the netherlands. the study is based on the second wave of survey social networks and fertility (in dutch: sociale relaties en kinderkeuzes onderzoek) within the liss panel in 2021, which was first fielded in 2018. the module's objective was to investigate fertility intentions and attitudes in relation to people's personal networks. for this round of our survey, 596 female participants were invited, and 464 women between the ages of 21 and 44 completed the questionnaire. the survey was conducted in dutch. the openended questions (henceforth: oeq) regarding fertility intentions are presented to respondents that are not currently pregnant (@xmath0. after removing 6 answers that were without information (e.g. "niets") or not in dutch, there were in total 427 responses available. the oeq was placed directly after a standard closed questions on fertility intention ("do you intend to have a/another child during the next three years?") from the generations & gender surveys (ggs) @xcite . respondents were presented with a text box, where they can input text answers. two versions of the oeq were tested: the answers contain 32 words on average. since answers to these two questions were similar on a suite of textual characteristics (e.g., sentence length, number of nouns), we did not differentiate between answers to these two questions in the subsequent analyses. ## evaluation each model was evaluated on three different metrics: topic coherence, topic diversity, and comparison to human-assigned labels. topic coherence measures how close the top n words (typically, @xmath1: if the words always co-occur in documents, they are considered "close" and the topic is considered coherent. it is calculated through non-negative point-wise mutual information (npmi, @xcite al. (2010) ), where w denotes a word: the calculation of topic coherence requires an external test corpus for calculating how frequently words in the topic occur together in real language usage blair et al. (2020). the external corpus was crawled from the viva forum, a dutch online discussion board mainly aimed at women. we prepared two corpora for cross validation: first the "child wish" corpus, which contains 436 threads and replies including the keyword "child wish foot_1 "; second the "pregnancy foot_2 " corpus, containing 5507 comments under the "pregnancy" board. coherence scores were calculated on both corpora. topic diversity measures how different the top ten words from all topics are; i.e. if topics share the same words. it is calculated through inversed rank-biased overlap (ρ; @xcite al. (2010) ), where the top ten words are compared. the score ranges from 0 to 1, representing topics that contain exact same words to totally different topics. the four authors of this paper intensively read on recent fertility trends and events in the netherlands. the first and fourth authors read through the dataset to develop qualitative insights and proposed themes of discussions from respondents; as we coded the responses iteratively, although exact labels were not yet given to each response, six themes were summarized. then, all authors interpreted lists of top words for each topic generated by models, and compared results with different number of topics k to develop the ideal k. eventually, we have together determined the optimal number of topics to be @xmath2. we ran grid search on each model for hyperparameter tuning under the same number of topics @xmath3; this did not apply to bertopic as it used hdbscan algorithm and did not require a pre-defined k. we use cow word embedding @xcite for scholar model, and we use robbert @xcite , a stateof-the-art dutch bert model for combinedtm and bertopic. ## results in this section, we first present the themes from qualitative insights. these are then compared to results from the four topic models. since human produced themes may correspond to more than one topic @xcite , we calculate how many themes were accounted for and present them together with other metrics. the themes and corresponding topics are presented in ## discussion demographers have long been calling for empirical evidence on fertility intention uncertainty and narratives @xcite . with the results from this study, we showed that neural topic models were able to provide insights similar to human judgments, thus providing a powerful tool for future demographic studies. our results also demonstrated the significant improvement of performance that neural topic models brought to text analysis on short survey data. the prior knowledge of language, incorporated by language models such as bert, enabled results of quality close to traditional qualitative analysis in social studies, while previously used models such as lda failed to so. this may have a direct applica-tion in processing online open surveys or interview data, and enabling qualitative analysis on a larger scale. the contrast between the ranking of scores in internal and external coherence revealed that the evaluation strategy on topic models may need to be reconsidered, especially in social science studies. although topics generated by scholar showed an extremely high internal coherence, a closer look showed that it is mostly due to some topics consisting of words that were almost exclusively from one document (response), dragging coherence of that topic up to almost 1 (i.e. words would always cooccur). this also explained its unusually high topic diversity (at 1, which entails no repeated words at all across topics), as the topics consist of only unique words from the one document. our results remind us that some metrics for topic models may be misleading on a smaller, shorter dataset, and choosing the right, field-relevant corpus is a key step in correctly evaluating topic coherence. moreover, using multiple criteria help us to avoid pitfalls and making more informed choices in selecting topic models for survey data.