A Statistical Approach for Quantifying Group Difference in Topic Distributions Using Clinical Discourse Samples (2023)

Abstract: Topic distribution matrices created by topic models are typically used for document classification or as features in a separate machine learning algorithm. Existing methods for evaluating these topic distributions include metrics such as coherence and perplexity; however, there is a lack of statistically grounded evaluation tools. We present a statistical method for investigating group differences in the document-topic distribution vectors created by Latent Dirichlet Allocation (LDA) that uses Aitchison geometry to transform the vectors, multivariate analysis of variance (MANOVA) to compare sample means, and partial eta squared to calculate effect size. Using a corpus of dialogues between Autistic and Typically Developing (TD) children and trained examiners, we found that the topic distributions of Autistic children differed from those of TD children when responding to questions about social difficulties (p = .0083, partial eta squared = .19). Furthermore, the examiners’ topic distributions differed between the Autistic and TD groups when discussing emotions (p = .0035, partial eta squared = .20), social difficulties (p < .001, partial eta squared = .30), and friends (p = .0224, partial eta squared = .17). These results support the use of topic modeling in studying clinically relevant features of social communication such as topic maintenance.

## introduction throughout the course of a dialogue many different topics are traversed with varying frequencies, and many analytical tasks depend on the ability to meaningfully quantify or otherwise characterize these patterns. for example, a system designed to automatically summarize meetings might need to detect when a new topic has been introduced; in a clinical context, we might wish to characterize the topics discussed during a patient visit to facilitate some sort of downstream analysis involving clustering or classification. topic modeling techniques such as latent dirichlet allocation (lda; @xcite allow us to capture and quantify the topic distributions across a collection of language samples. typical methods for evaluating the resulting topic distributions use intrinsic metrics such as within-topic coherence; however, to our knowledge there remains a shortage of methods for statistically comparing the topic distributions produced by a model. the application of topic modeling methods in clinical research has become more common in recent years @xcite @xcite . while topic modeling approaches have advanced significantly over the last twenty years, evaluation methods have lagged behind (see @xcite for a recent survey of methods). current metrics tend to focus on intrinsically assessing model performance (via perplexity on held-out data) or on attempting to measure the quality of the topics that a model produces using metrics based on constructs such as human interpretability of the topics themselves (sometimes referred to as "coherence"). in a clinical research setting, however, the topic distributions produced by a model are themselves often meant for use in meaningfully quantifying differences between clinical populations. in such a scenario, usefully evaluating the quality of a topic model's "fit", or comparing that "fit" to that of another model (perhaps trained via a different algorithm, or with a different choice of hyperparameters) becomes a question of extrinsic evaluation, as intrinsic metrics such as perplexity or coherence are unlikely to be sufficient. additionally, in clinical research, topic models are typically one piece of a larger analytical puzzle, one which often depends on traditional hypothesisdriven inferential statistical approaches (rather than stand-alone evaluation or use, as is more typical with topic models in machine learning scenarios). in this paper, we outline a statistical approach to explore and quantify group differences in topic distributions captured by topic models and demonstrate its application using lda and two different corpora. first, we validate our method on the 20newsgroup corpus, a widely-used reference corpus for developing and evaluating topic modeling algorithms @xcite , by comparing topic distributions between groups of documents that we expect to be similar and groups that we expect to be different. second, we use our method on a corpus of language samples of autistic foot_0 and typically developing (td) children. based on previous clinical evidence, we expect the topic distribution vectors of autistic children to differ from those of the td children. our proposed method allows for a robust and statistically meaningful evaluation of the output of a topic model in both clinical and non-clinical contexts. ## statistical motivation lda is a unsupervised, generative probabilistic model that is used on a corpus of text documents to model each document as a finite mixture over k topics @xcite . each document is treated as a bag-of-words (i.e., order does not matter) and is represented as a set of words and their associated frequencies. given m documents and an integer k, lda produces a m × k document-topic matrix (θ). lda also produces a k × v topic-word matrix (β), where v is the total number of unique words across the entire corpus of documents. since we will not be using the topic-word matrix in this analysis, from this point forward, we will use the phrases "lda model" and "document-topic matrix" interchangeably. in the document-topic matrix, each row represents a single document and each column represents one topic. the elements (θ 1,1 , . . . , θ i,j , . . . , θ m,k ) are the estimated proportion of words in a document that were generated by a topic. from this matrix, each document can now be represented as a k-dimensional topic distribution vector. these lda-derived topic distribution vectors often serve as useful document representations for downstream analyses, such as a feature vectors for documentation classification or clustering. they are also commonly used as proxies for document content in more qualitative analyses of the composition of text corpora. to our knowledge, a statistical method for comparing topic distribution vectors between groups of documents has not yet been proposed. one reason for this is due to the numerical properties of the resulting topic distribution vectors (each component θ i is bounded between {0, 1} with the further constraint of k @xmath0, which render them unsuitable for use with many parametric statistical methods. this is an important limitation, because as previously mentioned, as the applications of topic modeling methods expand in clinical and behavioral research, the need for statis-tically based evaluation tools grows. we realized that since the components in a topic distribution vector are proportions and all sum to one, they meet the definition of "compositional" data as formalized by @xcite , who also proposed a family of statistical approaches for such data. compositional data are vectors of positive numbers that together represent parts of some whole: e.g., the demographic profile of a city or the mineral compositions of rocks. there are three linear transformations that can be performed on compositional data: additive logratio (alr), center logratio (clr), and isometric logratio (ilr) transformation. the ilr transformation was introduced by @xcite in an effort to broaden the range of statistical methods that can be applied to compositional data by mapping compositonal data into real space. this transformation maps a composition from its original sample space (the d-part simplex) to the d -1 euclidean space (ilr: s @xmath1. once the compositions are in r d-1 , we are able to use classical multivariate analysis tools such as multivariate analysis of variance (manova) to explore group differences @xcite . 2 manova is used to compare multivariate sample means and examines the effect of one discrete, independent variable on multiple continuous, dependent variables. for the analyses described in this paper, the independent variable is topic label when using the 20newsgroup corpus and diagnosis (asd, td) when using the clinical corpus. the dependent variables in both analyses are the various topic distribution probabilities in the documenttopic matrix created by lda: θ i,1 , θ i,2 , . . . , θ i,k-1 where @xmath2, 2 . . . , m . it is important to note that a different discrete variable can be used as the independent variable, as long as it separates the documents into groups (e.g., author if modeling a corpus of newspaper articles); if one wished to incorporate multiple independent variables, one could could instead use mancova. since we used a k of 20 in both of our analyses and one dimension is removed during the ilr transformation, there are a total of 19 dependent variables. in the case that we do find a significant group difference, the next step is to find out the magnitude of the effect. after manova, we can use partial eta-squared (η 2 ) to calculate effect size. partial η 2 tell us what proportion of variance of the linear combination of the topics can be explained by the independent variable @xcite . manova is a compelling choice for this analysis for several reasons. as detailed above, it enables us to statistically determine whether the topic distributions learned by our topic model are significantly associated with our other variables of interest (group membership, etc.) under a conventional hypothesis-testing framework. second, manova allows us to calculate interpretable measurements of effect size, which in turn facilitate comparison between different models (even if they are trained using different modeling algorithms). third, this framework enables us to incorporate additional covariates as independent variables (via upgrading to mancova), in a way that a more traditional classification-centric downstream task would not. lastly, manova is a well-characterized and wellestablished statistical method and as such has numerous useful extensions; for example, it can be combined with post-hoc roy-bargmann stepdown procedure @xcite which enables detailed statistical analysis of the relationship between individual topics (or combinations of topics) and our independent variable, thereby facilitating a far richer quantitative interpretation of our topic model's output than other methods. note, however, that this would be slightly complicated under our protocol due to our use of ilr, which results in the loss of a dimension into a new feature space that is decoupled from the original topics learned by the model (but which preserves important semantic properties of the original feature space). in this work, we explore only the first two points mentioned, leaving the rest for future work. ## corpora we demonstrate our approach on two separate corpora: a subset of the 20newsgroup corpus and a corpus of transcribed natural language samples of asd and td children. ## results lastly, the results of the statistical analyses performed on the examiner speech are reported in ## discussion the autistic children and td children had significantly different topic distributions for one of the four conversation analyzed: social difficulties and annoyance. we expected to observe a group differ-ence in all four of the conversation activities instead of only one. incorporating additional participantlevel information such as iq and age or examining other measures of conversational reciprocity such as the length and complexity of utterances may help shed some light as to why a group difference was only seen in one of the four activities analyzed. in addition, further investigation into sampling context differences between the conversation activities is needed before conclusions can be drawn. this finding illustrates the value of our proposed statistical approach, in that we have numerous ways we could incorporate these additional covariates into our analysis in quantitatively useful ways within the same statistical framework. the examiners' topic distributions differed significantly between the asd and td groups for two of the four activities: emotions and social difficulties and annoyance. this is surprising as our initial hypothesis was there would not be any significant group differences for the examiners' topic distributions. ados examiners are instructed to cover the same questions for each child, regardless of diagnosis, and are trained to a high standard of consistency and repeatability, as the assessment is meant for clinical use. since one goal of the conversation activities is to foster a dialogue, the examiner would likely avoid actions that could discourage the child from conversing and sharing their interests. it may be the case that the examiners are mirroring the topics introduced by the children during the activities and those topics are being picked up by the topic distributions created by lda. 6 this could be explored in the future by investigating pairwise group differences. ## conclusion in this paper we presented a novel application of existing statistical methods to evaluate the documenttopic distribution vectors created by topic models in order to investigate group differences. by treating the document-topic distribution vectors as compositional data @xcite , we are able to use the ilr transformation @xcite to map the vectors from their original sample sample, the d-part simplex, into the d -1 euclidean space (ilr: s @xmath4. once in r d-1 , we are able to use classical multivariate analysis tools such as manova @xcite . when applied to an lda model fitted to the 20newsgroups corpus, our method successfully identified that the topic distributions for documents from different categories (computer hardware vs. sports) and also documents from related subcategories (pc hardware vs. macintosh hardware; baseball vs. hockey) were significantly different. the effect size, measured with partial η 2 , also varied 6 an anonymous reviewer brought to our attention that interviewers have been found to adjust their conversational patterns when speaking to patients with other cognitive conditions, such as alzheimer's disease @xcite . across these comparisons, with the effect size being the largest when comparing computer hardware vs. sports and smallest when comparing macintosh vs. pc hardware. furthermore, our method did not find that topic distributions are significantly different when comparing groups of documents from the same category. we also demonstrate the application of this method using lda and a corpus of child-examiner dialogues of autistic and td children, where prior clinical research gave us reason to expect to find group differences. we found that the topic distributions of autistic and td children were significantly different during one of the four ados conversation activities examined. this result aligns with prior clinical research that autistic children often have difficulties with topic maintenance in a conversational context. interestingly, we also found that examiners' topic distributions were significantly different whether they were conversing with an autistic child or a td child for two of the four ados conversation activities examined. this may indicate that although the examiners are trained to ask the same set of questions irrespective of diagnosis status, tangential topics introduced by the child during the conversation may be mirrored by the examiner and thus are reflected in the associated topic distributions. there are a few points about the statistical approach outlined in this paper that should be highlighted. although we demonstrate this method using the document-topic distribution matrix created by lda, this method can be extended to any topic modeling algorithm that outputs a topic distribution that can be treated as a composition. we decided to use lda here as it is a well-established technique that has been extended and built upon many times over since it was first introduced in 2003. another important point to highlight is that, although not shown in here, this analysis has the potential to be extended further with a post-hoc roy-bargmann step down procedure to explore how much each topic (or combination of topics) contributes to the significant effect of the independent variable @xcite . however, as previously mentioned, the loss of a dimension during the ilr transformation would need to be addressed first. overall, the statistical approach presented in this paper represents a very promising direction for methods of making topic models more interpretable in a quantitative way, beyond human inspection of topics. in the future we would like to extend this specific analysis to include additional participant-level, independent variables (e.g., age, sex, iq) by using multivariate analysis of covariance (mancova). since social communication skill level can vary throughout the asd spectrum @xcite , we would also like to look at differences within the asd group by exploring within group variance metrics. we would also like to explore the use of other methods of topic modeling, beyond lda, for this application. as the application of topic modeling methods continues to grow into areas such as clinical and behavioral research, so does the need for statistically based methods for evaluation and comparison. our hope is that the statistical approach described in this paper contributes to bridging that gap by focusing on improving evaluation metrics for existing topic modeling methods.