WMD ecompose: A Framework for Leveraging the Interpretable Properties of Word Mover’s Distance in Sociocultural Analysis (2021)

Abstract: Despite the increasing popularity of NLP in the humanities and social sciences, advances in model performance and complexity have been accompanied by concerns about interpretability and explanatory power for sociocultural analysis. One popular model that takes a middle road is Word Mover’s Distance (WMD). Ostensibly adapted for its interpretability, WMD has nonetheless been used and further developed in ways which frequently discard its most interpretable aspect: namely, the word-level distances required for translating a set of words into another set of words. To address this apparent gap, we introduce WMDecompose: a model and Python library that 1) decomposes document-level distances into their constituent word-level distances, and 2) subsequently clusters words to induce thematic elements, such that useful lexical information is retained and summarized for analysis. To illustrate its potential in a social scientific context, we apply it to a longitudinal social media corpus to explore the interrelationship between conspiracy theories and conservative American discourses. Finally, because of the full WMD model’s high time-complexity, we additionally suggest a method of sampling document pairs from large datasets in a reproducible way, with tight bounds that prevent extrapolation of unreliable results due to poor sampling practices.

## introduction: the paradox of word mover's distance the present paper introduces wmdecompose, an iteration of the word mover's distance (wmd) @xcite model commonly used for determining the semantic distances between pairs of documents. leveraging word vectors from models such as word2vec @xcite , fasttext @xcite and glove @xcite , wmd was presented as a method that was not only hyper-parameter free and thus easy to use, but also highly interpretable @xcite . arguably, this still makes the model a viable alternative for document similarity tasks, despite the recent and rapid rise of contextual and rich embeddings from transformer-type models @xcite such as bert @xcite . however, wmd is computationally expensive, with the full model running at cubic time complexity @xcite . this may explain why, despite its initial pitch as an interpretable alternative, wmd has mainly been further developed and applied in ways that focus on decreasing the model's high runtime, while ignoring or undermining the inherent interpretability of the model @xcite . to our knowledge, no current wmd implementation provides an out-of-the-box means of retaining word-level information, despite its utility to many research agendas. to confront this paradoxical situation, we introduce a set of methods and python code for retaining the individual word distances that make wmd interpretable while simultaneously suggesting a simple trick for efficiently estimating the distance between large sets of documents. specifically, we propose using implementation of the "relaxed" wmd @xcite with linear time complexity @xcite to first estimate the distances between a full set of documents, then using optimal pairing with the gale-shapley algorithm @xcite on this estimate to find the pairs between two sets of documents that minimize the average pairwise distance. next, full wmd is calculated between each pair, while progressively adding the contributions of individual words to the overall distance between document sets. words are, in turn, grouped using k-means clustering and vector dimensionality reduction to decompose the distances not only by word, but by thematic cluster. each cluster is defined by its constituent words, and hence highly interpretable. having introduced wmdecompose, we demonstrate its utility in a social scientific context by exploring interrelated trends over time between two social media corpora: r/conspiracy and r/the_donald, the primary reddit communities for self-identified conspiracy theorists and donald trump supporters, respectively. while this presents only a cursory and exploratory engagement with these thematically complex data, we hope that it demonstrates the analytic potential of wmdecompose for social science and humanities research. finally, we also provide a complementary analysis using a well-known yelp review dataset, to act as a sanity check on the validity of our method. this analysis can be found in appendix b. ## conclusion we have introduced a novel iteration of wmd, one which departs from the wmd variations which, despite their high performance on a variety of tasks, do away with one of the key contributions of the original model: the inherent and decomposable relationship between document-level semantic distances and the lexical semantics that underlie them. in so doing, we heed a recent, but hopefully longlived, call for ml and particularly nlp researchers to prioritize models which are interpretable, providing explanatory value at the level of social meaning. we hope that our python package might aid future researchers similarly interested in interpretable computational approaches. furthermore, through the gale-shapley algorithm, we propose an approach for combining interpretability with environmental sustainability. down the line, this framework could also be expanded to support dynamic embeddings from models such as bert, which due to their technical opacity have yet to be widely adapted in sociocultural analysis. our vignette presented how one might use wmdecompose in a comparative and exploratory context. that is, we imagine this as somewhat of an inductive starting point-in this case, for an analysis of the relationship between trumpist and conspiracy theorist online communities-revealing trends related to one's particular research question that can be pursued with more focused and rigorous analysis, be it qualitative, discourse analytic, or statistical. however, because of the rich, word-level quantitative measures wmdecompose provides, future research might attempt to employ it in, for example, causal models seeking to estimate the effect of a treatment condition on particular lexical features, time series analyses, and so on.