HOTTER : H ierarchical Optimal Topic Transport with Explanatory Context Representations (2021)

Abstract: Natural language processing (NLP) is often the backbone of today’s systems for user interactions, information retrieval and others. Many of such NLP applications rely on specialized learned representations (e.g. neural word embeddings, topic models) that improve the ability to reason about the relationships between documents of a corpus. Paired with the progress in learned representations, the similarity metrics used to compare representations of documents are also evolving, with numerous proposals differing in computation time or interpretability. In this paper we propose an extension to a specific emerging hybrid document distance metric which combines topic models and word embeddings: the Hierarchical Optimal Topic Transport (HOTT). In specific, we extend HOTT by using context-enhanced word representations. We provide a validation of our approach on public datasets, using the language model BERT for a document categorization task. Results indicate competitive performance of the extended HOTT metric. We furthermore apply the HOTT metric and its extension to support educational media research, with a retrieval task of matching topics in German curricula to educational textbooks passages, along with offering an auxiliary explanatory document representing the dominant topic of the retrieved document. In a user study, our explanation method is preferred over regular topic keywords.

## introduction topic models have been employed for more than a decade to capture latent semantics that help to organize documents in a dataset. latent dirichlet allocation (lda) by @xcite illustrates this approach. on the other hand, word embedding models have been proposed in recent times, for uses in many natural language processsing (nlp) tasks. word embeddings are able to map phrases and documents to a dense, high-dimensional space, where (according to how the embedding model was trained) semantic similarity or analogy relationships between expressions are facilitated. in recent years, both approaches -topic models and word embeddings -are used in combination. an emerging metric for such a scenario is the hierarchical optimal topic transport (hott) by @xcite , which computes the distance between top n topic words using their word embedding representation, weighted by the document-topic distribution. as approaches for creating word embeddings evolve, newer contextual variants emerge. hence, a reasonable first research question could be to consider the impact on hott; does hott benefit from contextual word embeddings? in this paper we perform targeted experiments regarding this first research question: we select the bert language model @xcite and try different variants of applying its contextual word embeddings to extend the hott metric. a particular advantage of using hott over current solutions relying solely on the word mover's distance (wmd) is a better interpretability of the document distances, since the metric relies on the known top n words of each topic for the respective document. supporting a better interpretability is a demand for practical applications in many domains @xcite . in this context, it is natural to ask how truly interpretable are those top n topic words to a domain expert? while the answer to this question depends highly on the quality of the topic model and parameter choices, we instead develop an alternative, simpler method for explaining the results by hott and then compare this method to the interpretability of the top n topic words. to this end, we employ the hott metric in a retrieval setting. particularly, we use the dominant topic keywords of the retrieved document for extracting a representative auxiliary document from the corpus, containing the respective topic word and scoring highest on the particular topic that the word belongs to. this in turn suggests for our study a second research question: does an auxiliary document that is close to the dominant topic keyword for retrieving a document offer a better explanation to a humanist than the top n topic keywords? we carry out experiments for the second research question on educational media research data, where the aim is to match themes in a teaching curriculum to the parts of a textbook corpus covering those themes. to summarize, the overall aim of this work is to adapt the hierarchical optimal topic transport to explanatory context representations (hotter). first, we investigate whether using contextual word embeddings offers a benefit for hott performance, and second, we examine the interpretability of hotter results. our core contributions can be stated as follows: the remainder of this work is structured as follows: section 2 collects related work about combining word embeddings with topic models, section 3 contains foundations required to understand our contribution, covering document distances in the word embedding space, hierarchical optimal topic transport and contextual word embeddings. section 4 describes our proposed hotter approach in detail. section 5 includes our experimental results and the corresponding discussion. section 6 concludes our findings. ## related work in this section, we briefly describe the most important related work about combining the lda topic model with word embeddings. topic models such as lda are popular for clustering a document collection. they learn a topic distribution for each document in a corpus and infer a word distribution for each topic. viewing the top n words of a topic can lead to insights about the themes the topic captures. however, this is not the only application of a topic model. the probabilistic nature makes it easy to interpret since the topic distribution of a document and the word distribution of each topic, respectively, sum up to one. this property makes them suited for featurizing and tagging documents for both, end user applications and further processing. a drawback in standard lda implementations is connected to the text representation they use. often a simple bag-of-words approach is chosen, leading to positional information being lost and the resulting topics carrying a notion of "relatedness" between words instead of semantic similarity @xcite . while there are n-gram topic model implementations such as the work by @xcite , this is not employed frequently due to model sparsity. therefore, the traditional unigram lda approach is still applied and combined with other approaches, such that they can complement each other. word embeddings by @xcite are nowadays a common choice for experimentation with other text representations due to their disruptive performance on many natural language processing tasks, and their capability of capturing term analogies and semantic similarity. in this approach word vectors are trained by maximizing the average log probability of the next word, which is different from topic model probabilities that are normalized to one. for this reason, standard word embedding values cannot be interpreted as probabilities, but we can assume that words with similar vectors have a similar meaning. another difference is that word embeddings are usually pre-trained on substantially larger corpora than topic models and then optionally fine-tuned on domain-specific text. taking those aspects into account, word embeddings and topic models have the potential to enrich each other. the inclined reader may refer to a more in-depth discussion on differences between topic models and word embeddings by @xcite or @xcite which we omit due to space restrictions. there are numerous works which combine topic models and word embeddings, resulting in two main groups of approaches @xcite : those using a topic model architecture with features from word embeddings, as opposed to those using the neural network architecture to obtain word embeddings and topic representation during training, such as lda2vec @xcite or twe @xcite . the basis for our work -the hott meta distance -belongs to the former group, hence we focus on related research in this regard. the vec2topic approach extracts word embeddings and combines them with a topic model that has been trained with k-means clustering, while using agglomerative clustering on the word vectors to score the topic based on the keyword similarity and importance (depth and degree) @xcite . @xcite use word embeddings to improve topic models with gibbs sampling to exchange top n topic words from the topic model with more salient terms. the gpu-dmm method is specifically developed for short text, since short texts rarely contain co-occurrences of semantically similar words @xcite . another approach incorporating topic correlation has been proposed by @xcite . overall, the numerous works in this field suggest a complementary nature of word embeddings capturing local semantic similarity, compared to topic models which generate a notion of semantic relatedness. to the best of our knowledge, there is no work on enhancing topic models with contextual word embeddings yet, which is one key contribution. ## background we proceed with the fundamental concepts required for understanding our approach. first, we cover the word mover's distance (wmd), second, we explain the hott meta distance and afterwards, we describe contextualized word embeddings. ## hotter in this section, we present the hotter approach by first incorporating contextual word embeddings into the hott document distance metric. furthermore, we explore retrieving representative sentences based on the obtained contextual embeddings for the top n topic words. ## conclusion in this paper we investigated two research questions for the recently proposed meta-distance hott, which computes optimal transport between documents using topic models and word embeddings. we showed that enhancing hott by contextual word embeddings from the bert model is competitive. our experiments on public datasets indicate that further pre-training of the language model offers an advantage over the original static hott variant. leaving out further pre-training shows static word embeddings to perform better on the explanation component which we developed for the second research question in a retrieval setting on educational media data. therefore, adapting contextual word embeddings to their domain via further pre-training may make a difference. overall, the explanations offered by our hott variants are more interpretable than dominant topic keywords. given those findings, we may improve the existing method by enforcing crisp topics. although realworld corpora potentially include multiple versions of the same document, it may be worthwhile to employ document consolidation to different document versions in order to obtain coherent topics. finding subtle differences between several textbook issues can be treated as a separate task in order to reduce language artifacts in the corpus.