T ext E ssence: A Tool for Interactive Analysis of Semantic Shifts Between Corpora (2021)

Abstract: Embeddings of words and concepts capture syntactic and semantic regularities of language; however, they have seen limited use as tools to study characteristics of different corpora and how they relate to one another. We introduce TextEssence, an interactive system designed to enable comparative analysis of corpora using embeddings. TextEssence includes visual, neighbor-based, and similarity-based modes of embedding analysis in a lightweight, web-based interface. We further propose a new measure of embedding confidence based on nearest neighborhood overlap, to assist in identifying high-quality embeddings for corpus analysis. A case study on COVID-19 scientific literature illustrates the utility of the system. TextEssence can be found at https://textessence.github.io .

## introduction distributional representations of language, such as word and concept embeddings, provide powerful input features for nlp models in part because of their correlation with syntactic and semantic regularities in language use @xcite . however, the use of embeddings as a lens to investigate those regularities, and what they reveal about different text corpora, has been fairly limited. prior work using embeddings to study language shifts, such as the use of diachronic embeddings to measure semantic change in specific words over time @xcite , has focused primarily on quantitative measurement of change, rather than an interactive exploration of its qualitative aspects. on the other hand, prior work on interactive analysis of text collections has focused on analyzing individual corpora, rather than facilitating inter-corpus analysis @xcite @xcite . we introduce textessence, a novel tool that combines the strengths of these prior lines of research by enabling interactive comparative analysis of different text corpora. textessence provides a multiview web interface for users to explore the properties of and differences between multiple text corpora, all leveraging the statistical correlations captured by distributional embeddings. textessence can be used both for categorical analysis (i.e., comparing text of different genres or provenance) and diachronic analysis (i.e., investigating the change in a particular type of text over time). our paper makes the following contributions: the remainder of the paper is organized as follows. §2 lays out the conceptual background behind textessence and its utility as a corpus analysis tool. in §3 and §4, we describe the nearestneighbor analysis and user interface built into textessence. §5 describes our case study on scientific literature related to covid-19, and §6 highlights key directions for future research. ## background computational analysis of text corpora can act as a lens into the social and cultural context in which those corpora were produced @xcite . diachronic word embeddings have been shown to reflect important context behind the corpora they are trained on, such as cultural shifts @xcite @xcite , world events @xcite , and changes in scientific and professional practice @xcite . however, these analyses have proceeded independently of work on interactive tools for exploring embeddings, which are typically limited to visual projections @xcite . textessence combines these directions into a single general-purpose tool for interactively studying differences between any set of corpora, whether categorical or diachronic. ## identifying stable embeddings for analysis while embeddings are a well-established means of capturing syntax and semantics from natural language text @xcite , the problem of comparing multiple sets of embeddings remains an active area of research. the typical approach is to consider the nearest neighbors of specific points, consistent with the "similar items have similar representations" intuition of embeddings. this method also avoids the conceptual difficulties and low replicability of comparing embedding spaces numerically (e.g. by cosine distances) @xcite . however, even nearest neighborhoods are often unstable, and vary dramatically across runs of the same embedding algorithm on the same corpus @xcite . in a setting such as our case study, the relatively small sub-corpora we use (typically less than 100 million tokens each) exacerbate this instability. therefore, to quantify variation across embedding replicates and identify informative concepts, we introduce a measure of embedding confidence. 2 we define embedding confidence as the mean overlap between the top k nearest neighbors of a nearest neighbors 3x replicates dry cough sneezing fever "rhinitis" neighbors confidences rhinitis fever 0.56 0.7 ... overlap analysis 70076002 rhinitis dry cough sneezing fever "rhinitis" neighbors dry cough fever nausea "rhinitis" neighbors dry cough sneezing fever "rhinitis" neighbors shivering nausea dry cough "rhinitis" neighbors 3 2 ( ) replicate a replicate b average 0.56 0.33 0.66 dry cough fever nausea "rhinitis" neighbors shivering nausea dry cough "rhinitis" neighbors 0.66 given item between multiple embedding replicates. formally, let e 1 . . . e m be m embedding replicates trained on a given corpus, and let knn i (c) be the set of k nearest neighbors by cosine similarity of concept c in replicate e i . then, the embedding confidence ec@k is defined as: an example of this calculation is illustrated in figure 1 . we can then define the set of high-confidence concepts for the given corpus as the set of all concepts with an embedding confidence above a given threshold. a higher threshold will restrict to highlystable concepts only, but exclude the majority of embeddings. we recommend an initial threshold of 0.5, which can be configured based on observed quality of the filtered embeddings. ## the textessence interface the workflow for using textessence to compare different corpora is illustrated in figure 2 . given the set of corpora to compare, the user (1) trains embedding replicates on each corpus; (2) identifies the high-confidence set of embeddings for each corpus; and (3) provides these as input to textessence. textessence then offers three modalities for interactively exploring their learned representations: (1) browse, an interactive visualization of the embedding space; (2) inspect, a detailed comparison of a given concept's neighbor sets across corpora; and ## case study: diachronic change in cord-19 the scale of global covid-19-related research has led to an unprecedented rate of new scientific findings, including developing understanding of the complex relationships between drugs, symptoms, comorbidities, and health outcomes for covid-19 patients. we used textessence to study how the contexts of medical concepts in covid-19-related scientific literature have changed over time. ## discussion our case study on scientific literature related to covid-19 demonstrates that textessence can be used to study diachronic shifts in usage of domain concepts. we highlight three directions for future work using textessence: mining for new shifts and associations in changing literature ( §6.1); applications between comparative analysis of corpora ( §6.2); and further investigation of embedding confidence as a tool for analysis ( §6.3). ## conclusion textessence is an interactive tool for comparative analysis of word and concept embeddings. our implementation and experimental code is available at https://github.com/drgriffis/ text-essence , and the database derived from our cord-19 analysis is available at https:// doi.org/10.5281/zenodo.4432958 . a screencast of textessence in action is available at https://youtu.be/1xeefsmwl0k . all associated resources for textessence may be found at https://textessence.github.io .