Identifying Emerging Concepts in Large Corpora (2025)

Abstract: We introduce a new method to identify emerging concepts in large text corpora. By analyzing changes in the heatmaps of the underlying embedding space, we are able to detect these concepts with high accuracy shortly after they originate, in turn outperforming common alternatives. We further demonstrate the utility of our approach by analyzing speeches in the U.S. Senate from 1941 to 2015. Our results suggest that the minority party is more active in introducing new concepts into the Senate discourse. We also identify specific concepts that closely correlate with the Senators’ racial, ethnic, and gender identities. An implementation of our method is publicly available.

## introduction the identification of new ideas and concepts within large corpora is of core interest both in computational linguistics, the social sciences, and the humanities. for instance, @xcite discover new innovations in a corpus of scientific articles. they find that minorities often introduce novel contributions to the scientific discourse, but those innovations are disproportionately discounted. @xcite examine how new stereotypes towards ethnic and racial minorities evolved during the 19 th and 20 th centuries. and @xcite identify misinformation at its conception and track its spread and influence on public discourse. recent advances in natural language processing offer more robust alternatives. transformer-based models, such as bert @xcite and t5 @xcite , leverage contextual embeddings to encode semantic meaning beyond individual words, making them well-suited for tracking emerging concepts that lack stable lexical forms. however, most existing methodologies for text analysis are not specifically designed for identifying emergent concepts. instead, they often apply general-purpose techniques that do not account for the distinct temporal patterns associated with conceptual emergence. as a consequence, existing approaches often lack sensitivity, allowing for the accurate identification of emerging concepts only after these concepts have become well-represented in the underlying corpora. to the extent that methods exist that are specifically targeted at emerging concept identification, they tend to be supervised @xcite , thus requiring the investigator to know ex ante what new concepts to look for. they also tend to operate at the word-level, thus making it impossible to identify concepts that are not easily representable using a distinct unigram or bigram. but consistent with the semantic perspective, we understand concepts as abstract objects-propositions that can be expressed in a multitude of ways @xcite . while some concepts can be captured by individual words or phrases (e.g., "climate change"), we assume that many concepts are more complex and cannot be reduced to a single lexical unit (e.g., "negative sentiment towards the affordable care act as a form of governmental invasion" @xcite ). this in turn requires a methodological approach that goes beyond the word-level and is able to capture the broader semantic structures that define emerging concepts. in this paper, we introduce a novel methodology to identify emerging concepts in large text corpora that we also make publicly available. 1 intuitively, our methodology leverages heatmaps of the embedding space to identify those regions that are subject to sudden, long-lasting increases in density. because we benchmark the density increase against commonly observed, non-systematic changes, our method performs particularly well at identifying emerging concepts shortly after their inception, well before they become broadly represented in the underlying corpus. our method operates at the sentence level (or any other, larger textual unit), thus allowing for identification of complex concepts that cannot easily be captured with a distinct word or phrase. in several evaluations, we demonstrate the performance of our proposed methodology, and compare it to other approaches that have been employed to identify emerging concepts in text. in a last step, we illustrate the utility of our method for the social sciences and digital humanities by exploring the introduction of new concepts in the u.s. senate debates from 1941 (77 th congress) to 2015 (114 th congress). at a macro level, we find a consistent pattern showing that the minority party introduces new concepts and ideas with greater frequency than the majority. this finding lends support to claims by other scholars about strategically different behavior of the minority party in legislative bodies @xcite , including their way to converse @xcite . ## related work perhaps closest in spirit to our motivation is a recent study by @xcite . they identify novel ideas using word-level perplexity, such that sentences with unexpected word combinations are deemed to reflect novelty. they provide convincing evidence that their approach identifies novel ideas at a macro level. in their approach, syntactic idiosyncracies are receiving the same weight as unusual semantic occurrences. and since it operates at the word level, there are likely limits to the linguistic complexity with which new ideas can be identified. in contrast, our approach centers around textual embeddings, a literature with a long tradition. a number of early contributions interested in the identification of new concepts characterized the problem in terms of diachronic meaning shift detection. that is, a new concept would be identified by the emergence of a new word sense for a given word. notably, @xcite demonstrated that word meaning evolution follows predictable trajectories, influenced by frequency and semantic drift, highlighting the importance of tracking concept emergence over time. naturally, the associated methodologies are centered around the use of word embedding models. several studies have explored different approaches to detecting semantic change, including cosine distance between word embeddings @xcite @xcite , bayesian models for temporal word representations @xcite , and topic modeling techniques such as latent dirichlet allocation (lda) @xcite . other refinements incorporate contextual embeddings @xcite or focus on modeling semantic drift in historical corpora @xcite . surveys such as @xcite provide an overview of these methodologies and their relative strengths and limitations. despite their effectiveness in tracking known shifts, these methods face two key limitations. first, since they analyze shifts at the word-level, they are not well-suited to detect emerging concepts that cannot easily be associated with a single word or phrase @xcite . second, these methods are supervised in the sense that the investigator needs to predefine the list of words for which a new meaning may emerge. in that way, they are better suited to identify when a known semantic shift has occurred. however, they are not aimed at identifying previously unknown, emergent concepts. more recent approaches have relied on the detection of concepts by applying clustering algorithms to text embeddings @xcite @xcite . these unsupervised methods do not require predefined knowledge about the changing concepts, making them better-suited to detect unknown concepts in large corpora. however, clustering methods are insensitive to the temporal nature of the data generating process, thus preventing them from taking into account the dynamic features that accompany an emergent concept. in addition, the most commonly employed clustering methods like @xcite and kmeans variants @xcite suffer from parameter selection and difficulty in controlling sensitivity and specificity @xcite . for instance, the widely used hdbscan algorithm @xcite (see, e.g. grootendorst, 2022) is very sensitive to its core parameters, including the minimum number of points required to form a cluster, the minimum number of samples in a neighborhood for a point to be considered a core point, and the distance threshold for merging clusters. only most recently have researchers begun to adapt these approaches by examining the temporal or usage changes within clusters @xcite , but the current clustering methods lack the sensitivity to capture more nuanced topics. we will further evaluate this in section 4.1. our method was inspired by these clustering approaches and aims to improve upon them by using blob detection with laplacian of gaussian filtering @xcite on differences of heatmaps. this technique mitigates the complexity of parameter selection and enhances control over robustness and sensitivity. instead of clustering all data points across time periods together and analyzing data composition within each cluster, we track the development of concepts by detecting and examining regions of high, novel density in the embedding space. by focusing on differences between time periods, this approach removes noise and allows for more accurate and sensitive detection of emerging concepts. ## method this section outlines our method to detect emerging concepts. in doing so, we follow vicinanza et al. @xcite and define new linguistic concepts as those that emerge as separate and distinct from existing discourse and remain with some permanence. our process involves several key steps: (1) embedding sentences to capture their semantic features, (2) reducing the dimensionality of these embeddings and generating heatmaps to summarize the embedding space and its distribution, (3) detecting significant changes among the heatmaps with blob detection on the differences among heatmaps, and (4) tracking the progression of these changes and interpreting them as newly emerging concepts. each step is designed to ensure that the analysis is both comprehensive and efficient, and gives informative results for downstream analysis. figure 1 illustrates our pipeline, which is explained in more detail in the following subsections. ## evaluation we next turn to evaluating our algorithm, contrasting its performance to the popular alternatives introduced by @xcite and @xcite . a holistic evaluation of new concept detection is inherently difficult and prohibitively costly, for at least three reasons: first, there is little unanimity or consistency in defining the outer contours of a concept. two experts might disagree, for instance, on whether the pacific battle during world war 2 is a separate concept from the attack on pearl harbor, or whether one is a smaller concept inside the other. second-and relatedlyvalidating the existence of an emerging concept might require extensive domain expertise. prior papers have often validated their approaches using historical corpora (vicinanza et al., 2022; giulianelli et al., 2020), but it can be difficult for an untrained human evaluator to identify complex and nuanced concepts without sharp contours, such as those promoting economic mobility. 5 in effect, this means that the generation of human labels can be prohibitively costly, especially in contexts where domain expertise is required. third, creating a comprehensive list of all concepts in any given domain is infeasible due to the sheer number of conceptual developments. this makes it impossible for researchers to assess recall with any significant reliability. although it is not possible for us to remedy these shortcomings directly, we are taking a multipronged approach to alleviate concerns as much as possible. in doing so, we take the following steps: our first evaluation uses synthetic data. in the synthetic dataset, we are able to randomize the baseline corpus, introducing new concepts artificially in a controlled way. randomization ensures that the baseline corpus-in expectation-does not include any coherent concepts, thus allowing us to evaluate both precision and recall of our approach. that said, a synthetic corpus has the shortcoming that it does not represent documents that were created through a realistic data generating process, threatening external validity of the results. we thus complement the evaluation on a synthetic corpus with an assessment using real data, specifically the corpus of historical american english (coha) @xcite . as pointed out above, coha does not allow us to holistically evaluate recall because no dataset exists that contains the complete set of historical concepts. in addition, we cannot evaluate precision holistically because there are simply too many concepts to have each verified by domain experts. we thus limit our evaluation on the real data to the identification of a limited set of concepts that do not require extensive domain knowledge for evaluation: important world events. ## application: emerging concepts in the u.s. congress to illustrate the utility of our proposed pipeline in a real-world scenario, we employ it to analyze u.s. senate speeches derived from the u.s. congressional record (matthew gentzkow, 2018) from 1941 (77 th congress) to 2015 (114 th congress), which includes a total of 2,254,427 speeches foot_5 . our method uncovers emergent concepts and patterns that are not readily detectable using conventional approaches that predominantly focus on syntactic feature extraction. we analyze these concepts both at the macro level by party ideology and at the micro level by senator identity. as such, our findings contribute to a broader literature on viewpoint diversity at the intersection of nlp and politics @xcite @xcite . figure 6 provides summary statistics on ideological, gender, and racial/ethnic identity representation, both in terms of personnel and in terms of speech, during our period of observation. these show that, unsurprisingly, the proportion of speeches closely tracks representation in congress. interestingly, the results show that women senators initially only rarely spoke in the senate. indeed, meaningful content to the concepts of interest from 1970 to 1990, the proportion of such speeches was close to 0, although the share of women senators increased steadily over that time period. even today, women speak disproportionately less in the senate than men. we observe a similar trend for racial minority senators, which we define as senators with asian, black, hispanic, native american, or pacific islander identity. ## conclusion we have introduced a new, unsupervised methodology to identify emerging concepts in large text corpora. our approach is able to identify new concepts shortly after their inception, before they become deeply entrenched in the discourse. in doing so, we hope our efforts contribute to recent developments that leverage computational linguistics to support new discoveries, especially within the social sciences and digital humanities @xcite .