BERT rend: Neural Topic Modeling for Emerging Trends Detection (2024)

Abstract: enter abstract here

## introduction the concept of weak signals, introduced by @xcite , refers to early indicators of emerging trends that can have significant implications across various domains. these include events like shifts in public opinion in social trends, early disruptive technologies in innovation, changes in activist groups and public sentiment in politics, and potential disease outbreaks in healthcare. monitoring and analyzing weak signals offers valuable insights for organizations, researchers, and decisionmakers, aiding in informed decision-making. key data sources for identifying these trends include large text corpora such as news, social media, research and technology journals or reports. the challenges are: distinguishing meaningful weak signals from irrelevant noise, dealing with context ambiguity, and tracking the extended period over which weak signals may gain significance. with advances in nlp and ai, researchers have developed various techniques to detect weak signals across different fields,including statisticsbased methods, graph theory, machine learning, semantic-based approaches, and expert knowledge. however, most solutions fall short in fully addressing the challenge of detecting emerging trends @xcite , either by relying solely on keyword-based analysis, which misses contextual nuances, or by being static and unable to dynamically track evolving weak signals. in this work, we introduce bertrend, a novel framework for detecting and monitoring emerging trends and weak signals in large, evolving text corpora. bertrend leverages neural topic modeling, specifically bertopic, in an online learning setting to identify and track topic evolution over time. its key contribution lies in dynamically classifying topics as noise, weak signals, or strong signals based on their popularity trends. the proposed metric quantifies topic popularity over time by considering both the number of documents within the topic and its update frequency, incorporating an exponentially growing decay if no updates occur for an extended period. by combining neural topic modeling with a dynamic popularity metric and adaptive classification thresholds, bertrend provides a comprehensive solution for detecting and monitoring emerging trends in large-scale, evolving text corpora. we discuss the qualitative results on two comprehensive datasets, including the overall evolution of trends and specific case studies. combined with large language models (llms), the method an efficient way of interpreting the detected trends of events through various dimensions indicating how they evolve over time. 1 2 background among past works about weak signals detection, many are keyword-based. thus, portfolio maps, pioneered by @xcite , involves constructing keyword emergence maps (kem) and keyword issue maps (kim) based on two key metrics: degree of visibility (dov) that quantifies the frequency of a keyword within a document set; and degree of diffusion (dod) that measures the document frequency of each keyword. weak signals are identified as keywords with low frequency but high growth potential. numerous studies, such as @xcite , @xcite , @xcite , @xcite , @xcite , @xcite , have extended and refined this approach with multi-word analysis, signal transformation analysis, and domain-specific applications. however kems and kims present two major drawbacks: by focusing on keywords only, they can miss the context surrounding a weak signal ; and the output is a single snapshot, which does not gives clear clues of evolution over time. topic modeling has emerged as a promising approach for weak signal detection, particularly in large textual datasets. unlike general topic evolution or drift analysis, which focus on tracking changes in established topics over time, our task aims to identify early indicators of emerging trends. it emphasizes the temporal behavior and growth of small, nascent topics rather than specific content changes within established ones. thus, @xcite and @xcite apply text mining and latent dirichlet allocation (lda) @xcite , to identify future signals in the domain of land administration and policy research databases. @xcite integrates lda and word2vec to detect weak signals in weakly structured data. el @xcite introduce furthermore two functions for deep filtering: weakness, which measures the significance, similarity, and evolution of topics using coherence, closeness centrality, and autocorrelation metrics; and potential warning, which further filters the terms of the previously filtered topics to identify potential weak signals. while traditional topic modeling methods like lda have been useful for weak signal detection, they have notable limitations: it heavily relies on pre-set topic numbers and fails to benefit from the sophisticated, contextual embeddings provided by modern pre-trained models, resulting in less nu-anced analysis. additionally, it operates on a static basis, overlooking the crucial temporal dynamics of weak signals. rollinglda @xcite uses a rolling window for the identification of gradual topic shifts comparing topic distributions across consecutive windows, rollinglda can detect changes in the prominence of topics over time. the fixed number of topics is a drawback. it is rather used for long-term evolution monitoring rather than detecting weak signals; interpretability of shifts is limited to keyword comparison. in contrast, our approach leverages dynamic, high-quality contextual embeddings from pretrained models. our embedding-based technique provides a richer, more adaptive analysis that does not require preset topic counts. this shift from static, keyword-based methods to dynamic, embedding-based analysis allows for a more granular and accurate tracking of the evolution and significance of weak signals over time. ## bertrend in this section, we describe bertrend (figure 1 ), a method for identifying and tracking weak signals in large, evolving text corpora. it focuses on identifying emerging signals at a given moment, rather than tracking long-term topic evolution. it leverages the power of bertopic @xcite , a state-of-the-art topic model, and wraps it in an online learning framework. in this setting, new data arrives on a regular basis, allowing bertrend to capture the dynamic evolution of topics over time. the method employs a set of metrics to characterize these topics as noise, weak signals, or strong signals based on their popularity trends. by combining the strengths of neural topic modeling with a dynamic, incremental learning approach, bertrend enables the real-time monitoring and analysis of emerging trends and weak signals in vast, continuously growing text datasets. bertopic leverages pre-trained large embedding models to generate high-quality contextual embeddings of documents, enabling the discovery of meaningful and coherent topics. it utilizes @xcite , a hierarchical density-based clustering algorithm, which is robust to outliers and does not require the number of topics to be specified in advance, allowing the model to automatically determine the optimal number of topics based on the inherent structure of the data. ## results quantitative results about weak signal analysis are very challenging to obtain due to the lack of established metrics and methodology as detailed in section 9.3. therefore, as in many past works in this research area (e.g. @xcite , we focus on a qualitative analysis, including retrospective analysis of known outcomes, to highlight its effectiveness and potential applications. ## interpretation of trends with llms topic modeling methods often output topics as sets of keywords, which can be difficult to interpret and may not fully capture the semantic meaning of the topic @xcite . llms can be leveraged to enhance the interpretation of signals detected by bertrend and of their evolution over time. although this field of topic analysis through llms is new, it is quite promising @xcite . in this work, we go several steps further by using llms not only for having human-readable descriptions of topics, but also useful insights about their evolution between two timestamps, such a summary of the key developments of the event signal since previous timestamp, as well as novelty about the signal w.r.t. previous time period. in addition, we use the llm to obtain an in-depth analysis of the signal, including: (1) impact, i.e. potential effects of this signal on various sectors, industries, and societal aspects, with both short-term and longterm implications; (2) evolution scenarios -both optimistic and pessimistic scenarios; (3) potential interactions /conflicts with other current trends; (4) drivers and inhibitors (factors/barriers related to the development of the signal. the associated prompt templates are provided in section a.2. in the example of figure 5 , we use the gpt-4o model 2 with a temperature of 0.1 to generate insightful summaries and highlight new information at each timestamp for a weak signal related to the new bluetongue viral disease (catarrhal fever) affecting ruminants that appeared in france in july 2024. this example was selected for its recency to ensure it lies beyond the llm's training data, minimizing the risk of analysis bias from the model's as an elite strategic foresight analyst, i've evaluated the potential signal of the bluetongue virus outbreak in france. here's a comprehensive analysis of its impact and evolution: ## conclusion in this paper, we introduced bertrend, a novel framework for detecting and monitoring weak signals in large, evolving text corpora. bertrend models the trends of topics over time and classifies them as weak signals, strong signals, or noise based on their popularity metric. the classification is performed using empirically chosen thresholds based on the distribution of topic popularities over a sliding window. the other contributions of this work include: (1) an extensive evaluation on two real-world datasets that demonstrate the effectiveness of our approach; (2) proposals to leverage llms to enhance the interpretation of topic evolution. we are currently exploring llm-generated evolving knowledge graphs as a structured method for interpreting signals. these graphs monitor topic evolution by tracking the appearance and disappearance of entities and relationships. future work will involve exploring new datasets, integrating live data, and developing metrics to compare weak signal detection methods.