Monitoring geometrical properties of word embeddings for detecting the emergence of new topics. (2021)

Abstract: Slow emerging topic detection is a task between event detection, where we aggregate behaviors of different words on short period of time, and language evolution, where we monitor their long term evolution. In this work, we tackle the problem of early detection of slowly emerging new topics. To this end, we gather evidence of weak signals at the word level. We propose to monitor the behavior of words representation in an embedding space and use one of its geometrical properties to characterize the emergence of topics. As evaluation is typically hard for this kind of task, we present a framework for quantitative evaluation and show positive results that outperform state-of-the-art methods. Our method is evaluated on two public datasets of press and scientific articles.

## introduction for a company receiving hundreds of thousands of client feedbacks per month, it is crucial to analyze the dynamics of the textual content in an efficient way. while it is common to detect events and sudden bursts, slowly emerging new topics are often detected too late. early detection of new topics could lead a company to better understand their clients' feedback, to detect implicit problems in their infrastructure that can cause problems for certain types of clients and then anticipate marketing or communication responses. we consider that detecting emerging topics is a task close to the fields of event detection and linguistic change detection. the former focuses on detecting groups of words (e.g., topics) that are evolving fast in the form of a burst. the latter focuses on analyzing single word meanings evolving slowly. in this work, we are interested in detecting lexical fields that are evolving slowly over time (i.e., topics becoming dominant). in this work, we represent our textual data in the form of words in an embedding space and observe their evolution through time. we notice that single words evolve differently in high dimensional space according to the type of temporal dynamic they are linked to. instead of quantifying change of meaning and polysemy, we analyze changes through the scope of events and slow emergence of topics. during our observations, we notice that a positive correlation is linked with event-like topics while a negative correlation is a sign of emergence (see figure 1 ). we develop a method to detect words associated with these topics as early as possible. additionally, we develop a framework to artificially introduce emerging topics into our data in order to build a gold standard for detection. after reviewing the literature on the subject (section 2), we present a framework to evaluate our task by simulating the dynamic of novelty in a textual dataset (section 3). we develop a system based on our intuition that there is a specific correlation between word frequencies and movement in an embedding space (section 4). then, we explain how we got this intuition and illustrate it with two datasets (section 5). finally, we show that our method is the best in terms of qualitative and quantitative results with respect to state-of-the-art baselines of the literature (section 6). ## related works events are discussed in news stories every day as they describe important change in the world. event detection in news stories has long been addressed in the topic detection and tracking (tdt) program @xcite . several approaches have been studied to solve this task @xcite @xcite @xcite @xcite @xcite . on a longer time frame, textual data has also been studied through the scope of language evolution. change of meaning in words are the consequences of the appearance of new concepts, of the emergence of new technologies that change our way of life and then, our way of talking. topic evolution, which represents how words are mixed together, has been studied with the help of topic modeling algorithms like latent dirichlet allocation (lda) @xcite and its extension in @xcite @xcite . recently, the emergence of embedding representations, that represent word meanings, in textual data like word2vec @xcite has boosted the field of studies around language evolution. works like @xcite @xcite have analyzed the structure of vector representation of words in order to illustrate their change of meaning through a long period of time. in @xcite and @xcite , they quantify the correlation between movement in an embedding space and frequency of a word by viewing it through the scope of semantic change and polysemy. finally, contextual language models such as bert @xcite and @xcite have considerably improved the field of nlp. while it is known that this type of models slightly improve results in the task of semantic changes, they are also very time and resources consuming when we are monitoring an entire vocabulary @xcite . in the literature, some works as @xcite , @xcite and @xcite focus on detecting emerging topics with applications on twitter. a work like @xcite try to predict the future distribution of words using tf-idf scores in order to predict emerging topics. events are characterized by high and sudden bursts in the data. their appearance is easily detectable on a micro time scale. linguistic changes are slow, taking several years to change and can be illustrated on a macro time scale. in this work, we focus on the task of detecting as soon as possible weak signals in word movements associated with the slow emergence of a new topic. ## novelty definition the underlying task to resolve in the field of novelty detection is ill-posed. there is no clear consensus on the definition of "novelty" and no existence of a general framework for evaluation @xcite . in common sense, the term novelty refers to something that has not been observed before, therefore a single point in the data. in the literature, novelty is often linked to a signal @xcite and corresponds to an unexpected evolution of it. the work of @xcite presents a review of novelty detection in various fields and makes the distinction between novelty detection and anomaly or outlier detection. anomaly and outlier are single-point observations in a dataset while novelty (especially emerging novelty) corresponds to a series of small anomalies that leads to a brand new cluster or topic. in this work we define novelty as follows: novelty is observed when an underlying significative change in the distribution of the data is detected. textual data can be modeled as a word, topic or document. we develop a method for detecting, as early as possible, a slow emerging new topic by observing changes in word meaning. at the beginning of our observation, this topic is almost non-existent, that is, mixed up with noise. over time, this topic grows slowly and finally become a major topic in our data. evaluation for novelty detection is challenging: no annotated dataset for this task exists in the literature and quantitative evaluation is essential to support the domain. with that idea in mind, we propose to artificially insert novelty in a dataset in order to simulate the emergence of a new topic. datasets with annotated categories associated with each document are common in the nlp literature. we select one category in our dataset (for example: category about basketball or theater in the new york times) and re-order each of its documents with regards to time. this category now acts as a (controlled) emerging topic. the rate at which we introduce new documents into the dataset is defined as a logistic function, which is given by: we set α = 1 in order to have a centered emergent signal. the function is monotonically increasing from 0 to k. it allows to describe novelty as a rapid growth at geometric rates during the first introduction. at the saturation stage, the growth slows down to arithmetic rates until maturity. parameter r allows us to control the speed of the emergence. in this work, we experiment with @xmath0.3 for slow emergence, @xmath1.5 for normal emergence and @xmath2.0 for fast emergence. with one category dynamics matching exactly this signal, we make sure to have a quantifiable gold standard as a proxy of ideal ground truth. this approach allows us to organize which category will act as our emerging novelty, to quantify the importance of the rate at which the novelty emerges. there is only one category introduced as novelty during each experience but we are repeating our experiences several times on several categories in order to have stable results. ## cend methodology the use of embedding representation spaces for analyzing language evolution is now a common field in the nlp literature. in this kind of representation, similar entities (e.g., words or topics) are close if they are used in the same context: their meaning depends on the entities around them. over time, their meaning may change, therefore their representation in the embedding space may be modified: there is a movement. in this work, we consider one type of movement in this space. let @xmath3, . . . , w n } be a set of words. for each word @xmath4, say v @xmath5, using a word embedding algorithm, say a (e.g.,svd word2vec, glove, fasttext). typically, we examine how the vector v w changes when consecutive bunches of documents are used to update the parameters of model a. let us consider v w 1 , v w 2 , . . . , v w t a sequence of vectors representing the word w at each time slice @xmath6, . . . , t . now, we define a measure to quantify the changes in the representation of the word w. magnitude of the change. for two consecutive vectors, we look at the size of the change using the euclidean distance: in section 5, we show that a negative correlation seems to exist between words movement in an em-bedding space and its frequency if this word is part of an emerging topic. following this hypothesis, we based our algorithm on this correlation and we monitor its evolution through time for each word in our vocabulary. as our goal is to detect as soon as possible if a word is a part of an emergent topic, we monitor the spearman correlation ρ x,y between movement d w and frequency f w on a portion of the signal: ## experimental setup in this work, we focus on the correlation between word frequency and word movement in the chosen embedding space. by artificially inserting documents related to an emerging new topic in a corpus as described in section 3, we notice that the amplitude of the movement of a word in the embedding space is linked with the dynamic of its frequency. in this section, we explain how correlation between these two measures and the dynamic of a topic are linked in a textual corpus. ## results in this section, we present the results obtained by our method for detecting words carried by a slow emerging topic. as we developed an artificially introduced signal (section 3) and a gold standard corresponding to this signal (section 5), it is possible to evaluate quantitatively with metrics such as precision/recall/f-measure and auc. additionally to the general results, we investigate the effect of 3 parameters: the rate of emergence, the threshold and the size of the sliding window. finally, we confirm, with the help of a control group, that our method effectively detects words linked to our artificially introduced topic. ## discussion and conclusion in this work, we presented a method for detecting weak signals associated with slow emerging topics in textual streams. we designed simple experiments to test and measure quantitatively the performance of our approach. we analyzed the impact of hyper-parameters and showed that our method outperforms other algorithms from the literature. we based our method on the hypothesis that signals associated with slow emerging topics present a specific type of movement in embeddings spaces built with svd and sgns. we verified this hypothesis and observed that words associated with a slow emergence tend to present a negative correlation between their movement and frequency. we noticed that the positive correlation between movement and frequency represents a documented characteristic of word2vec: the more a word is used in a common context, the larger the norm of its vector is @xcite . we also noticed that correlations are mostly negative when using svd on the sppmi matrix for modelization. this difference could be explained by the fact that svd is a much more stable algorithm and less biased towards new observation as sgns. this conclusion is supported in @xcite but should deserve more analysis in our specific framework. in this work, we chose to use simple embeddings algorithms and not contextual approaches as bert @xcite and @xcite because we wanted to show that our observation is effective to detect novelty in a simple and inexpensive manner. our observations (i.e. negative correlation) have been done on two different corpora and used for a detection task. it can be explained by the fact that a word is poorly defined when it has not been used enough in a corpus and its position in an embedding space become more precise when its frequency increases. whether this observation is more largely valid or not should be confirmed by future research. in particular, we believe that the quality of the text may play an important role. in this work we only used curated documents with high quality writing coming from journalistic and scientific articles. several extensions for our method are easily conceivable. we showed in figure 1 the differences in the distribution of correlation between novel and pre-existing words and it seems that these distribution could be separated by a statistical test inspired by @xcite . also, some approaches in the literature weight each word in relation to its part-of-speech tagging. we could imagine that it would make the detection task easier because discriminative words for a category tend to be nouns. it would be easier, for future industrial use, to automatically cluster detected words in order to better illustrate the detected emerging topic.