Enhancing Digital History – Event discovery via Topic Modeling and Change Detection (2022)

Abstract: Digital history is the application of computer science techniques to historical data in order to uncover insights into events occurring during specific time periods from the past. This relatively new interdisciplinary field can help identify and record latent information about political, cultural, and economic trends that are not otherwise apparent from traditional historical analysis. This paper presents a method that uses topic modeling and breakpoint detection to observe how extracted topics come in and out of prominence over various time periods. We apply our techniques on British parliamentary speech data from the 19th century. Findings show that some of the events produced are cohesive in topic content (religion, transportation, economics, etc.) and time period (events are focused in the same year or month). Topic content identified should be further analyzed for specific events and undergo external validation to determine the quality and value of the findings to historians specializing in 19th century Britain.

## introduction the field of digital history involves the application of computer science techniques to historical data. research in this field is aimed at uncovering both obvious and latent information about specific time periods from the past, allowing for a deeper understanding of historical events. specifically, using natural language processing techniques on historic text data can be valuable in determining what factors are catalysts for change. issues, ideas, and sentiments can suddenly become viral and become triggers for influential events. in this paper, we present our work on detecting these factors by pinpointing which topics gain or lose prominence over certain time periods in history. we test our methods by applying them to a dataset of 19 th century british parliamentary debates from the house of commons. we define our task as one that discovers when political, cultural, and economic trends grow and/or shrink with respect to significant increased or decreased discussion of certain topics in parliamentary discourse. we model the ideas by turning to standard (generative) topic models, such as lda @xcite . these models are well studied and have been applied in a variety of fields, including the humanities @xcite @xcite . in many such models, there is an intuitive description for topics that makes it feasible for users to detect what ideas are being represented in their data. the next thing we need to model is the change in prominence of topics. there is a class of topic models known as dynamic topic models @xcite that attempts to determine the evolution of the most prominent topics over time. while this is useful in a lot of applications, this information does not necessarily show us the scope of the change in each topic's prominence. for instance, while a topic that is generated in time t does not appear in time t-1, it is not easy to determine whether its prominence suffers just a small drop or falls off a cliff. thus, in this work, we took the approach of developing a single topic model for all documents across all time slots. we intentionally generate a larger number of topics and use measures to quantitively measure the importance of each topic for each time slot. thus, we can generate a time series of importance for each topic. we then apply a changepoint/breakpoint detection algorithm on the time series to detect major changes in the time series and capture where certain topics enter/leave parliamentary debate. another issue we look at is the robustness of the results. topic model algorithms generate different results for each run, an undesirable characteristic @xcite . methods have been proposed for combating this instability @xcite @xcite . in our work, we incorporate methods to overcome the instability by running the model multiple times and using clustering techniques to combine the results and enhance stability. ## experimental results we create a subset of the data for use in our experiments by selecting 500 samples from each month of the dataset's representative time period using stratified random sampling. we set the number of clusters detected by agglomerative clustering to @xmath0. we evaluate results for the distance metrics used in both methods on the basis of both cluster cohesion and topic distinctiveness. for each approach-metric combination, we analyze cluster tendency plots and the spread of topics across clusters. cluster tendency plots used include vat and ivat, which reveals hidden cluster structures as dark blocks along the diagonal of the image representation. we also analyze topic annotations created by extracting the top documents from each cluster based on the aggregated probability of a document belonging to the cluster topics. as mentioned in the previous section, we utilize the jaccard coefficient, jaccard-p coefficient, and jensen-shannon distance as distance metrics for agglomerative clustering. for the jaccard approach, the sets of terms used to calculate the coefficient include the top terms from each topic with a topic-word probability 100x greater than the overall probability the term would appear in a random document. we further add rare words (those occurring less than 10 times across all topics) and unique words (those that were completely unique to the given topic). for the jaccard-p approach, the sets of terms used to calculate the coefficient include the top terms from a topic that have a topic-word probability of greater than 0.25%. for the jensen-shannon approach, the topic-word vectors used to calculate the distance include the top 1000 terms from each topic with the highest probabilities. the jaccard approach resulted in a less effective extraction and clustering of topics. over 70% of the total topics were contained within one cluster, indicating one large generic cluster and many small specific ones. the topic annotations corroborate this finding -the top documents from the large cluster have a variety of topics, and the number of topics in the other clusters are too small. the jaccard-p approach seems to mitigate the original issues of using jaccard due to its different word set composition and probability threshold. one larger cluster still exists, but the topics are more evenly spread across the identified cluster, as shown in figure 3 . moreover, the vat diagram (figure 4 ) shows greater cluster distinctiveness. the topic annotations generated from the jaccard-p approach indicate that certain clusters do exhibit topic cohesion. relevant speeches for the clusters show thematic similarities, and topic-generated time series show similar trends in rise and fall across time intervals. figures 5 and 6 shows the information of one such cluster. the breakpoint mapping over time in figure 5 shows that there are identifiable periods of time (spanning months or years) for the events or trends associated with this cluster. figure 6 shows a selected subset of the time series of the lda topics from this cluster. there are similarities in time series across multiple runs, showing that there are corresponding rises and falls in topic prominence over time. the similarities also show that the agglomerative clustering was effective in combating lda instability. the examined jaccard-p cluster's topic annotations indicate topic cohesion. the top documents of this cluster cover discussions on treasury legal tender, the value of money used international trade, and the interest rates established by the bank of england. these points of discussion are related in the areas of economics, finance, and trade. common important terms extracted from the documents include "gold," "payment," "price," and "bank." one note is that clusters, including the one being examined, can include certain documents that are not as related to the common theme. for example, this cluster's fifth most important document relates to education, instead of economics. this indicates that we can continue to improve upon our approach to filter out unrelated documents. the jensen-shannon approach behaves somewhat better than the jaccard-p approach in terms of topic distribution across clusters (figure 7 ). the vat diagram (figure 8 ) shows internal cluster cohesion, and the topic cohesion is present for many clusters. in addition, we discovered a jensen-shannon cluster about finance and economics, containing similar content, documents, and breakpoints to the jaccard-p cluster discussed earlier. this observation indicates that we can compare clusters across approaches. the jensen-shannon cluster chosen for examination here (figures 9 and 10 ) highlights ideas that reoccur frequently across the century. the selected subset of lda topic time series from this cluster again show similarities between time series and robustness across runs of lda. the topic annotations of the jensen-shannon cluster also indicate topic cohesion. the top documents of this cluster have a focus on educational systems, with additional commentary on government and political systems. common important terms extracted from the documents include "school," "teacher," "election," and "representative." future work can focus on distinguishing between these somewhat discrete topicsbreaking down larger clusters into smaller ones on other criteria can yield more specific identifications of events and trends. the difference in the word sets used for each approach contributed to the differences seen in the results. with the jaccard approach, we saw less success with clustering and identification of topics, indicating that we can modify the jaccard word set composition to be similar to those used in the jaccard-p and jensen-shannon approaches for future experiments. ## conclusions and future work the problem of event discovery using topic modeling and change detection is a challenging one. the two experimental methods we define in this paper yielded results with varying degrees of success. our most reliable results came from the jaccard-p and jensen-shannon approaches from method 1, where generated lda topics were clustered based on their document content. we were able to create clusters with distinct areas of discussion, such as finance or education, which we can continue to do analysis on to identify specific historical events. our first approach can be improved by increasing the number of samples analyzed per time interval or increasing the granularity of the time interval used. we plan to break down each cluster into smaller sub-clusters to examine more specific topic content -for example, our jaccard-p cluster could be dissected to explore historical discussions on specific components of the british economic system. we would also like to explore dynamic time warping technique to measure time series similarity. the simple distance metrics used in our approach suffer from a misalignment problem, where computations rely on a one-toone mapping of corresponding observations in time series. dynamic time warping solves the misalignment issue by exploring different warping paths and finding the optimal one that allows for matching of similar time series with different phases. finally, we aim to consult with historical experts specializing in the analyzed time period. these experts can provide external validation of the topics generated and insight into what potential changes can be made to our approaches to benefit future historical work.