NLP for Digital Humanities: Processing Chronological Text Corpora (2024)

Abstract: The paper focuses on the integration of Natural Language Processing (NLP) techniques to analyze extensive chronological text corpora. This research underscores the synergy between humanistic inquiry and computational methods, especially in the processing and analysis of sequential textual data known as lexical series. A reference workflow for chronological corpus analysis is introduced, outlining the methodologies applicable to the ChronoPress corpus, a data set that encompasses 22 years of Polish press from 1945 to 1966. The study showcases the potential of this approach in uncovering cultural and historical patterns through the analysis of lexical series. The findings highlight both the challenges and opportunities present in leveraging lexical series analysis within Digital Humanities, emphasizing the necessity for advanced data filtering and anomaly detection algorithms to effectively manage the vast and intricate datasets characteristic of this field.

## about digital humanities digital humanities (dh) today is a broad domain of research and practical applications of various techniques for automatic processing of data, representing linguistics, literary studies, history, art history, cultural anthropology, or archaeology, among others. it can be defined as a system of interrelated resources, functionalities, and cognitive practices created by transferring to the digital realm and creatively expanding the heritage of the humanities that grew out of print culture @xcite @xcite . however, behind the apparent plethora of different descriptions of digital humanities lies the same recurring set of characteristics, among which the most important are: the study of large data sets, the linked open data approach, the extensive use of metadata, the fusion of natural and artificial intelligence, transdisciplinarity, multimedia, an almost radical empiricism, and last but not least, the dominance of interactive, dynamic infrastructures over the static products of the gutenberg era, such as articles, chapters, or monographs. describing the current state-of-the-art of digital humanities is therefore a task that is all the more difficult because the discipline is undergoing a phase of rapid, dynamic expansion today. moreover, the efforts of many dh centers focus on digitizing the resources produced by humanity in the past centuries and enriching them with the tools of computational intelligence. the obsolescence of dh definitions, theories, and research practices is hence rapid. here, we adopt a research perspective in which digital humanities is a strand at the intersection of applied computer science and natural language processing (nlp). this assumption does not contradict the principles of hc, but it exposes the methodological aspects on which nlp's attention is focused, rather than the problem of digitizing and/or sharing resources. ## language and nlp vs digital humanities language has not been a major area of digital humanities in recent years. as the topics of papers submitted to major dh conferences 1 show, interdisciplinary topics dominated there, combining history, art history, cultural anthropology or archival science. the formats processed in dh were, in varying proportions, text, but also image, sound, geographic coordinates (used to formally represent geolocation of objects) and time (dates, hours). however, on a more general level of reflection, natural language preserves its privileged role in the humanities, and this is something that the digital world will not change. after all, language is the primary and universal tool of communication, shaping in the human mind a system of representation of the world and modeling the processes of thinking. taking the above statement about the privileged role of language in the digital humanities as correct, the potential for collaboration between humanists and language engineers is enormous. nlp applications start with simple text string processing tasks, including segmentation, disambiguation, and morphosyntactic annotation of lexemes, followed by calculating the frequency of any segments. a more advanced level of nlp applications involves generating segment relationships (lexemes or multiword units) that take the form of networks (e.g., wordnets), hierarchical structures (e.g., dendrograms) or point clouds. the next level of nlp application is semantics analysis, which is usually based on distributional relationships of segments (e.g., topic modeling) and/or uses neural networks. a specific class of textual resources is the so-called chronological corpora, which contain samples arranged on a timeline. below is an example of the application of nlp methods and mathematical statistics to such a corpus. ## sequential data and chronological corpora by textual sequential data, we mean data representing consecutive segments of a corpus on a linear time axis. representation here means the moment of text production, but is not related to style, genre, age of the writer, or context. out of the wide spectrum of discourses, the best material to construct chronological corpora are media texts generated by the press, radio, television and all internet formats. the press is of particular importance here, since it covers long time spans: media institutions in the modern sense have existed in various countries of the world for at least two hundred years. the granularity of the existing resources varies. the earliest texts may have daily or even monthly dates, whereas contemporary news is marked with minute or second accuracy. the chronology of texts can also be discovered in corpora of literary and applied texts, but the granularity in such a situation is at least annual, which makes it necessary to have really large volumes of evenly distributed data. sequential text data, which by analogy with time series can be referred to as 'lexical series', obviously have their weaknesses. when extracting electronic text from print, acquiring clean ocr output is a problem. recordings from radio, television, and the internet require transcription supervised by a human. cleaning and curing the data thus adds significantly to the cost of producing press corpora suitable for chronological analysis. as a result, research on such corpora is not as developed as on large corpora of general language. an important issue here is to distinguish chronological analysis from diachronic studies. the former deals with changes in the frequency of relevant lexemes at fixed intervals of time, while the latter describes the evolution of language forms over time. diachronic research can be carried out with quantitative methods, but the nature of the phenomena under study is quite different: it is about change, disappearance and/or appearance of new lexical (rarely morphological or syntactic) forms. an example of a quantitative diachronic research is a study of lexical changes in the polish language over a period of 600 years @xcite . one of the methods for quantitative modeling the dynamics of language change is piotrovsky's law. this soviet linguist with polish roots noted that linguistic changes usually have non-linear patterns, resembling a logistic function @xcite . for the above reasons both approaches should not be confused: they deal with completely different problems, and the reference to time is only an apparent similarity. bringing them together is all the more difficult because chronological research by nlp methods assumes stable orthography and does not require corpora, derived from long (preferably several hundred years) periods. in contrast, the object of chronological studies are corpora covering shorter periods with stable orthography. ## purpose of the project and test data the purpose of the project is to create a reference processing workflow for textual chronological data and to prepare nlp tools that would serve for lexical series analysis. the first step of the processing flow is to determine the characteristics of the data that are suitable for such analyses and to prepare the test material. the second stage involves the definition of patterns of lexical series, relevant to the needs of the humanities (in particular, linguistics and cultural anthropology), and the preparation of algorithms for extracting such patterns (a priori approach). for some lexemes (represented by lexical series), this stage may include the estimation of trend models and/or stochastic processes if the se-ries contains periodic oscillations. the third stage involves conducting an unsupervised taxonomy of lexical series, leading to the empirical extraction of semantic classes (a posteriori approach). as to the first issue, sequences of text samples of the same volume, produced at equal intervals of time (the benchmark example of such a source is the press) should be considered the data best suited for chronological analysis. additionally, the volume of such samples must be large enough to generate statistically significant lexeme frequencies. equal sample lengths eliminate in the simplest possible way a troublesome feature of linguistic systems, which is that the dependence of the frequency of the vast majority of lexemes and the volume of the sample is not linear, and, in addition, for each lexeme follows a slightly different curve (in simple terms, if the frequency of a lexeme l in a sample of volume n is l i , it does not mean that in a sample of 10 * n it will be 10 * l i -actually it will be lower due to the constant appearance of new words). thus, only comparing samples of the same length gives reliable and indisputable results. as for the sought-after patterns in lexical series, they duplicate and minimally extend the patterns identified in classical time-series analysis, that is, stable trend and periodic oscillations @xcite . in addition, random series and anomalies ("catastrophes") as non-deterministic patterns are also important in lexical series analysis. the first two types of series express either longterm processes of cultural change (trends) or periodic phenomena, driven by natural cycles (weather, agricultural work, seasonal diseases, etc.) and rituals of culture (anniversaries, holidays, cyclical political events). a non-deterministic pattern with significant informational value is an anomaly, i.e., a sudden jump in the value of a series, caused by some sudden event (the death of a well-known figure, a natural disaster, a change in the name of a great city, etc.). interestingly enough, in the context of language and culture, researchers may also be interested in random lexical series. the third stage of the lexical series processing flow potentially includes two modules. if periodicity of the series is detected, a process model (ar, ma, arma, arima) can be estimated. however, in the digital humanities, the usefulness of such models is low and it is also difficult to interpret their parameters. as for trend estimation, only working on large datasets covering long periods gives reliable results. on the other hand, a much more interesting and so far unused approach in this respect is the taxonomy of series. each lexical series can be treated as a vector representing a point in some multidimensional space. thus, all lexical series can be projected into this space and can, after dimension reduction, generate human-readable dendrograms or point clouds. it can be expected that such a taxonomy will not be very transparent since placing tens or hundreds of thousands of points in a single space produces a result that is opaque and difficult to interpret. nevertheless, lexical series (and thus lexemes) with similar frequency characteristics are likely to form at least some visible clusters. the above processing workflow was tested on the chronopress corpus, which represents 22 years of polish press from the period 1945-1966 foot_1 . this corpus has a volume of ca. 24 milions of segments, evenly distributed by year and month. the texts are lemmatized and chronologically annotated. one year is represented on average by 1,098,526 segments with a standard deviation of 69,843, and the average volume of monthly segments is 91,544 with a standard deviation of 6,201. the volume of the corpus thus allows for annual and monthly granulation. here, monthly granulation was used, which made it possible to generate lexical series (or vectors) of 264 (=22*12) units in length. ## research methods in the case of linguistic data, the study of time series consists of two modules. the first comprises nlp tools necessary to generate the input data from the corpus, and the second includes numerical methods of time-series analysis and taxonomy. as for the nlp methods module, the corpus was lemmatized using the wcrft2 tagger for polish @xcite . each sample includes publication data, allowing us to calculate the number of lexeme occurrences in each month by summing the lexeme numbers obtained from samples published in that month, obtaining time series. in terms of time series analysis, we rely on the standard model of @xcite , exposing, however, the specifics of text corpora. the standard model assumes that any time series consists of a trend, periodic oscillations, and noise. the processing flow includes: (1) identifying the trend and cutting it off from the data, thus making the series stationary; (2) identifying periodic oscillations by calculating the autocorrelation function (acf) and partial autocorrelation function (pacf). the shape of the acf and pacf functions allows one to choose the optimal type and order of the model -autoregressive, moving average, or mixed (arma, arima). after cutting off the trend and periodic oscillations from the time series, the residual series that remains should meet the criteria of white noise, and the percentage of explained variance indicates the contribution of the deterministic component to the series. however, from the perspective of the digital humanities and text processing, some of the functionalities of the standard box and jenkins model are not as useful as in economics or engineering. this model, as the title of the cited work indicates (time series analysis: forecasting and control), was created to predict and/or steer processes. so, while one can understand an economist trying to predict in advance the price of some raw material or the exchange rate of a currency, a humanist does not wonder what the frequency of some lexeme in a stream of media texts will be next month. therefore, the module for modeling stochastic processes in lexical series of the type described here is not particularly important. an autoregressive model identified in a lexical series would at most show the depth of cultural or societal 'memory'. the previous application of this method in linguistics or textual studies confirms this statement, since the object was to explain some linguistic phenomena, not predict them @xcite @xcite . for the above reasons, the acf and pacf functions should be considered key tools for the humanities to identify periodic phenomena in the great mass of data. this situation raises a fundamental challenge for nlp, namely the need to generate and filter data from a corpus. chronological analysis of a corpus of texts assumes that there are as many time series as there are different lexemes in the corpus, and the task of the researcher is to identify among them those that are for some reason significant. despite the great cognitive capabilities of human mind, this task is not feasible without the support of nlp. for example, in the chronopress test corpus processed here, the initial number of series was close to 100,000, and this is more likely to be the lower limit, since we are talking about a corpus with an average volume. an additional dif-ficulty is that the anomaly (catastrophe) pattern can involve series that have subsequent values close to zero almost throughout the run but once their value unexpectedly spikes. an example of such a lexeme is 'comet'. normally, the press does not write about comets, so the word is almost absent from the media discourse. but, like a real comet, it suddenly appears every few or a dozen years and has higher frequencies. similarly, lexemes with relatively low frequency, and therefore irrelevant in the perspective of big data, can appear rhythmically. therefore, the low average frequency of a lexeme is not a sufficient criterion for its elimination. the same applies to the variance of the series, which may be too low to indicate interesting cases of anomalies. to overcome these difficulties, we have developed algorithms for automatic series filtering and anomaly detection. ## conclusions while traditional time series analysis seems to be a task of fortune-telling (who wouldn't want to know next week's stock prices...), lexical series analysis is akin to looking for a needle in a haystack. in the research presented here, the 'needle' was the time series, containing the trend, periodic oscillations and anomalies, and the haystack was the corpus of 24 million words, divided into 264 sections. in addition, we were looking for lexemes that display similar shapes of time series. the techniques elaborated during our research allowed us to identify automatically a set of lexemes from the corpus of nearly 100,000 that we found relevant in some way. in particular anomaly / catastrophe detection helps us to pinpoint lexemes that undergo rapid changes in occurrence, with some unexpected cases like "plebiscite" or "dz-erzhinsky". another promising technique involves detecting lexemes with similar time-series patterns. the semantic similarity between words has been a foundational concept in modern nlp, based on deep neural networks and generative models. the transformers architecture @xcite originates from the word2vec method @xcite , which creates word representations using large data sets and word occurrences in similar contexts. the method presented here, utilizing normalized time series, shares similarities with word2vec, as it constructs vectors from a large corpus. however, the novelty of our approach is the reliance on co-occurrence in recurring units of time (e.g., years) rather than cooccurrence in the text. the results shown in