Life still goes on: Analysing A ustralian WW 1 Diaries through Distant Reading (2020)

Abstract: An increasing amount of historic data is now available in digital (text) formats. This gives quantitative researchers an opportunity to use distant reading techniques, as opposed to traditional close reading, in order to analyse larger quantities of historic data. Distant reading allows researchers to view overall patterns within the data and reduce researcher bias. One such data set that has recently been transcribed is a collection of over 500 Australian World War I (WW1) diaries held by the State Library of New South Wales. Here we apply distant reading techniques to this corpus to understand what soldiers wrote about and how they felt over the course of the war. Extracting dates accurately is important as it allows us to perform our analysis over time, however, it is very challenging due to the variety of date formats and abbreviations diarists use. But with that data, topic modelling and sentiment analysis can then be applied to show trends, for instance, that despite the horrors of war, Australians in WW1 primarily wrote about their everyday routines and experiences. Our results detail some of the challenges likely to be encountered by quantitative researchers intending to analyse historical texts, and provide some approaches to these issues.

## introduction world war i (ww1) was a defining event of the 20th century, and impacted millions worldwide. researchers have studied the war, especially the experiences of those on the front lines. primarily, this has been done through close reading of primary sources such as diaries and letters. however, recent advances in computational methods to analyse large text corpora offers the opportunity to analyse sources such as these through distant reading. distant reading involves the application of mathematical and computational techniques from natural language processing (nlp) to perform statistical analysis of text @xcite . distant reading has several advantages, including the ability to analyse large quantities of data and see overall patterns as well as the reduction of researcher bias. further, distant and close reading can be combined such that interesting patterns found through distant reading can be more closely examined using close reading to determine why they occur. this work aims to use distant reading to understand what australian soldiers went through and how they felt over the course of ww1, by analysing a unique historical data set: a large collection of transcriptions of australian soldiers' diaries, held by the state library of new south wales. to our knowledge this paper represents the first nlp analysis of this data set. this research takes advantage of the fact that diaries contain temporal information. however, extracting dates is a difficult task due to the varying manner in which dates can be written. this is further complicated by the desire to focus on the dates on which entries were written and not dates mentioned within the entries as these may refer to times and events from outside the war or at least out of the context of the current entry. in order to extract and clean dates we use a combination of regular expressions and optimisation. once dates were extracted we were able to apply topic modelling and sentiment analysis as a function of time. we are able to detect topics corresponding to particular developments of the war, and the associated sentiment for those periods. further, we show that the diarists wrote more about everyday experiences, e.g., the time of day and meals, than they did about training and battles. this might be surprising as the war was one of the most traumatic events of the twentieth century and conventional historical narratives concentrate on the pain and suffering of the soldiers. however, in the diaries, we see the war's participants adapting their everyday lives to their circumstances, and in fact their overall sentiment across the war is surprisingly positive. ## corpus we focus on australian ww1 diaries held by the state library of new south wales. after the war ended, the european war collecting project was created by principal librarian william ifould and the trustees of the library (state library of new south wales, 2019). their aim was to collect documents, including diaries, letters, war narratives, memoirs and photographs, which gave the experiences and personal feelings of those who served. in total, this collection has 966 documents, 557 of which are nonempty war diaries. a complete breakdown of the collection is given in these documents can be individually downloaded from the state library's website (state library of new south wales, 2020), however, we obtained the corpus directly from the library. before this corpus could be used it went through a variety of cleaning steps. first, the raw data was converted to a single text file per document, and a metadata table was created by using regular expressions to extract information from the document titles. then dates had to be extracted so that we could perform analysis over time. raw dates were extracted using regular expressions, however, several issues were found in these raw dates requiring us to clean them through optimisation. more information regarding this is given in section 4.1. finally, we changed all text to lowercase, removed numbers and punctuation, singularised words, converted abbreviations to the full word, and for topic modelling we removed stop words. converting to lowercase, singularising words and converting abbreviations were all done to ensure that the various versions of a word are considered as the same word when performing our analysis. for example, "kill", "killing", and "kills" all have the same base word: "kill". the stop words we removed were based on the stop words data set in the tidytext package @xcite in r. figure 1 shows the number of words in our diary corpus per month after this process. unsurprisingly, the majority of entries were written between august 1914 and december 1919 -britain, and consequently 3 related work and background ## conclusion and future work this research aimed to analyse australian ww1 diaries in order to determine what the soldiers wrote about and how they felt over the course of the war. through the application of distant reading techniques we have seen that we can analyse large amounts of data to determine trends. interestingly, while many people typically think of the war as a horrific experience we find that the diarists primarily wrote about their day-to-day activities. as such the diaries had an overall slightly positive sentiment, which is consistent with the positivity bias seen across human languages @xcite , but is surprising for this particular corpus. we focused on dbms for sentiment, and found that the dictionaries used covered less of the diaries than standard texts such as the brown corpus. this suggests that dbms may not be the most accurate method for determining sentiment in ww1 diaries and as such in the future we will investigate other sentiment analysis techniques such as embedding-based methods to determine if they are more applicable. further, in the future we will write a paper detailing the difficulties with date extraction, as well as our approach and the accuracy of our method, as this is not a trivial issue.