Effect of data quality on the automated identification of register features in Eighteenth Century Collections Online (2023)

Abstract: Many large-scale investigations of textual data are based on the automated identification of various linguistic features. However, if the textual data is of lower quality, automated identification of linguistic features, particularly more complex ones, can be severely hampered.Data quality problems are particularly prominent with large datasets of historical text which have been made machine-readable using optical character recognition (OCR) technology, but it is unclear how much the identification of individual linguistic features is affected by the dirty OCR, and how features of varying complexity are influenced differently.In this paper, I analyze the effect of OCR quality on the automated identification of the set of linguistic features commonly used for multi-dimensional register analysis (MDA) by comparing their observed frequencies in the OCR-processed Eighteenth Century Collections Online (ECCO) and a clean baseline (ECCO-TCP). The results show that the identification of most features is disturbed more as the OCR quality decreases, but different features start degrading at different OCR quality levels and do so at different rates.

## introduction large-scale textual datasets have become increasingly common in computational linguistics and in various subfields of digital humanities. naturally, such datasets cannot easily compare to the high quality of traditional (but much smaller) linguistic corpora, which have been carefully curated for balance and manually edited to ensure the accuracy of the textual data to as large a degree as feasible. instead, it is the expectation that when analyzed in aggregate, the large amount of data can smooth over many of the flaws which would prove very difficult to work around with smaller datasets or when focusing the analysis on individual texts or individual instances of items. even then, the overall lower quality of the data causes issues for many linguistic and other digital humanities analyses, such as in the case of, for example, the analysis of social media data (e.g. eisenstein, 2013). however, a major source of large-scale lowquality textual humanities data, particularly in fields such as historical linguistics and computational history, are texts which have been turned from scans of physical documents into machinereadable format using optical character recognition (ocr) technology. for instance, the ocr quality of eighteenth century collections online (ecco) foot_0 , a collection of over 200,000 mostly english-language works published in the united kingdom during the 18th century (see tolonen et al., 2021) and a central resource for dh scholars (gregg, 2020), is extremely variable. a main contributing factor of the often low ocr quality for ecco is the ocr being run on bitonal scans of microfilms with ocr algorithms which have not been fine-tuned for eighteenth-century typefaces or trained to recognize e.g. the long s character s . earlier studies on the effects of ocr errors in ecco (e.g. hill and hengchen, 2019) are largely focused on individual tokens, characters, and ngrams. in contrast, the present study focuses on the effect of the ocr errors in ecco on the automated identification of a set of more complex linguistic features commonly used for the multi-dimensional method of register analysis (mda) (see e.g. biber, 1988; biber and conrad, 2009). ## results figure 1 shows the results of the analysis for each of the features as a function of the ocr quality of the text. every facet represents a different linguistic feature, with the ocr quality estimate for the text on the vertical axis, higher ocr quality at the top and ocr quality decreasing towards the bottom. the horizontal axis represents the proportional change in the observed frequency of the feature from ecco-tcp to ecco-ocr. because there is a lot of variation in the data, a smoothing trend line 10 was also included for each feature. in other words, points at 0.0 on the horizontal axis in figure 1 , i.e. on the dark vertical line, represent texts with the exact same frequency in both ecco-tcp and ecco-ocr. such texts are the ideal case, since they have no change in the observed frequency from the clean version to the ocr version of the text, suggesting that all instances of the feature have been correctly identified. however, in practice, as expected, most texts deviate from the ideal. points to the left of the middle line represent texts which have a lower frequency of the feature in ecco-ocr than in ecco-tcp. for instance, in a text at the -0.5 mark, the ocr version has only half the observed frequency of the feature as compared to the clean version. similarly, the points to the right of the middle line indicate texts with a higher frequency of the feature in ecco-ocr than ecco-tcp, with the 0.5 mark meaning a 50% increase in the observed frequency. based on figure 1 , it is clear that as the ocr quality decreases, the observed frequency of identification typically also decreases, as evidenced by the trend line tending down and to the left. but different features are also clearly affected differently by the decrease in ocr quality. for some features, the fall towards the left is very direct and rapid, beginning very close to the highest ocr quality texts, meaning that their identification is instantly 10 ggplot2 (wickham, 2016): geom smooth(@xmath0, @xmath1, @xmath2 figure 1 : proportion of ecco-ocr rate of occurrence of the ecco-tcp rate of occurrence by ocr quality affected when the ocr quality decreases. these include features such as direct wh-questions 11 and that relative clauses on object position 12 . features which appear to be particularly adversely affected by dirty ocr include whclauses 13 , which have on average lower observed frequencies even in the ocr texts with the best ocr quality, and the frequency drops very rapidly with the ocr quality. possibly because of ocr problems with the long s, necessity modals 14 also start at a lower observed frequency even at highest ocr quality, and keep decreasing in frequency with decreasing ocr quality, if somewhat less quickly than wh-clauses do. in contrast to features of the above type, which start decreasing as soon as the ocr quality decreases, there are a number of features which stay relatively stable for a larger portion of the ocr quality range and only show larger changes in their average observed frequency when the ocr quality drops too much, possibly because they tend to be relatively simple to identify. these features include present tense verb forms, which only show a drop below 60-70% ocr quality, attributive adjectives 15 , which show an increase below a similar ocr quality level, and predicative adjectives 16 , which show a drop below 70-80% ocr quality. there is also an even larger group of features which do show a small difference from the clean version at higher ocr quality levels but for which this difference is only relatively minor, including e.g. the analytic negation not, first person plural pronouns 17 and total adverbs 18 , all of which only show a larger change in their observed frequency below about 70% ocr quality. another, rather curious group of features are those whose identified frequencies increase instead of decrease in the ocr dataset. typically, this increase can be attributed to the ocr process and other processing of the dataset, particularly the tokenization and the part-of-speech tagging. for instance, the total other nouns 19 category shows higher frequencies in ecco-ocr compared to the clean baseline throughout the ocr quality range. this is because the imperfect ocr process creates many strings of characters which the part-of-speech tagger cannot recognize, and most taggers have a tendency of tagging such tokens as nouns, particularly if there are too many unrecognizable tokens in succession to infer a different part of speech from the surrounding structure. similarly, first person singular pronouns appear to be identified at a close to correct rate in the ocr texts close to the top ocr quality, but there is an increasing number of misidentifications as the ocr quality decreases. this is largely because as the ocr quality degrades, the ocr process produces more and more random i characters, which then get misidentified as the first person pronoun i. finally, there are a few features which appear to be barely affected by the decrease in ocr quality. these are likely relatively simple to identify features which also happen to consist of characters which tend to have been recognized more reliably than average in the ocr process. these features include most prominently synthetic negation @xcite , but features such as pronoun it and infinitives could also be considered to not be affected very much at all by changes in ocr quality. ## conclusion the results show that, as expected, lower ocr quality leads to lower reliability of identification for most of the features analyzed. however, the effect of ocr on all of the features is not the same, with features which are simpler to identify and less likely to invite ocr errors appearing generally more resistant to lower ocr quality, while features whose identification involves multiple consecutive items from specific lists appear generally more at risk. it is not possible based on this study alone to recommend any single ecco ocr quality level which could be considered "good enough" for most analyses, as what is good enough depends on the features one is interested in. on the other hand, mda studies consistently produce "compatible" results regardless of the exact set of features analyzed (mcenery and hardie, 2012), suggesting that for mda, focusing on a smaller set of more resilient features may be enough to get better results even from lower-quality textual data. hadley wickham. 2016. ggplot2: elegant graphics for data analysis. springer-verlag new york. a list of features included agentless passives amplifiers analytic negation: not attributive adjectives be as main verb by-passives causative adverbial subordinators: because concessive adverbial subordinators: although, though conditional adverbial subordinators: if, unless conjuncts contractions demonstrative pronouns direct wh-questions discourse particles downtoners emphatics existential there first person plural pronouns first person singular pronouns hedges indefinite pronouns independent clause coordination infinitives necessity modals nominalizations other adverbial subordinators past tense perfect aspect phrasal coordination pied-piping relative clauses place adverbials possibility modals predicative adjectives predictive modals present tense private verbs pro-verb do pronoun it public verbs second person pronouns seem & appear split auxiliaries split infinitives suasive verbs subordinator-that deletion synthetic negation that adjective complement that relative clauses on object position that relative clauses on subject position that verb complement third person personal pronouns time adverbials total adverbs total other nouns total prepositional phrases wh relative clauses on object positions wh relative clauses on subject position wh-clauses