“Don’t worry, it’s just noise’”: quantifying the impact of files treated as single textual units when they are really collections (2021)

Abstract: Literature works may present many autonomous or semi-autonomous units, such as poems for the first or chapter for the second. We make the hypothesis that such cuts in the text’s flow, if not taken care of in the way we process text, have an impact on the application of the distributional hypothesis. We test this hypothesis with a large 20M tokens corpus of Latin works, by using text files as a single unit or multiple “autonomous” units for the analysis of selected words. For groups of rare words and words specific to heavily segmented works, the results show that their semantic space is mostly different between both versions of the corpus. For the 1000 most frequent words of the corpus, variations are important as soon as the window for defining neighborhood is larger or equal to 10 words.

## introduction if we were to take the example of the perseus's "canonical latin literature" repository @xcite , some files are actually composite works, such martial's epigrammata, a collection of more than a thousand poems of varying but short lengths, while some others are "more" monolithic such as an oratio of cicero. while there are (rare) studies of noise in digital humanities, and the few that exist focus on ocr quality and its impact @xcite , none seems to have addressed the potential impact of treating texts as continuous strings of tokens, as they are found in digital format (plain text or the like) instead of treating them as collections of independent textual units within the same file. we note, however, the study of @xcite which explicitly studies two forms of segmentation, plays as a whole and arbitrary segmented plays, and their effect in a bag of words approach. ## evaluation of impact once all combinations have been run, we want to evaluate three different kinds of differences or effects: a first raw effect on lemma co-occurrences and how the raw matrices would differ without any normalization step, the second effect on the selection of secondary classes (major co-occurrences) and finally the effect on more advanced analysis, here using clustering, through the evolution of features.