Examining Language Modeling Assumptions Using an Annotated Literary Dialect Corpus (2024)

Abstract: We present a dataset of 19th century American literary orthovariant tokens with a novel layer of human-annotated dialect group tags designed to serve as the basis for computational experiments exploring literarily meaningful orthographic variation. We perform an initial broad set of experiments over this dataset using both token (BERT) and character (CANINE)-level contextual language models. We find indications that the “dialect effect” produced by intentional orthographic variation employs multiple linguistic channels, and that these channels are able to be surfaced to varied degrees given particular language modelling assumptions. Specifically, we find evidence showing that choice of tokenization scheme meaningfully impact the type of orthographic information a model is able to surface.

## introduction orthographic variation, the deviation from one system of spelling in favor of another, occurs due to a range of intentional and unintentional motivations. unintentional variation may occur when a writer misspells a word relative to their intended system, or when an optical character recognition system misidentifies a particular character. intentional deviations are instead used to create a desired political or literary effect @xcite . for example, adhering to a system of simplified spelling may signal one's dedication to egalitarian politics, while embedding a literary character's speech in a particular orthographic form may signal an authorial desire to present that character as belong to a particular race, class, region or gender @xcite . this latter class of intentional variations proves especially diverse. supported by the availability of surrounding context and reader-familiar stereotypes of speech, literary orthographic edits are frequently unsystematic ("eye dialect") or not fully beholden to phonetics or morphology @xcite . instead, the means by which they convey a "dialect effect" is likely multidimensional. we present a dataset that includes a novel humanannotated layer of dialect family tags designed to support investigations into these varied signalling pathways. we perform an initial set of experiments and discover indications that literary orthographic variation communicates its dialect effect by modifying information along multiple axes: word-level semantics, context-level semantics, and character edits. in the spirit of previous work investigating the phonetic @xcite , semantic @xcite and contextual @xcite information token and character level models capture, we also provide analysis of the literary orthographic understanding of these model types. we additionally offer evidence that character-level models distinguish between intentional literary orthovariants and constructed unintentional variants. ## conclusions and further work these experiments offer indications that the dialect effect presented by literary orthographic variation utilizes multiple channels of information: contextual semantics, word semantics and character edits. they also offer evidence that while both contextual token and character level language models can capture all of these aspects, they do so unevenly, justifying further work on the best combination of their information streams.