Detecting Structured Language Alternations in Historical Documents by Combining Language Identification with F ourier Analysis (2024)

Abstract: In this study, we present a generalizable workflow to identify documents in a historic language with a nonstandard language and script combination, Armeno-Turkish. We introduce the task of detecting distinct patterns of multilinguality based on the frequency of structured language alternations within a document.

## introduction this work emerges from the goal to create a corpus in armeno-turkish-vernacular turkish written in armenian script. this historic language was actively used from the early 18th century to the early 20th century in a variety of locations in the middle east, europe and the us, including istanbul, venice, vienna and boston @xcite . there are lists of works in armeno-turkish available @xcite , but searching hathitrust manually points to the existence of works that are not recorded in these lists due to challenges in the bibliographical recording of these works in library catalogues (missing titles in the original script, wrong or missing language labels). unable to collate works in armeno-turkish by using the metadata, we used language identification. this process did not produce a clean dataset in armeno-turkish, but showed that a significant portion of these works are multilingual. based on this observation, this study is a first attempt at modeling some of the interesting multilingual phenomena with structured language alternations that emerge in this process: bilingual translations, dictionaries, original-language text followed by commentary in a different language, language study books. as opposed to unstructured code switching (oral interviews), structured multilingual patterns involve an organized alternation of two or more languages. these language alternations may occur at different frequencies (every sentence, paragraph, every page, every chapter). a structured language alternation may serve various purposes, including making the content available in multiple languages in a legal document to reach its target audiences, as shown in the two-column ottoman legal code in armenian and armeno-turkish in figure 1 . especially for historic languages, detecting structured language alternations is valuable for identifying clean monolingual segments which can then be used to create new resources for nlp tasks. this analysis also provides insight into material history of physical books and translation studies, by showcasing different formats of page segmentation in structured multilingual books (bi-column, top-bottom) @xcite . this project introduces the task of detecting structured language alternations and makes the following contributions: ## results and discussion the clustering of the frequency spectra yielded three coherent groups: this experiment identified 30 new records in armeno-turkish that were not in the training set, including translations of the bible, dictionaries, textbooks for learning foreign languages, and legal documents.