Towards a Clean Text Corpus for O ttoman T urkish (2024)

Abstract: Ottoman Turkish, as a historical variant of modern Turkish, suffers from a scarcity of available corpora and NLP models. This paper outlines our pioneering endeavors to address this gap by constructing a clean text corpus of Ottoman Turkish materials. We detail the challenges encountered in this process and offer potential solutions. Additionally, we present a case study wherein the created corpus is employed in continual pre-training of BERTurk, followed by evaluation of the model’s performance on the named entity recognition task for Ottoman Turkish. Preliminary experimental results suggest the effectiveness of our corpus in adapting existing models developed for modern Turkish to historical Turkish.

## introduction natural language processing (nlp) has been extensively facilitated through widely spoken modern languages. these models cover various fields, from sentiment analysis to medical assessments and question-answering. such applications require extensive data sources, which are relatively more straightforward to collect and optimize for modern languages due to the abundance of digitized documents available on the internet. however, integrating such applications into historical and less commonly spoken languages presents significant challenges due to the need for more available resources in these languages. collecting and optimizing documents in these languages is more complex, necessitating more efficient data extraction methods to achieve comparable performance levels. the development of software tools and the application of automation for historical languages are crucial for scholarly research, offering invaluable insights into political, sociological, and historical contexts. among these languages, ottoman turkish has a significant legacy in history, literature, culture, and science, influencing three continents over 600 years. this extensive historical impact highlights the importance of applications and studies involving ottoman turkish, as they can generate profound value across various fields. one of the pathways to such value creation is through the use of pre-trained language models (plms), which have revolutionized natural language processing by achieving state-of-the-art performance across many tasks. however, the availability of clean text corpora is essential for the automation of any language, as it is necessary for training algorithms to perform tasks such as text analysis @xcite , language modeling @xcite , and information extraction @xcite . this is particularly important for pre-training language models, which require a comprehensive understanding of a language's statistical properties and intricate patterns. therefore, our study aims to create a clean data corpus for the natural language processing of ottoman turkish. despite its importance, integrating ottoman turkish into modern natural language processing (nlp) frameworks presents significant challenges. ottoman texts' unique linguistic and structural characteristics require specialized approaches for effective digitization, standardization, and analysis. leveraging advanced nlp techniques, such as pre-training bert models, has the potential to scale these studies and meet the growing demand for research in this area. while striving to create a clean corpus for ottoman turkish texts, we faced several challenges that impeded effective data collection. these challenges were addressed in four phases: (i) converting pdf documents into clean text files, (ii) normalizing unique characters, (iii) handling intertwined bidirectional text in arabic and latinized turkish, and (iv) minimizing the impact of decorative textures. the solutions needed to be simple and cost-efficient in terms of computational power allocation, adhering to the philosophical principle of occam's razor @xcite , which advocates simplicity. we chose regular expression (regex) methods over competing hypotheses and sophisticated machine learning techniques because regex requires minimal additional memory, in which optimizing computational overhead, and allows for immediate application through modes of non-deterministic finite automata (nfa). this paper summarizes our initial efforts to create a clean text corpus of ottoman turkish texts that can be used for various purposes including automatic processing of ottoman turkish. we mention some related work on data cleaning and extraction in section 2. we explain the methodology followed to create the intended corpus of ottoman turkish texts and state the main challenges faced and possible solutions to them in section 3. then we provide a case study in section 4 where we further pre-train the berturk (schweter, 2020) model using our corpus to adapt it to ottoman turkish texts and fine-tune the model on a named entity recognition (ner) dataset for ner tagging of ottoman turkish. the preliminary experiment results suggest that further pre-training the berturk model, initially designed for modern turkish, with ottoman turkish data is effective for ottoman ner tagging. we conclude the paper and state future directions of this study in section 5. to the best of our knowledge, this study represents the first attempt to provide language resources and models for state-of-the-art natural language processing of ottoman turkish. ## related work modern languages provide relatively clean corpora when obtained from web text sources @xcite . in contrast, historical languages require a different approach due to the variety of digitization methods used by various institutions @xcite . unique challenges, such as non-standard orthography, mixed scripts, and the scarcity of digitized texts, necessitate specialized approaches. this section reviews several studies that have addressed these challenges and contributed to developing effective methods for language processing, which can be utilized in historical document automation. the impresso project @xcite focuses on the semantic indexing of a multilingual corpus of digitized historical newspapers. this interdisciplinary research involves computational linguists, designers, and historians collaborating to transform noisy and unstructured textual content into semantically indexed, structured, and linked data. the authors highlight the challenges posed by large-scale collections of digitized newspapers, including incomplete collections, extensive and messy data, noisy historical text, and the need for robust system architecture. the project emphasizes the importance of transparency and critical assessment of inherent biases in exploratory tools, digitized sources, and annotations. @xcite highlights the importance of text normalization in processing historical languages. the study presents various techniques for handling non-standard orthography, including using historical dictionaries and context-based normalization algorithms. these methods help standardize the text, making it more suitable for nlp tasks. the challenges of interpreting private use area (pua) foot_0 characters and integrating arabic sentences within ottoman turkish texts are addressed through mapping systems and regular expressions. @xcite propose an extensible parsing pipeline to process unstructured data, particularly within network monitoring and diagnostics. their methodology employs rule-based extraction techniques to transform unstructured data into structured formats, utilizing pattern mining strategies and heuristics-based analysis. this approach demonstrates resilience to changes in data structure and effectively filters out extraneous information. notably, the use of regular expressions for pattern recognition and data extraction mirrors the techniques employed in our study for processing ottoman turkish texts. the pipeline's capability to handle diverse data structures without necessitating labeled training data or manual intervention underscores its robustness and efficiency in data extraction tasks. nundloll et al. ( 2021 ) discuss the automation of information extraction from historical texts using the journal of botany as a case study. they document the use of ocr-based software for digitization and the subsequent application of nlp frameworks to customize entity recognition models. tools like prodigy and spacy were employed to identify specific entities such as plant names, observers, locations, and attributes. the authors em-phasize the importance of creating training and test datasets to evaluate the accuracy of the entity recognition models, highlighting the iterative process of model training, error-checking, and annotation. ## methods, challenges, and solutions this study employs a systematic methodology to extract, standardize, and analyze ottoman turkish texts from historical documents. the process involves several key steps and leverages various tools and libraries to ensure efficient and accurate data handling. in the following subsections, we first explain the source of data and then give the two main steps of our method foot_1 for creating a clean corpus of ottoman turkish texts. for each step, we state the challenges faced and explain our solutions to them. ## evaluation on the named entity recognition task as the result of the data extraction and cleaning steps explained in section 3, we created a 17mtoken corpus of ottoman turkish texts. in its current version, this corpus is too small to pre-train a transformer-based language model from scratch. hence, in order to see its effect on a downstream nlp task, we further pre-train a plm which was already pre-trained on various modern turkish corpora. further or continual pre-training is a common approach to train language models for low-resource languages @xcite . by this way, we hope to benefit from the plm's prior knowledge on turkish words and grammatical structures that are common in modern turkish and ottoman turkish. in our preliminary experiments, we observed that among different architectures and models, berturk @xcite , a turkish language model utilizing the bert architecture and pretrained on modern turkish text, reached the highest f1 scores on several nlp tasks. hence, we chose to utilize berturk for our experiments. as in the architecture of the original bert base model, berturk has 12 transformer layers. each transformer layer consists of 12 attention heads and the number of hidden units is 768. the model includes a total of 110 million parameters that are fine-tuned during the pre-training phase on a large corpus of turkish text data. ## conclusion in this study, we presented the first foundations of a clean ottoman turkish text corpus. we discussed the challenges faced in extracting clean text data from documents with complex structures and explained our approaches in handling these challenges. by domain adapting a plm initially created for modern turkish to ottoman turkish using our corpus, we demonstrated the potential to bridge the gap between historical languages and modern nlp frameworks. the preliminary experimental findings highlight the effectiveness of our corpus in enhancing ner tagging for ottoman turkish, showcasing its utility for various nlp applications. our study takes one of the first steps towards providing comprehensive resources for the stateof-the-art natural language processing of ottoman turkish. future research directions may involve expanding the corpus, refining preprocessing techniques, and exploring additional nlp tasks to further enrich the resources available in this historical language.