A multi-level multi-label text classification dataset of 19th century Ottoman and R ussian literary and critical texts (2024)

Abstract: This paper introduces a multi-level, multi-label text classification dataset comprising over 3000 documents. The dataset features literary and critical texts from 19th-century Ottoman Turkish and Russian. It is the first study to apply large language models (LLMs) to this dataset, sourced from prominent literary periodicals of the era. The texts have been meticulously organized and labeled. This was done according to a taxonomic framework that takes into account both their structural and semantic attributes. Articles are categorized and tagged with bibliometric metadata by human experts. We present baseline classification results using a classical bag-of-words (BoW) naive Bayes model and three modern LLMs: multilingual BERT, Falcon, and Llama-v2. We found that in certain cases, Bag of Words (BoW) outperforms Large Language Models (LLMs), emphasizing the need for additional research, especially in low-resource language settings. This dataset is expected to be a valuable resource for researchers in natural language processing and machine learning, especially for historical and low-resource languages. The dataset is publicly available.

## introduction in recent years, there has been remarkable progress in natural language processing with the introduction of large language models @xcite @xcite , and wellcurated, massive pre-training datasets. however, this progress has disproportionately benefited high-resource languages. for low-resource languages, the effectiveness of large language models is compromised by a number of issues including data scarcity that hinders generalization performance; tokenization processes that result in an inadequate representation of word meaning, and the bias in digital texts toward particular topics. to address these issues, there has been an increased focus on developing more inclusive and comprehensive models, introducing diverse datasets, and establishing benchmarks. low-resource languages, often comprising ancient or historical linguistic forms, are a key area of inquiry in computational linguistics characterized by linguistic diversity and evolutionary trajectories over time. however, the archival preservation of written documents poses a notable challenge to computational efforts, particularly in terms of resource availability. this challenge is even present in the case of relatively recent historical languages from the 19 th century, where not only limited textual resources but also non-standardized writing systems hinder computational analysis and linguistic research efforts. with the recent advancements in deep learning research, the effort to digitize and apply machine comprehension on these type of archival resources has become a growing field of inquiry, fostering inter-disciplinary collaborations between humanities, social sciences, and computer science. one example of such endeavor that aims to overcome the limitation of lack of resources, is a large-scale project focusing on aggregation, refinement, and digitizing historical newspapers for the european library foot_2 and europeana foot_3 funded by the european commission @xcite . in the project, several refinement methods were used on the raw content including, but not limited to optical character recognition (ocr), optical layout recognition, and named entity recognition. the project included many languages and contributions from many national libraries having over 16 million items. it largely covered main western european languages. there were, therefore, several limitations for those languages that were out of the scope of the project, which resulted in limitations for refinement techniques for those particular non-european languages (e.g. ottoman) @xcite . in order to address the limitations of digitization and refinement of non-european languages, this paper focuses on historical non-western languages, primarily ottoman and russian literature texts from the 1830s to the 1910s, the most prolific years for periodical publications in the long 19 th century. the rise of mass readership in periodical press roughly corresponds to these dates across all cultures with mass printing presses and (proto) print capitalism @xcite . while russian is not considered a low-resource language, computational efforts, particularly for nineteenthcentury texts, remain limited @xcite . ottoman turkish, on the other hand, presents a unique case. it was a written language distinct from the vernacular, featuring vocabulary and syntactic elements borrowed from persian and arabic @xcite . the orthography was also not standardized, being written in the arabo-persian alphabet with variations in spelling and punctuation. this is in a stark contrast to modern turkish which has undergone systematic phonocentric standardization toward the vernacular and is written in the latin alphabet. therefore, any computational study conducted in modern turkish would have significant limitations in ottoman turkish, as in any non-standardized language @xcite . in this paper, we provide an open-access, multi-label text classification dataset consisting of ottoman and russian literary texts from the 19 th century. the dataset collection was prepared with due diligence from data collection to labeling. we used an ocr pipeline and developed a web-based platform for a team of experts to label the digitized content. our dataset collection has a specific hierarchical structure (see appendix a) and is annotated according to pre-determined categories as either single-label or multi-label. example instances from both the ottoman and russian datasets are given in figure 1 . we expect that these text classification datasets will prove instrumental in advancing future research endeavors. the availability of these datasets will enable researchers to train a wide array of models tailored to specific applications. this includes the automated categorization of digitized resources, offering enhanced efficiency and accuracy in managing digital information. we also note that the russian and ottoman datasets consist mainly of literary and critical texts, which offer a potential for unsupervised learning. ## related work a key milestone behind the recent breakthroughs in nlp is the extensive pre-training datasets. these datasets are collected from digital sources, where western languages, predominantly english, are the primary languages represented. specifically, by january 2024, w3techs estimates that western languages make up 83.1% and english 51.7%, of the websites (w3techs, 2024). in recent years, efforts have been made to bridge this gap by introducing of multilingual and low-resource datasets and developing methods to improve model performance. these studies have targeted different nlp tasks, including machine translation @xcite ), text classification (cruz and cheng, 2020; @xcite @xcite , and part of speech tagging @xcite . efforts have been made to develop new augmentation techniques ( şahin and steedman, 2019) and training strategies @xcite . @xcite proposed an english dataset spanning the late 19 th to late 20 th centuries to examine the shift between poetry and prose resources. cruz and cheng (2020) introduced two datasets containing over 14,000 samples in filipino for binary and multi-label classification tasks. @xcite proposed datasets for nepali-english and sinhala-english translations, which provide benchmarks for evaluating methods trained on low-resource language pairs. mueller et al. (2020) used the bible for machine translation tasks in 1107 languages, creating a multilingual corpus by varying the number and relatedness of source languages. @xcite provided a dataset curated for multi-label classification and a pre-training dataset in tigrinya, a semitic language spoken primarily in eritrea and northern ethiopia. regatte et al. (2020) introduced a dataset for sentiment analysis in telagu, which is one of the most spoken languages in india, but lacks adequate digital resources. @xcite presented a dataset in sumerian, one of the earliest known written languages, and proposed a crosslingual information extraction pipeline. as a low-resource language that is no longer in use, ottoman turkish has been minimally used in nlp research. one such study used traditional machine learning methods to classify poems from the 15 th to 19 th centuries by period and genre @xcite . besides these low-resource dataset proposals and models trained on top of them, there are studies on improving training/finetuning approaches for low-resource languages in a multilingual setting @xcite . in this paper, we present the first expertcurated and labeled dataset in ottoman turkish and russian and propose the first study employing llms for multi-level classification tasks in both languages. ## data collection our dataset collection procedure had two steps: article curation and labeling by human experts. ## dataset analysis our dataset comprises articles from russian and ottoman sources spanning the late 19 th and early 20 th centuries. the structures of datasets have a hierarchical scheme, where the datasets follow 4 levels of categorization. the articles are primarily grouped into 3 categories based on their relationship to literary and cultural phenomena, namely "literary text," "cultural discourse" and "other," which marks the first level of categorization (l1). "other" refers to everything that remains outside of literary or cultural texts, such as news articles, advertisements, and obituaries. literary text and cultural discourse categories also have lower-level categories. there is no lower-level category for the articles in the class "other." the article counts and the names of the subcategories are shown in the categorization goes up to the fourth level of categories, and the tree structure displaying these lower levels is given in within the ottoman dataset, we have a collection of 1,819 articles from 685 journals, with an average of 1,005.29 words, and 43.4 sentences per article. the russian dataset contains 1,058 articles from 198 journals, with an average of 4,630.69 words and 212.26 sentences per article. ## modeling experiments we conducted multiple experiments using ottoman and russian datasets to establish baseline results for classification tasks. ## discussion the experiments show that a simple model like bow+nb can be comparable to prominent models like llama-2 and falcon in a low-resource setting. however, it is important to point out that llama-2 and falcon, which have billions of parameters, underwent training with a frozen backbone and 4-bit integer quantization. this approach can substantially hurt the performance of the models. based on map (weighted), llama-2 is the bestperforming model in both languages in level-1 (l1) tasks, where our dataset has the largest sample size and is well-balanced. for level-2 (l2) and level-3 (l3) tasks, bow+nb outperforms llama-2, falcon, and mbert in ottoman classification tasks; yet in russian, the leading model is still llama-2. in russian, the leading model remains llama-2 in two of the three classification tasks of l3, while bow+nb is the best-performing model in one of the tasks. here, it is important to note that, as the sample size gets smaller, we observe the performance difference between bow+nb and llama-2 decreases. the results show that, in an imbalanced setting, different metrics elect different models. due to the nature of our dataset, we can observe this more clearly at lower-level (l2, l3) classification tasks. at lower-level categories, the classification tasks become more challenging, both in terms of generating a sufficient number of instances and establishing coherent and objective labeling across experts. for example, the subcategories of the literature/ movements labels are open to interpretation, over which experts might disagree. these subcategories, which signify the subject of any critical article on literature, are fluid categories that may overlap in many instances. in our dataset annotation process detailed in section 3.2, we opted against the traditional cross-check annotation, and did not calculate inter-annotator agreement (iaa) scores due to the specific nature of our annotation task. our annotation teams consist of experts proficient in the relevant languages and the historical context of the 18 th and 19 th centuries. unlike tasks such as question answering or sentiment analysis, our categorization process is not inherently subjective but requires specialized expertise. also, comprehensive guidelines and training were provided to annotators, and regular consensus meetings were held to resolve challenging cases. consequently, we implemented a hierarchical cross-check system: initial annotations by a team member were reviewed by a peer expert, and then revised by the team leader, whose domain knowledge surpassed that of the other annotators, overwriting the previous annotations. this structure ensured high-quality annotations and rendered traditional iaa metrics less meaningful. ## conclusion this paper proposes a collection of carefully curated datasets of some prominent historical non-western languages, along with a set of benchmarks obtained from contemporary state-of-the-art llms and the bow naive bayes model. we observed the shortcomings of current leading llms in handling downstream tasks with low-resource languages that are not included in their pre-training data. notably, we also observed a minimal performance difference between bow and llms. this indicates a need for further research on llms. we believe that the democratization of contemporary nlp efforts requires the inclusion of low-resource languages. even if the majority of the population does not use these languages, this line of research can provide significant assistance to historians, linguists, and humanities scholars. and innovation program (grant agreement no. 950513). dr. akbas is supported by the "young scientist awards program (bagep)" of science academy, türkiye. we would like to express our sincere gratitude to the following individuals for their invaluable contributions to this project: mehtap öztürk, zeynep nur şimşek, hazal bozyer, jennifer flaherty, nikita drozdov, ilya kliger and kiril zubkov.