Dependency Annotation of O ttoman T urkish with Multilingual BERT (2024)

Abstract: This study introduces a pretrained large language model-based annotation methodology of the first dependency treebank in Ottoman Turkish. Our experimental results show that, through iteratively i) pseudo-annotating data using a multilingual BERT-based parsing model, ii) manually correcting the pseudo-annotations, and iii) fine-tuning the parsing model with the corrected annotations, we speed up and simplify the challenging dependency annotation process. The resulting treebank, that will be a part of the Universal Dependencies (UD) project, will facilitate automated analysis of Ottoman Turkish documents, unlocking the linguistic richness embedded in this historical heritage.

## introduction the ottoman turkish is one of the historical versions of turkish. it was in use for nearly six centuries before it evolved into modern turkish. during that period, it went through many changes in terms of lexicon and, to some extent, syntax. there is an increasing trend in digitization campaigns that aim to preserve historical documents and facilitate user access to these resources. numerous institutions publicly share images of their historical documents written in ottoman turkish. as these historical materials become more accessible in digital form, a crucial need for automated processing and retrieval of their content arises. however, unlike modern languages with well-established linguistic resources and corpora, ottoman turkish has quite a limited availability of annotated data, dictionaries, and linguistic references. that scarcity of resources hinders the development and training of robust nlp models for the ottoman turkish. a possible solution for the problem of resource scarcity can be leveraging turkish resources based on the fact that they are actually two versions of the same language. however, the linguistic evolution from ottoman turkish to modern turkish involves substantial changes over time. this transition impacts semantic nuances, vocabulary, and grammatical structures, making it challenging to apply contemporary turkish nlp techniques directly to historical texts. it is a common approach to prepare the necessary resources manually in the case of low-resource languages. however, manual tagging and annotation are both time-consuming and expensive. in the case of dependency annotation, there is also a need for experts specialized in the task at hand. to simplify and speed up the manual annotation process for our objective of constructing an ottoman turkish dependency treebank, we propose to use pretrained large language models (llms) as pseudo-annotators. llms have proven themselves as powerful tools widely used in various fields due to their adaptability. they excel in understanding natural language, making them valuable across diverse areas, from downstream tasks like machine translation @xcite and sentiment analysis @xcite to data annotation @xcite . we experiment with a multilingual bert-based parsing model for dependency annotation in our treebank. our proposed methodology suggests generating pseudo-annotations using the parsing model, then correcting them manually and fine-tuning the model with the corrections, and then using the updated model to pseudo-annotate new data in a repeated cycle. in this study, we present our llm-enhanced syntactic annotation methodology for the first ottoman turkish dependency treebank, which will be a part of the universal dependencies (ud) project @xcite . we investigate the impact of enhancing the accuracy of pseudo-annotations by finetuning the parsing model with labeled ottoman turkish data. we show that utilizing a multilingual bert-based parsing model in dependency anno-tation leads to better pseudo-annotations, which eases the manual annotation of the ottoman turkish treebank job for human annotators. with the creation of this treebank, we aim at facilitating the automatic analysis and understanding of ottoman turkish documents, thereby facilitating a more comprehensive exploration of this linguistically rich heritage. ## background and related work we present some background on the ottoman turkish and discuss its differences from modern turkish in this section. the section includes a summary of the related works from the literature as well. ## ota-boun: a ud treebank for ottoman turkish as an initial endeavor towards enriching ottoman turkish natural language processing (nlp) resources and enhancing the effectiveness of existing nlp tools on ottoman turkish texts, we have initiated the creation of the ottoman turkish ud treebank. in section 3.1, we provide information about the source and language of the treebank. section 3.2 details our annotation process, and section 3.3 states the challenges encountered in annotating ottoman turkish sentences, accompanied by illustrative examples, and clarifies the approaches undertaken to overcome these challenges. ## iterative annotation with multilingual bert because dependency annotation is both cost-and time-intensive, and given the challenges outlined in the previous section, we recently contemplated whether we could ease our manual annotation phase by leveraging state-of-the-art large language models (llms). to test this idea, we conducted an experiment where we utilized the multilingual bert model foot_5 @xcite . the experiment is set up as follows: figure 4 depicts this iterative annotation scheme for two iterations. thus, in this initial phase of annotation, we have 100 manually annotated ottoman turkish sentences in two iterative batches of 50 sentences each. we should note that, because of the difficulty in finding parallel ottoman turkish and modern turkish data, the sentences in these two batches originated from different sources and different genres. this has led to variations in the annotation complexity of the two batches due to the diversity in their origins. with the data obtained in this manner, we have the opportunity to address our research question: has fine-tuning the multilingual bert model with ottoman turkish data facilitated the manual annotation and correction phase? to answer this question, we did the experiment using the two batches of annotated ottoman turkish sentences.