Tafsir Dataset: A Novel Multi-Task Benchmark for Named Entity Recognition and Topic Modeling in Classical A rabic Literature (2022)

Abstract: Various historical languages, which used to be lingua franca of science and arts, deserve the attention of current NLP research. In this work, we take the first data-driven steps towards this research line for Classical Arabic (CA) by addressing named entity recognition (NER) and topic modeling (TM) on the example of CA literature. We manually annotate the encyclopedic work of Tafsir Al-Tabari with span-based NEs, sentence-based topics, and span-based subtopics, thus creating the Tafsir Dataset with over 51,000 sentences, the first large-scale multi-task benchmark for CA. Next, we analyze our newly generated dataset, which we make open-source available, with current language models (lightweight BiLSTM, transformer-based MaChAmP) along a novel script compression method, thereby achieving state-of-the-art performance for our target task CA-NER. We also show that CA-TM from the perspective of historical topic models, which are central to Arabic studies, is very challenging. With this interdisciplinary work, we lay the foundations for future research on automatic analysis of CA literature.

## introduction all languages deserve equal technologies. named entity recognition (ner) and topic modeling (tm) are a crucial part of various downstream tasks in natural language processing (nlp), such as entity linking, relation extraction, and ultimately question answering. for such tasks, many research institutes and individual scholars put their emphasis on popular, high-resource languages like english, where there is already a large amount of previous work and resources available @xcite @xcite . this definitely accelerates the progress of the ongoing data-driven nlp. however, many historical languages, such as ancient egyptian, ancient greek, and especially classical arabic (ca), which used to be the lingua franca of science and arts, have been mostly neglected by the nlp community. these languages possess large volumes of historical literature (ca: e.g. liber algebrae et almucabola, canon medicinae, tafsir al-tabari), which were and still are to this date relevant for many communities and societies, lay their foundations and even shape their further development. in order to perform historical analysis which are relevant for our modern age, we need to let these forgotten low-resource languages benefit from the wave of machine learning (ml) progress, thus making historical texts accessible to modern studies and approaching ethically an egalitarian state of nlp research. to this end, within the project linked open tafsir @xcite , firstly, we create the tafsir dataset by annotating the ca encyclopedic books of tafsir al-tabari on exegetical studies of law, ethics and philosophy. this is done with respect to span-based nes, sentence-based topics and span-based subtopics, thereby producing over 51,000 sentences and presenting the first multi-task benchmark for ca with three independent tasks. secondly, we develop a novel script compression method for arabic text in order to examine its influence on the performance of neural models (see figure 1 ). for this, we take the modern vocalized arabic script and gradually transform it to its antique form of skeleton script rasm from the 7th century by removing first, the vocalization marks tashkil (consisting of dashes and circles), and second, the diacritic marks i'jam (consisting of dots), thus lowering the vocabulary size drastically by reducing the number of distinct letters from 280 (vocalized) over @xcite to @xcite . from a historical critical perspective, the usage of this skeleton script is quite interesting as this was the first one to be used for documenting the text of the quran. thus, on a side note, by analyzing this ancient script, we shed light on the historical critical question of its readability. thirdly, we analyze our newly generated dataset, apply the leightweight bilstm @xcite and contrast its usage with @xcite , a toolkit for multi-task learning in nlp. this toolkit ideally fits to our multi-task benchmark, allowing us to conduct over 119 many-fold experimental setups with various arabic pre-trained language models (lm), such as arabert, araelectra, rem-bert. with these optimization steps, we produce the first major results for ca-tm and on top establish a state-of-the-art performance for ca-ner by achieving a value of up to 95.58% f1-score. our work facilitates an automatic extraction of theological information so far buried in the bulk of paper manuscripts and volumes. by creating the necessary training data for tackling the task of ner and tm with various ml algorithms, we provide an open-source gold standard for the nlp community and hereby lay the foundations for future work on digitization of historical arabic juridical and theological studies. the remainder of the paper is organized as follows: section 2 reviews related work, section 3 presents the dataset, its historical source and provides details on the annotation tasks and their guidelines, section 4 presents a sketch of the underlying methods, section 5 reports and discusses our results, and, finally, section 6 draw the conclusion. ## related work not much work has been done in the field of nlp for ca as this language suffers from resource poverty in the ml community. for modern standard arabic (msa), there are only a handful of studies and resources open-source available. noteworthy work specifically for msa-ner has been done so far mainly by @xcite on anercorp dataset and by @xcite on aqmar dataset; both datasets along their ner models will be used as baselines here (see ## tafsir dataset: annotation of classical arabic literature in this section, we describe the data source, the textual conversions performed to prepare the annotation task, the annotation guidelines and the annotation process itself. resource platform gawami' al-kalim foot_0 , whose text is the most refined and accurate one according to a review of the linguists in our annotation team. ## results in this section, we present the results which are obtained while utilizing the methods and their setups described in the previous section. the evaluation of the ner predictions are performed by running the official evaluations script from the conll 2003 shared task @xcite on the test set of the tafsir dataset. ## conclusion in this work, we presented the tafsir dataset, the first large-scale multi-task benchmark on ner and tm in classical arabic literature. we demonstrated how useful resources can be for languages which have been historically important but now forgotten by the ongoing nlp research. besides, we also performed a first evaluation of this newly generated dataset. while doing so, we empirically saw that adding a crf layer was beneficial to the transformer-based models, with which we ultimately established a state-of-the-art performance for ca-ner. although tm was not the primary focus of this paper, we generated first results for ca-tm, thereby leaving room for future improvements. this refers to a scenario of tm in which topic labels were originally determined in a historical, normative, exegetical setting, whereas they need to be learned using modern nlp tools, based on their relevance to ca research. such scenarios are likely to be increasingly encountered as more historical languages come into nlp focus. we therefore believe that our benchmark induces a new challenge for the nlp community that can lead to progress for our target low-resource language. the tafsir dataset and its accompanying material are made open-source available for the research community. furthermore, a website foot_2 is published which offers a comprehensive research tool in english and arabic for accessing our dataset in a more user-friendly environment and performing various search queries on it. the web-based tool is freely available and provides over 400 filter options along the categories of our dataset. additionally, it provides the option of graphical visualization (bubble or pie chart) of the dataset and of the query results performed on it. this digital tool makes it possible for scholars from historical and theological fields to access the dataset without any prior technical skill sets, thus allowing them to find systematically the answers to their long-lasting research questions. on a side note, by analyzing the historical skeleton script, we shed light on a centuries-old historical critical question regarding the readability of the rasm text: whether the first quranic manuscripts (i.e. uthmanic codex) can provide a precise reading of the canonized oral text, or whether there is a large amount of ambiguity in it. our scriptdependent analysis shows that from an information retrieval perspective, the usage of the skeleton script is robust enough to deliver a similar performance compared to the usage of the standard script. we can thus conclude that if the ml model is able to deal with the skeleton script, then humans will also not face major difficulties after gaining sufficient training on the same ancient script. future work our work gives indications that script compression seems to be a promising direction to reduce the amount of data and tackle the question of which resource-size actually matters @xcite . in this work, for the case of arabic we came down from 28 to 16 letters while keeping the performance stable. this shows that we do not need (1) vowels, and (2) different letters for each phoneme. in fact, just some minimum amount of consonantal distinction is needed. what is this amount, can we determine it exactly for each target language? phonetic algorithms such as metaphone @xcite pose to be a first language-independent approach, be that as it may, only future work can give us the answers.