Tracing Traditions: Automatic Extraction of Isnads from Classical A rabic Texts (2020)

Abstract: We present our work on automatically detecting isnads, the chains of authorities for a re-port that serve as citations in hadith and other classical Arabic texts. We experiment with both sequence labeling methods for identifying isnads in a single pass and a hybrid “retrieve-and-tag” approach, in which a retrieval model first identifies portions of the text that are likely to contain start points for isnads, then a sequence labeling model identifies the exact starting locations within these much smaller retrieved text chunks. We find that the usefulness of full-document sequence to sequence models is limited due to memory limitations and the ineffectiveness of such models at modeling very long documents. We conclude by sketching future improvements on the tagging task and more in-depth analysis of the people and relationships involved in the social network that influenced the evolution of the written tradition over time.

## introduction in classical arabic texts, lists of the names of authorities that transmitted a piece of information (isnads) are often attached to a statement or report (the matn) to confirm its reliability. the study of isnads is an integral part of the study of hadith and the history of the arabic written tradition in general. with the increasing availability of digitized texts, new methods are required to automatically locate and analyze isnads at scale in a wider variety of text than the canonical hadith collections used in smaller-scale studies @xcite @xcite . isnads are often seamlessly integrated into running text, and are therefore difficult to distinguish from the surrounding text based on visual layout or punctuation information alone. the textual content of the isnad itself must therefore be used to determine its location. for instance, in the example hadith below, the names and transmissive terms indicate that the underlined section in the beginning of the text is the isnad, while the remainder is the matn. long documents often contain passages in an embedded genre, distinct from the surrounding text. we can locate certain kinds of embedded texts using visual layout information to segment the text into smaller units, like articles in a newspaper, which could then be classified using any number of text classification methods (lee et al, 2020), or using formatting and punctuation cues to locate poetic passages embedded in prose works @xcite . if one does not have the ability to divide the text up into sufficiently granular units, or the embedded genre of interest is not easily separable from the surrounding background text using layout cues alone (e.g., indented lines of poetry), these approaches are not possible. however, isnads tend to have linguistic cues that indicate their presence, like the presence of a large number of names or particular transmissive terms in a small region of the text. in this paper we will follow a two-step approach to solving the problem of identifying isnads in long documents by first applying a retrieval model to find sections of a document likely to contain start points of isnads, then training a sequence tagging model on the retrieved sections to identify where within those spans the isnads begin. upcoming work will involve inferring the endpoints of the isnads given the start points identified by the tagging model using a span prediction model. this paper is organized as follows; in section 2 we present an overview of related work. section 3 provides a deeper discussion of the data sources we use for our experiments. in section 4, we present the results of an inter-annotator agreement study done as part of the process of creating training data as a benchmark for human performance on the task. section 5 discusses the models used in our experiments, as well as hyperparameters and training processes. section 6 presents the results of our evaluation. section 7 presents possible avenues of future work. ## related work this work is closely related to the problem of named entity recognition. most named entity recognition models are evaluated using datasets like the conll 2003 dataset @xcite . in contrast to our data, the target entities in such datasets are at most a few words long, the training and test documents are much smaller than the whole texts we are working with, and almost all documents contain named entities. neural models such as the bilstm and bilstm-crf, as used by @xcite have been shown to perform well on a performing named entity recognition in multiple languages. additionally, a great deal of effort has been expended to improve the performance of named entity recognition on short documents like those found on social media sites like twitter (e.g., @xcite . we seek to do the opposite, and explore performance on longer documents. while we could have approached the tasks described above as named entity recognition tasks, looking for names in isnads in particular, downstream tasks like information extraction or network inference would likely benefit from the additional structure present in the text around the names. unlike prior work on automatic ḥadith tagging, we focus exclusively on trying to identify isnads, rather than also trying to simultaneously extract the corresponding matn for each chain @xcite @xcite or extract information from the identified isnads @xcite , as others have done. we are working with a much larger and correspondingly more diverse collection of texts, rather than limiting our analysis to hadith collections. finally, much of the existing work, like that cited above, focuses on using rule-based systems to identify isnads and the individual transmitters within them which limits the generalizability of these models to previously unseen texts. ## data and preprocessing the data for these experiments comes the open islamicate texts initiative (openiti) corpus @xcite a collection of 4,285 transcribed texts in classical arabic collected from digital libraries by scholars from the university of vienna and university of leipzig totaling 1.5 billion words. the individual documents in the openiti corpus are complete texts and are often quite long, with the longest single text containing over 112 million words. the texts in the corpus are largely unpunctuated, and what punctuation there is is a modern editorial intervention, so we have removed punctuation from the texts when it exists. 2 additionally, training any models on punctuated text would harm the performance on the model on the unpunctuated texts that make up the majority of the corpus. since most of the corpus is unpunctuated, it is difficult to break the texts down into smaller units than complete texts. we have also performed orthographic normalization to remove different variants of the same character so that the model is not influenced by the orthographic choices of any particular author. the training dataset we have created consist of data from fifty labeled texts, with 163 distinct tagged regions of text. while isnads are most common in hadith collections, they are also commonly used in historical writing, exegesis (tafsir), geographies, and literature. the specific texts were chosen over the course of several rounds of annotation, in which a crf was trained and tested on existing training data, then the resultant tags in were analyzed by scholars to see where the model tended to fail to properly label isnads. when a weakness was found, new texts were selected that, in the experts' opinion, contain examples that could be used to give the model an understanding of how to avoid such failures in the future. in total, the tagged text consists of 907,110 tokens containing 3,071 isnads. the average length of the isnads in the tagged data is 31.9 tokens. ## inter-annotator agreement study to create training data for this task, we have used an iterative process of human-in-the-loop model training. working in conjunction with experts in the fields of islamic history and religious studies with a strong command of classical arabic, we began by annotating a single text. using these initial annotations, a crf with token features (see section 5) was trained and used to automatically annotate other texts, which were then corrected by the annotators, creating additional training data while being able to use the automatic tags as a starting point. this not only makes annotation easier, as it presents the annotators with the task of modifying pre-existing tags created by the model rather than inserting tags from scratch, but also can be used to create training data that specifically addresses failures by the previous version of the model. as an illustrative example, one version of the model would often mislabel lists of students in a scholar's biography as chains of transmission, likely due to the presence of a high number of names in those regions of the text and a lack of examples to the contrary which would encourage the model not to label such sections as chains of transmission. this annotation process allowed the annotators to recognize that particular kind of language as a failure point of the model and create new training data which the model can use as evidence that those sorts of linguistic constructs are not chains of transmission. if the human annotators did not perform this process with the initial output from the model, it is possible that their annotations would not catch these sorts of cases, resulting in a less useful training dataset. by specifically providing examples of the kinds of data that confuse the model, we end up with a model whose judgements more closely resemble those of humans over several iterations, both as a result of the increased training data and the careful selection of the new text to annotate. to get some understanding of how difficult the task of labeling isnads actually is for humans, we performed an inter-annotator agreement study using data from five of the annotators involved, each of whom annotated the same text, totaling 2000 lines of text across different sections of one work. token level agreement between individual annotators is reported in figure 1a , while the variance in the locations of start and end points of overlapping labeled spans is shown in figure 1b . overall, we see high agreement at the token level, which indicates that the task is not overly difficult for expert annotators, with krippendorf's α of 0.84 indicating very strong agreement across all raters. additionally, when multiple annotators marked overlapping spans, they all agreed on the starting point of the span seventy percent of the time, while the location of the endpoints varied more often, with complete agreement among the annotators occurring only fifty-three percent of the time. however, the vast majority of the disagreements were very small. the outliers in figure 1b were found to be mostly cases where annotators merged two adjacent isnads. this is a rare occurrence in the training data, with around three percent of identified isnads beginning right after the end of an isnad. there are two main points to take from this annotation when evaluating the results of any model attempting to solve this task. first, that identifying the end points is more difficult then identifying the start points for this particular task, and second, that properly segmenting adjacent spans is difficult even for human annotators. we initially experimented with solving this task by sequence labeling alone. these models tag every token in an input document as either the beginning of an isnad ("b"), inside an isnad ("i") or outside an isnad ("o"), as would be done for tasks like named entity recognition. as a baseline for comparison, we train crf models @xcite using counts within a fixed-size window on either side of the word to be tagged as well as the token to be tagged itself as features for the model. thus, if the word "the" occurred twice before the token being tagged, the feature "the_before" would be created with the value 2. we also investigated using unsupported features (features for words that do not occur), but this created a large number of features that made model training too memory-intensive. the intuition behind these features is that the spans of text we seek to identify share common language that these features will enable the model to learn. additionally, this form of model is able to use negative examples provided by the annotation to learn how the context of a word affects its likelihood of membership in a target span. using information about the words around the word it is trying to tag, the model is able to learn how the context of a word affects its meaning, and not naively assume that all instances of the same word should be treated equally. one might try to solve this problem by training a pair of language models using example isnads to train an isnad-specific model and a general background model. using these two models, one could attempt to label entire spans using the ratio of log odds between the two models to decide how to segment a text into isnad and background sections using a viterbi-like inference method rather than making decisions at the level of individual tokens. however, such a model's inability to understand context would result in a model with no ability to determine when common words in the embedded genre should and should not be treated as part of one, giving many false positives as single words common in the embedded genre would often be labeled as part of one, regardless of their context. we also train crfs using 200-dimensional glove embeddings trained on the complete openiti corpus using the default parameters @xcite as features, concatenating the embeddings of the words within a window around the token to be tagged and using that as input to the model. since different authors have different styles of citation, using different terms to indicate transmission, and draw from different sources, thus including different names in their chains, the unseen texts in the test set will contain chains that have a distinct vocabulary from those seen by the model in training. for the surface level token features described above, such mismatches in vocabulary are tokens that the model has no ability to understand the meaning of, as it never encountered them in training. using the distributed representation provided by the glove embeddings, the model can leverage information about tokens with similar distributional semantics (i.e. other names or transmissive terms with similar embeddings) and gain some understanding of the meaning of an unknown token through its similarity to previouslyseen ones, improving the generalizability of the model to previously unseen texts. additionally, for each model, we select a retrieval threshold which optimizes the f2 (recall-weighted f-score) of the resultant model. the default threshold of 0.5 tended to have comparatively low recall at the chunk level, as can be seen in figure 2 , which presents an example precision-recall curve for a retrieval model with 2-gram, tfidf-weighted features and a chunk size of 75. note that chunk level recall (the fraction of chunks that contain isnad beginnings that are retrieved, rather than the fraction of isnad beginnings that are retrieved) is only around 0.2 with the default retrieval threshold of 0.5, while if a threshold around 0.15 were used, recall would be around 0.95 at the cost of only a small drop in precision. this has the added advantage of adding the "attractive distractors" that were mislabeled by the retrieval model to the training and test set for the tagging model, while those chunks that, from the retrieval model's perspective, clearly do not contain the starting point of an isnad are excluded. this results in training and test sets that do not contain the very easy examples that the retrieval model can dismiss, focusing the training data on the more informative examples. other parameter settings exhibit this same trend, albeit with different optimal operating points and upper bounds on recall. for the later experiments with isnad start labeling, we will use a bigram retrieval model with tfidf scores and a chunk size of 75, as it gives the highest upper bound on recall in this corpus, with an optimal threshold of 0.14. 6 evaluation 6.1 tagging results the token-based crf, in addition to detecting start points, also labels tokens as part of an isnad using the "i" tag, as noted above. as such, we can evaluate not just the ability of the model the find starting point, but at identifying entire isnads. it should be noted that all results are given for models using eighty percent of the data for training, and ten percent each as validation and test data. the results for this task can be seen in ## future work as noted above, the most immediate direction for future research is finishing the process of locating complete isnads in the hybrid model regime. additional work will also be done to improve the performance of the isnad start point tagging model. it may be that additional features such as a gazetteer of common transmissive terms or the output of a named entity recognition system may improve the model's ability to detect isnads. it might also be worthwhile to look into modifying the retrieval model used in the above experiments to use more generalizable document representations than simple bag of ngram representations, which may generalize better to unseen texts. additionally, once we can reliably identify isnads in text at scale, several new research questions about the social networks these chains represent become feasible, since isnads can be thought of as textual descriptions of a social network of individuals involved in the dissemination of a given piece of information. a reasonable first step would be to take a single isnad and extract the names of the individuals involved and the relationship between them. with that accomplished, we could then attempt to infer the social network in which the information is spread by linking names to the individuals they represent using the evidence provided by multiple isnads to differentiate between individuals when, for instance, one name is shared by multiple people or different names are used to refer to the same individual, essentially creating a social map of the written tradition's evolution over time. to this end, it will be useful to leverage information from biographical dictionaries and other sources of data about the relationships between individuals involved in transmitting the knowledge present in the corpus.