Quantifying Contextual Aspects of Inter-annotator Agreement in Intertextuality Research (2021)

Abstract: We report on an inter-annotator agreement experiment involving instances of text reuse focusing on the well-known case of biblical intertextuality in medieval literature. We target the application use case of literary scholars whose aim is to document instances of biblical references in the ‘apparatus fontium’ of a prospective digital edition. We develop a Bayesian implementation of Cohen’s kappa for multiple annotators that allows us to assess the influence of various contextual effects on the inter-annotator agreement, producing both more robust estimates of the agreement indices as well as insights into the annotation process that leads to the estimated indices. As a result, we are able to produce a novel and nuanced estimation of inter-annotator agreement in the context of intertextuality, exploring the challenges that arise from manually annotating a dataset of biblical references in the writings of Bernard of Clairvaux. Among others, our method was able to unveil the fact that the obtained agreement depends heavily on the biblical source book of the proposed reference, as well as the underlying algorithm used to retrieve the candidate match.

## introduction the automatic detection of cases of text reuse in literary collections has the ultimate goal of enabling literary scholars to explore networks of intertextual references between literary works. this goal materializes in more concrete use cases for computationally-aided scholarly work, which include, for instance, visualizing high-level patterns in the referential connections between collections of texts @xcite , or studying the influence foot_0 that a given writer has had on subsequent generations @xcite . more generally, curators of (nowadays more commonly digital) editions of literary works are concerned with making even the more subtle intertextual connections accessible to the contemporary public. in this quest, automated retrieval algorithms play a crucial role, since they can accelerate the task of identification at different retrieval phases. at the beginning, a focus on precision can help editors dealing with the bulk of more obvious cases. towards the end of the process, the majority of the references have been spotted, and high-recall algorithms can help suggesting potential candidates. in this context, the curation of benchmark datasets on which retrieval algorithms can be put to the test is an important milestone of applied research, since benchmark datasets are necessary not only for assessing the relative advantage of particular approaches but also in order to reliably measure the level of precision and recall that editors can expect from automated systems. however, the curation of benchmark datasets depends on reliable annotation processes. two aspects of text reuse studies in literary contexts turn the process of benchmark corpus compilation into a problematic enterprise. the first one is that the assessment of intertextual references is a highly interpretative matter. the second one is that the interpretation of these links demands a specific set of skills and expertise that is scarce. in this context, an important question-which has been however rarely approached in previous researchaddresses the level of agreement that expert anno-tators may reach. still, as it has been noted before @xcite , inter-annotator agreement studies of intertextuality are rare. 2 the present paper starts by addressing such research question, but moves beyond it and further aims towards an examination and understanding of the contextual factors that may affect interannotator agreement in intertextuality research. thus, we not only seek to establish an estimate of the achieved inter-rater agreement, but crucially to investigate contextual aspects in the experimental design that influence the process of agreement and can therefore serve as explanatory variables for the obtained agreement scores. we approach this research by means of a re-formulation of cohen's  for multiple raters @xcite that bases the computation of the observed agreement on a hierarchical statistical model. thanks to the incorporation of the statistical model, we are able to infer the dependency of agreement on a number of factors of variation present in the experimental design. finally, using modern bayesian modeling techniques to fit the statistical model we obtain estimates of agreement that naturally incorporate uncertainty arising from the inferential process. we target the text reuse retrieval context of ongoing efforts towards a digital edition of the sermons written by the influential medieval author bernard of clairvaux (1090-1153). bernard's sermons are known for their pervasive biblical intertextuality and are, therefore, rich in potential cases of reuse. the digital edition is currently in a latestage retrieval phase in which the goal is to exhaustively find relevant but missed cases of reuse. in this context, we examine the effect of the following contextual factors. first, we investigate the influence of the underlying retrieval methodexamining two competitor algorithms from the literature that, a priori, obtain similar performance results but target different aspects of intertextuality. depending on the biases towards particular types of reuse, algorithms may retrieve candidate pairs that are consistently more or less prone to disagreement. secondly, we inspect the effect of the biblical book from which the suggested source verse stems. annotators may be more or less familiar with particular books, and may have diverg2 bär et al. (2012) include an ad-hoc study of interannotator agreement of the annotation guidelines for their evaluation corpus-the wikipedia rewrite corpus @xcite . this corpus, however, contains examples that are hardly related to literary cases of reuse. contributions our contributions are as follows. we present a novel study of the inter-annotator agreement on the task of identifying cases of biblical intertextuality in a real-world scenario. we implement a bayesian variant of a popular interannotator agreement index that allows us to compute robust estimates of agreement in the presence of small sample sizes as well as control for and examine relevant factors of variation. we find that under certain circumstances a semantically motivated text reuse algorithm produces slightly higher inter-annotator agreement scores than an alternative retrieval method based on the text alignment paradigm-which has a bias towards more literal reuse styles. secondly, we statistically inspect additional factors of variation-related to both objective (style of reuse retrieved by the system) and subjective variables (knowledge of the collection from which the passages are borrowed)-that may help explain the obtained agreement scores. specifically, we find that the biblical book from which the source of the reference stems is a significant factor of variation and overshadows the effects of other variables. furthermore, we find that agreement is lowest with average values of lexical overlap in the candidate pair, and that the overall shape of the effect of lexical overlap on agreement diverges depending on whether the bernardine passage was already known to contain a reference or not. finally, we contribute a quantitative assessment of the main hurdles to agreement that our annotators encountered during the experiment, highlighting that not only expert knowledge on the target collections can have important consequences on the assessment of intertextual links, but also that choices in the experimental design may contribute method matches sermons matches/ sermon s-c 296 (72) 24 12.33 (± 10.66) s-w 292 (56) 22 13.27 (± 18.5) ## experimental design in the present section, we explain the underlying resources and methods used in this study. ## conclusion & future work our study has shown how to apply bayesian statistical methods to the computation of inter-annotator agreement indices. on the basis of a multi-level model, we were able to isolate the influence of co-variates on agreement and show how the interannotator agreement scores vary depending on the value of the independent variables. while the overall average inter-annotator agreement reached is fairly high, the amount of uncertainty arising from the source book resulted in very wide posterior distributions which drastically nuance the reported coefficients. our approach fits random intercepts in order to capture individual annotator behaviour and should, therefore, scale reasonably well to higher number of annotators. however, the multinomial approach presented in the current research requires fitting a number of linear models that is quadratic on the number of labels, and, thus, more complex tasks may become unfeasible to model in the same manner. future work should address this shortcoming. moreover, our approach is limited to annotation tasks defined in terms of categorical outcomes, and, thus, ordinal or continuous outcomes would require further research in order to be accommodated. as the post-experimental report highlighted, some of the experimental design choices introduced artificial hurdles to agreement, and future research should take this into account in order to produce a more robust experimental settings. finally, while our study constitutes one of the first dedicated to the problem of inter-annotator agreement in intertextual studies, the specific experimental setting and design should be complemented in the future by other case studies in order to offer a more general picture of the matter. laurence mellerin. 2013. methodological issues in biblindex, an online index of biblical quotations in early christian literature. in laurence mellerin, markus vinzent, and hugh houghton, editors, biblical quotations in patristic texts (papers presented at the sixteenth international conference on patristic studies held in oxford 2011), volume 2 of studia patristica, pages 11-32. peeters. laurence mellerin. 2014. new ways of searching with biblindex, the online index of biblical quotations in early christian literature. in claire clivaz, gregory andrew, and hamidovic david, editors, digital humanities in biblical, early jewish and early christian studies, volume 2 of digital humanities in biblical, early jewish and early christian studies, pages 175-192. brill. william a. scott. 1955. reliability of content analysis: the case of nominal scale coding. public opinion quarterly, 19(3):321. grigori sidorov, alexander gelbukh, helena gómez-adorno, and david pinto. 2014. soft similarity and soft cosine measure: similarity of features in vector space model. computación y sistemas, 18(3):491-504. t.f. smith and m.s. waterman. 1981. identification of common molecular subsequences. journal of molecular biology, 147(1):195-197. aki vehtari, jonah gabry, yuling yao, and andrew gelman. 2018. loo: efficient leave-one-out crossvalidation and waic for bayesian models. r package version, 2(0):1003. aki vehtari, andrew gelman, and jonah gabry. 2017. practical bayesian model evaluation using leaveone-out cross-validation and waic. statistics and computing, 27(5):1413-1432. tariq yousef and stefan janicke. 2021. a survey of text alignment visualization. ieee transactions on visualization and computer graphics, 27(2):1149-1159.