Annotators-in-the-loop: Testing a Novel Annotation Procedure on I talian Case Law (2023)

Abstract: The availability of annotated legal corpora is crucial for a number of tasks, such as legal search, legal information retrieval, and predictive justice. Annotation is mostly assumed to be a straightforward task: as long as the annotation scheme is well defined and the guidelines are clear, annotators are expected to agree on the labels. This is not always the case, especially in legal annotation, which can be extremely difficult even for expert annotators. We propose a legal annotation procedure that takes into account annotator certainty and improves it through negotiation. We also collect annotator feedback and show that our approach contributes to a positive annotation environment. Our work invites reflection on often neglected ethical concerns regarding legal annotation.

## introduction despite the success of self-supervised deep learning approaches @xcite , accurate human annotation remains essential for nlp research, and it is no different for its applications to the legal domain. the increasing availability of corpora of legal documents has given a tremendous boost to legal nlp @xcite , but this comes with serious ethical implications given the potential uses of systems trained on the annotated data (see @xcite for a brief overview). legal annotation is a complex task, where even expert annotators may fail to come to straightforward conclusions @xcite . this warrants particular reflection on the definition of legal annotation guidelines and on making sure they are appropriate and consistently agreed upon among annotators @xcite . to address the aforementioned issues, we present an annotation procedure that involves a group of legal experts in the very process of creating and negotiating the annotation guidelines. we also anonymously collect annotators' feedback and show that our procedure makes them more certain of and satisfied with their work. we believe this to be an important step towards a better treatment of annotators in the field of legal nlp. the italian legal system is currently undergoing significant changes in an effort to digitally transform and overall improve legal processes at all levels. at this stage, gathering high quality data is crucial to make sure that any downstream applications do not perpetuate errors and biases. the annotation procedure we describe is a preliminary step in the framework of the next generation upp (ngupp) project, funded by the italian ministry of justice, and aimed at improving the efficiency of the judicial system in italy. specifically, we intend to empower judges with advanced information management tools to facilitate the drafting of court judgements. such a tool would be used both retroactively, for legal search of case law, and proactively, for the creation of new judgments. the paper is organized as follows. in section 2 we discuss the relevant literature. in section 3 we present the experimental design. section 4 presents the annotation procedure. section 5 is dedicated to the discussion of the results. finally, in section 6 we provide concluding remarks and ideas for future developments. ## related work corpora of legal texts are increasingly available and accessible. this is especially true of legislation @xcite , but it also applies to court judgments @xcite @xcite and other types of legal texts (e.g. con-tracts, @xcite . there have already been several annotation efforts in the legal domain @xcite @xcite , with a particular interest towards arguments (see @xcite for an overview). while some types of annotation are relatively straightforward, obtaining consistent and accurate annotations in law is extremely challenging @xcite . nonetheless, legal annotation tasks often leverage law students as domain experts @xcite @xcite @xcite . we invite caution in using this approach due to a) ethical concerns on adequate annotator compensation and b) difficulty in ascertaining their domain expertise. legal annotation tasks may entail another potentially problematic aspect. it is not uncommon to involve a small group of annotators who initially annotate the same text, which is subsequently revised by a more expert annotator tasked with solving any discrepancies @xcite @xcite . although this is a widely accepted method used to obtain gold standard annotations in the legal domain, we will not be using this technique; rather, we embrace the line of research that sees variation in human annotation as something that may naturally arise due to, e.g., ambiguity, uncertainty of the annotator, genuine disagreement, or simply the fact that multiple options are correct @xcite . specifically, we follow @xcite , who argue that "removing the disagreement might lead to better evaluation scores, but it fundamentally hides the true nature of the task we are trying to solve". to address the aforementioned issues, we propose an annotation procedure that promotes guideline negotiation. previous work on legal annotation has featured modifications of annotation guidelines over time, either in a top-down manner or within small groups @xcite @xcite . @xcite experiment with collaborative guideline creation among pairs of annotators, albeit not in the legal domain. they show that negotiation leads to improved annotator agreement within the pair, but the performance decreases dramatically among annotators of different pairs. our group of annotators was not split into pairs for the negotiation; we are not aware of previous work that frames legal annotation as a peer-to-peer negotiation process among an entire group of legal professionals. in an effort to contribute to a positive annotation environment, we collect feedback from the annotators. following @xcite and @xcite , we collect measures of annotator certainty, checking whether they improve after the negotiation process. we also collect data on the overall satisfaction of the annotators. the dataset we obtain from our annotation will be used for the development of text segmentation models. segmenting court judgments into relevant sections can improve legal search and information retrieval; this has already been investigated by savelka and ashley (2018), @xcite and @xcite . @xcite segment italian civil judgements with simple regular expressions for bench-marking purposes. we operate in the italian legal context, which has been amply explored in previous literature @xcite @xcite . however, our proposed annotation procedure is language-agnostic. ## experimental design this section describes our experimental design, aimed at developing an annotation procedure for the legal domain. we briefly present the dataset, the task, the annotators, the annotation tool, and the agreement metrics. ## annotation scheme in this section we summarize the development of the annotation scheme: first, we describe the initial scheme as designed by a restricted pool of experts; then, we recount its subsequent negotiation; finally, we report the resulting annotation scheme. ## analysis and discussion of the results the results of our work include the annotation scheme as well as the output of the annotation activity. we evaluate inter-annotator agreement from both a quantitative and qualitative perspective and we report annotator feedback. ## conclusion and future work this paper introduces a novel annotation procedure based on the active participation of an entire group of domain experts in the process of creating and negotiating the annotation guidelines. an interdisciplinary research team, involving experts from the legal, linguistic and computer science fields, has actively collaborated in order to address the common issues faced in the annotation of legal documents. the result of this procedure is an annotation scheme tailored to italian case law judgments, which provides a unifying structure to integrate the sections mandated by the law and the ones used in practice by judges. we consider these to be preliminary results in the ongoing development of a reliable procedure that will be extended in future work. we are currently experimenting with the annotation of more fine-grained phenomena: the structure outlined by our annotation scheme serves as the basis for the annotation of legal arguments. since the work presented here is still ongoing, we are unable to release the annotated dataset and the annotation guidelines at present; however, the annotation scheme is presented in our project comes at a crucial time in the process of re-thinking how the judicial system works in italy. the work of law professionals is changing due to the introduction of increasingly sophisticated technological tools. the annotations we collect will be used to build corpora that represent the structure and argumentation of italian court judgments. leveraging segmented case law judgments can improve both keyword-based and semanticbased search of legal precedents. we are actively experimenting with different techniques, including few-shot learning, that can leverage this data to improve the efficiency of legal search. the longterm goal is to integrate these tools into a document builder that supports italian judges in the drafting of court judgments. the annotation of a small set of 50 judgments was used to elaborate, apply and evaluate a novel annotation procedure, capable of taking into account the nuances that the legal subject matter brings, especially when applied to complex cases, while also allowing domain experts to be adequately valued in their specific expertise. discussions on the ethics of legal nlp abound, with emphasis on the data and its potentially harmful uses. while crucial, these discussions would benefit from further reflection on how the data is being annotated. we hope that our results can inspire researchers and practitioners to carefully consider these issues in future work.