From Dissonance to Insights: Dissecting Disagreements in Rationale Construction for Case Outcome Classification (2023)

Abstract: In legal NLP, Case Outcome Classification (COC) must not only be accurate but also trustworthy and explainable. Existing work in explainable COC has been limited to annotations by a single expert. However, it is well-known that lawyers may disagree in their assessment of case facts. We hence collect a novel dataset RaVE: Rationale Variation in ECHR, which is obtained from two experts in the domain of international human rights law, for whom we observe weak agreement. We study their disagreements and build a two-level task-independent taxonomy, supplemented with COC-specific subcategories. To our knowledge, this is the first work in the legal NLP that focuses on human label variation. We quantitatively assess different taxonomy categories and find that disagreements mainly stem from underspecification of the legal context, which poses challenges given the typically limited granularity and noise in COC metadata. We further assess the explainablility of state-of-the-art COC models on RaVE and observe limited agreement between models and experts. Overall, our case study reveals hitherto underappreciated complexities in creating benchmark datasets in legal NLP that revolve around identifying aspects of a case’s facts supposedly relevant for its outcome.

## introduction the task of case outcome classification (coc) involves classifying the outcome of a case from a textual description of its facts. while high classification performance is of course desirable, its greatest potential utility for legal experts lies in the identification of aspects involved in the case that contribute to the outcome prediction. for example, if these extracted aspects could be grouped 1 our guidelines, dataset and code is available at https: //github.com/tumlegaltech/rave_emnlp23 * these authors contributed equally to this work into patterns, they can be used to index legal analytics databases that inform litigation strategy. in other words, the case outcome signal can serve as a proxy for legal relevance of certain parts of the fact description. a major obstacle in realizing this vision is the misalignment between what models predict from with what experts consider relevant. for instance, prior work by @xcite demonstrates that the alignment between models and experts rationale is fairly limited in the jurisprudence of the european court of human rights (ecthr). previous work @xcite shows that these models are drawn to shallow spurious predictors if not deconfounded. alignment is typically evaluated by deriving a saliency/importance map from the model's predictions on the input text and comparing it with the ground truth labelled by, in most cases, a single expert. aforementioned previous works have derived and evaluated the explanation rationales at the fairly coarse paragraph level, which may over-estimate alignment. in light of this, we introduce rave, a novel explanation dataset of coc rationales from two legal experts at a much more granular and challenging token level. in addition, while many cases involve multiple allegations, previous works typically col-lect only one annotation per case. in contrast, our approach involves collecting annotations for every alleged article under each case, enabling us to capture a more nuanced decision-making process of the annotators. however, we observed weak inter-annotator agreement (iaa) between experts' rationale annotations, indicating the challenge of marking up rationales consistently and emphasizing the perspectivist nature of the legal decisionmaking process (see figure 1 ). there is a growing body of work in mainstream nlp highlighting the presence of irreconcilable variations among annotations, and finding that such disagreement is abundant and plausible @xcite @xcite . researchers further argue that we should embrace such human label variation (hlv) @xcite -signal which provides rich information signal, and not as noise which should be discarded @xcite @xcite . this motivates us to study the disagreements we encountered between our experts on rationales for coc. to the best of our knowledge, this is the first work in the legal domain to systematically study hlv in explanation datasets. we propose a twolevel task-independent taxonomy of disagreement sources and introduce coc task-specific subcategories. we also quantitatively assess the effects of different subcategories on experts' annotation agreement. the results indicate that disagreements are mainly due to the underspecified information about the legal context in the annotation task, highlighting the importance of developing better annotation guidelines and schema in legal domain. further, we assess the state-of-the-art coc models on alignment with respect to obtained experts token-level rationales in various settings. despite differences in coc prediction performance, we observe consistently low alignment scores between models and experts rationales in different settings (fig 1 ), suggesting a need for further investigation on how to make models align its focus better with legal expert rationales. ## expert-model misalignment we evaluate the sota coc models' alignment with the experts' rationales in rave across two paradigms of models: (i) fact-only classification models which rely on case facts description as sole input for the violation outcome @xcite , and (ii) article-aware prediction models which take case facts description along with article text to predict the binary outcome of the case with respect to that article @xcite . ## conclusion in this paper we introduce rave, a novel corpus for coc annotated with rationale variation provided by multiple legal experts, that accounts for variation in rationales between annotators. we study the human label variation and propose a hierarchical, two-layer taxonomy of sources of disagreement. our analysis reveals that annotator discrepancies mainly arise from the underspecified allegation information during annotation. we evaluate the interpretability of current state-of-the-art coc models using rave and find that there is limited consensus between the models and domain experts. our results have important implications, highlighting the intricate nature of coc and emphasize the necessity for interdisciplinary collaboration between legal experts and ml researchers to integrate expertise into trustworthy and explainable coc systems that are aligned with legal principles.