Paragraph-level Rationale Extraction through Regularization: A case study on E uropean Court of Human Rights Cases (2021)

Abstract: Interpretability or explainability is an emerging research field in NLP. From a user-centric point of view, the goal is to build models that provide proper justification for their decisions, similar to those of humans, by requiring the models to satisfy additional constraints. To this end, we introduce a new application on legal text where, contrary to mainstream literature targeting word-level rationales, we conceive rationales as selected paragraphs in multi-paragraph structured court cases. We also release a new dataset comprising European Court of Human Rights cases, including annotations for paragraph-level rationales. We use this dataset to study the effect of already proposed rationale constraints, i.e., sparsity, continuity, and comprehensiveness, formulated as regularizers. Our findings indicate that some of these constraints are not beneficial in paragraph-level rationale extraction, while others need re-formulation to better handle the multi-label nature of the task we consider. We also introduce a new constraint, singularity, which further improves the quality of rationales, even compared with noisy rationale supervision. Experimental results indicate that the newly introduced task is very challenging and there is a large scope for further research.

## introduction model interpretability (or explainability) is an emerging field of research in nlp @xcite . from a model-centric point of view, the main focus is to demystify a model's inner workings, for example targeting selfattention mechanisms @xcite , and more recently transformer-based language models @xcite @xcite . from a user-centric point of view, the main focus is to build models that learn to provide proper correspondence to: ihalk.aueb.gr justification for their decisions, similar to those of humans, @xcite @xcite by requiring the models to satisfy additional constraints. here we follow a user-centric approach to rationale extraction, where the model learns to select a subset of the input that justifies its decision. to this end, we introduce a new application on legal text where, contrary to mainstream literature targeting word-level rationales, we conceive rationales as automatically selected paragraphs in multiparagraph structured court cases. while previous related work targets mostly binary text classification tasks @xcite , our task is a highly skewed multi-label text classification task. given a set of paragraphs that refer to the facts of each case (henceforth facts) in judgments of the european court of human rights (ecthr), the model aims to predict the allegedly violated articles of the european convention of human rights (echr). we adopt a rationalization by construction methodology @xcite @xcite , where the model is regularized to satisfy additional constraints that reward the model, if its decisions are based on concise rationales it selects, as opposed to inferring explanations from the model's decisions in a post-hoc manner @xcite @xcite . legal judgment prediction has been studied in the past for cases ruled by the european court of human rights @xcite @xcite and for chinese criminal court cases @xcite @xcite , but there is no precedent of work investigating the justification of the models' decisions. similarly to other domains (e.g., financial, biomedical), explainability is a key feature in the legal domain, which may potentially improve the trustworthiness of systems that abide by the principle of the right to explanation (goodman and flaxman, 2017). we investigate the explainability of the decisions of state-of-the-art models, comparing the paragraphs they select to those of legal professionals, both litigants and lawyers, in alleged violation prediction. in the latter task, introduced in this paper, the goal is to predict the accusations (allegations) made by the applicants. the accusations can be usually predicted given only the facts of each case. by contrast, in the previously studied legal judgment prediction task, the goal is to predict the court's decision; this is much more difficult and vastly relies on case law (precedent cases). although the new task (alleged violation prediction) is simpler than legal judgment prediction, models that address it (and their rationales) can still be useful in the judicial process (fig. 1 ). for example, they can help applicants (plaintiffs) identify alleged violations that are supported by the facts of a case. they can help judges identify more quickly facts that support the alleged violations, contributing towards more informed judicial decision making @xcite . they can also help legal experts identify previous cases related to particular allegations, helping analyze case law @xcite . our contributions are the following: we also introduce a new constraint, singularity, which further improves the rationales, even compared with silver (noisy) rationale supervision. legal judgment prediction has also been studied in chinese criminal cases @xcite @xcite . similarly to the literature on legal judgment prediction for ecthr cases, the aforementioned approaches ignore the crucial aspect of justifying the models' predictions. given the gravity that legal outcomes have for individuals, explainability is essential to increase the trust of both legal professionals and laypersons on system decisions and promote the use of supportive tools @xcite . to the best of our knowledge, our work is the first step towards this direction for the legal domain, but is also applicable in other domains (e.g., biomedical), where justifications of automated decisions are essential. rationale extraction by construction: contrary to earlier work that required supervision in the form of human-annotated rationales @xcite , @xcite introduced a self-supervised methodology to extract rationales (that supported aspect-based sentiment analysis predictions), i.e., gold rationale annotations were used only for evaluation. furthermore, models were designed to produce rationales by construction, contrary to work studying saliency maps (generated by a model without explainability constraints) using gradients or perturbations at inference time @xcite @xcite . @xcite aimed to produce short coherent rationales that could replace the original full texts, maintaining the model's predictive performance. the rationales were extracted by generating binary masks indicating which words should be selected; and two additional loss regularizers were introduced, which penalize long rationales and sparse masks (that would select non-consecutive words). @xcite proposed another constraint to ensure that the rationales would contain all the relevant information. they formulated this constraint through a minimax game, where two players, one using the predicted binary mask and another using the complement of this mask, aim to correctly classify the text. if the first player fails to outperform the second, the model is penalized. @xcite use a generative adversarial network (gan) @xcite , where a generator producing factual rationales competes with a generator producing counterfactual rationales to trick a discriminator. the gan was not designed to perform classification. given a text and a label it produces a rationale supporting (or not) the label. @xcite decoupled the model's predictor from the rationale extractor to produce inherently faithful explanations, ensuring that the predictor considers only the rationales and not other parts of the text. faithfulness refers to how accurately an explanation reflects the true reasoning of a model @xcite . all the aforementioned work conceives rationales as selections of words, targeting binary classification tasks even when this is inappropriate. for instance, deyoung et al. (2020) and @xcite over-simplified the task of the multipassage reading comprehension (multirc) dataset @xcite turning it into a binary classification task with word-level rationales, while sentence-level rationales seem more suitable. responsible ai: our work complies with the ecthr data policy. by no means do we aim to build a 'robot' lawyer or judge, and we acknowledge the possible harmful impact @xcite of irresponsible deployment. instead, we aim to support fair and explainable ai-assisted judicial decision making and empirical legal studies. we consider our work as part of ongoing critical research on responsible ai @xcite that aims to provide explainable and fair systems to support human experts. 3 the new ecthr dataset the court (ecthr) hears allegations regarding breaches in human rights provisions of the european convention of human rights (echr) by european states (fig. 1 ). 3 the court rules on a subset of all echr articles, which are predefined (alleged) by the applicants (plaintiffs). our dataset comprises 11k ecthr cases and can be viewed as an enriched version of the ecthr dataset of @xcite , which did not provide ground truth for alleged article violations (articles discussed) and rationales. the new dataset includes the following: facts: each judgment includes a list of paragraphs that represent the facts of the case, i.e., they describe the main events that are relevant to the case, in numbered paragraphs. we hereafter call these paragraphs facts for simplicity. note that the facts are presented in chronological order. not all facts have the same impact or hold crucial information with respect to alleged article violations and the court's assessment; i.e., facts may refer to information that is trivial or otherwise irrelevant to the legally crucial allegations against defendant states. allegedly violated articles: judges rule on specific accusations (allegations) made by the applicants @xcite . in ecthr cases, the judges discuss and rule on the violation, or not, of specific articles of the convention. the articles to be discussed (and ruled on) are put forward (as alleged article violations) by the applicants and are included in the dataset as ground truth; we identify 40 violable articles in total. 4 in our experiments, however, the models are not aware of the allegations. they predict the convention articles that will be discussed (the allegations) based on the case's facts, and they also produce rationales for their predictions. models of this kind could be used by potential applicants to help them formulate future allegations (articles they could claim to have been violated), as already noted, but here we mainly use the task as a test-bed for rationale extraction. violated articles: the court decides which allegedly violated articles have indeed been violated. these decisions are also included in our dataset and could be used for full legal judgment prediction experiments @xcite . however, they are not used in the experiments of this work. silver allegation rationales: each decision of the ecthr includes references to facts of the case (e.g., "see paragraphs 2 and 4.") and case law (e.g., "see draci vs. russia (2010)."). we identified references to each case's facts and retrieved the corresponding paragraphs using regular expressions. these are included in the dataset as silver allegation rationales, on the grounds that the judges refer to these paragraphs when ruling on the allegations. gold allegation rationales: a legal expert with experience in ecthr cases annotated a subset of 50 test cases to identify the relevant facts (paragraphs) of the case that support the allegations (alleged article violations). in other words, each identified fact justifies (hints) one or more alleged violations. 5 task definition: in this work, we investigate alleged violation prediction, a multi-label text classification task where, given the facts of a ecthr case, a model predicts which of the 40 violable echr articles were allegedly violated according to the applicant(s). 4 the model also needs to identify the facts that most prominently support its decision. ## methods we first describe a baseline model that we use as our starting point. it adopts the framework proposed by @xcite , which generates rationales by construction: a text encoder sub-network reads the text; a rationale extraction sub-network produces a binary mask indicating the most important words of the text; and a prediction sub-network classifies a hard-masked version of the text. we then discuss additional constraints that have been proposed to improve word-level rationales, which can be added to the baseline as regularizers. we argue that one of them is not beneficial for paragraphlevel rationales. we also consider variants of previous constraints that better suit multi-label classification tasks and introduce a new one. ## experimental setup for all methods, we conducted grid-search to tune the hyper-parameters λ * . we used the adam optimizer (kingma and ba, 2015) across all experi-ments with a fixed learning rate of 2e-5. 7 all methods rely on legal-bert-small @xcite , a variant of bert @xcite , with 6 layers, 512 hidden units and 8 attention heads, pre-trained on legal corpora. based on this model, we were able to use up to 50 paragraphs of 256 words each in a single 32gb gpu. in preliminary experiments, we found that the proposed model relying on a shared paragraph encoder, i.e., one that passes the same context-aware paragraph representations p we evaluate: (a) classification performance, (b) faithfulness (section 2), and (c) rationale quality, while respecting a given sparsity threshold (t ). classification performance: given the label skewness, we evaluate classification performance using micro-f1, i.e., for each convention article, we compute its f1, and micro-average over articles. faithfulness: recall that faithfulness refers to how accurately an explanation reflects the true reasoning of a model. to measure faithfulness, we report sufficiency and comprehensiveness @xcite . sufficiency measures the difference between the predicted probabilities for the gold (positive) labels when the model is fed with the whole text ( y + f ) and when the model is fed only with the predicted rationales ( y + ). comprehensiveness (not to be confused with the homonymous loss of eq. 3-5) measures the difference between the predicted probabilities for the gold (positive) labels obtained when the model is fed with the full text ( y + f ) and when it is fed with the complement of the predicted rationales ( y + c ). we also compare classification performance (again using micro-f1) in both cases, i.e., when considering masked inputs (using z) and complementary inputs (using z c ). rationale quality: faithful explanations (of system reasoning) are not always appropriate for users @xcite , thus we also evaluate rationale quality from a user perspective. the latter can be performed in two ways. objective evaluation compares predicted rationales with gold annotations, typically via recall, precision, f1 (comparing system-selected to human-selected facts in our case). in subjective evaluation, human annotators review the extracted rationales. we opt for an objective evaluation, mainly due to lack of resources. as rationale sparsity (number of selected paragraphs) differs across methods, which affects recall, precision, f1, we evaluate rationale quality with mean r-precision (mrp) @xcite . that is, for each case, the model ranks the paragraphs it selects by decreasing confidence, and we compute precision@k, where k is the number of paragraphs in the gold rationale; we then average over test cases. for completeness, we also report f1 (comparing predicted and gold rationale paragraphs), although it is less fair, because of the different sparsity of different methods, as noted. 6 experimental results ## conclusions and future work we introduced a new application of rationale extraction in a new legal text classification task concerning alleged violations on ecthr cases. we also released a dataset for this task to foster further research. moreover, we compared various rationale constraints in the form of regularizers and introduced a new one (singularity) improving faithfulness and rationale quality in a paragraph-level setup comparing both with silver and gold rationales. in the future, we plan to investigate more constraints that may better fit paragraph-level rationale extraction and explore techniques to de-bias models and improve rationale quality. paragraph-level rationale extraction can be also conceived as selfsupervised extractive summarization to denoise long documents, a direction we plan to explore in the challenging task of case law retrieval @xcite . to infer mistrial, because there was is verdict after a reasonable period of time. instead the model seems to be sensitive to references for the involvement of higher courts as justification of mistrial (paragraphs 10, 13, 18, and 21). this suggests that the model probably follows poor (greedy) reasoning, i.e., if the applicant appealed to higher courts, then the case is mistreated. 6 , the main argument in this case is mistrial because there was a verdict after a reasonable period of time (paragraphs 5 and 18-19). the model selected paragraph 11, which does not indicate plausible violations. given the model's prediction for allegations with respect to article 1 of the 1st protocol on the protection of property, we believe that paragraph 11 was selected by the model as justification on that matter.