VECHR : A Dataset for Explainable and Robust Classification of Vulnerability Type in the E uropean Court of Human Rights (2023)

Abstract: Recognizing vulnerability is crucial for understanding and implementing targeted support to empower individuals in need. This is especially important at the European Court of Human Rights (ECtHR), where the court adapts Convention standards to meet actual individual needs and thus to ensure effective human rights protection. However, the concept of vulnerability remains elusive at the ECtHR and no prior NLP research has dealt with it. To enable future research in this area, we present VECHR, a novel expert-annotated multi-label dataset comprising of vulnerability type classification and explanation rationale. We benchmark the performance of state-of-the-art models on VECHR from both prediction and explainability perspective. Our results demonstrate the challenging nature of task with lower prediction performance and limited agreement between models and experts. Further, we analyze the robustness of these models in dealing with out-of-domain (OOD) data and observe overall limited performance. Our dataset poses unique challenges offering a significant room for improvement regarding performance, explainability and robustness.

## introduction vulnerability encompasses a state of susceptibility to harm, or exploitation, particularly among individuals or groups who face a higher likelihood of experiencing adverse outcomes due to various factors such as age, health, disability, or marginalized social position @xcite . while it is impossible to eliminate vulnerability, society has the capacity to mitigate its impact. the european court of human rights (ecthr) interprets the european convention of human rights (echr) to address the specific contextual needs of individuals and provide effective protection. this is achieved through various means, such as displaying flexibility in admissibility issues, and shifting the burden of proof @xcite . however, the concept of vulnerability remains elusive within the ecthr. while legal scholars have explored vulnerability as a component of legal reasoning @xcite , empirical work in this area remains scarce and predominantly relies on laborious manual processes. to address this challenge, nlp can offer valuable tools to assist experts in efficiently classifying and analyzing textual data. besides high classification performance, the true utility of nlp in the legal field is its ability to identify relevant aspects related to vulnerability in court cases. these aspects can be extracted, grouped into patterns, and used to inform both litigation strategy and legal policy. even so, a significant obstacle to progress in this area is the lack of appropriate datasets. to bridge these research gaps, we present the dataset vechr foot_0 , which comprises cases dealing with allegation of article 3 "prohibition of torture" and is obtained from legal expert's empirical study foot_1 . our proposed ## vulnerability typology in ecthr the inescapable and universal nature of vulnerability, as posited by @xcite , underscores its significance in legal reasoning. for instance, the european union has acknowledged the concept by establishing a definition for vulnerable individuals @xcite . however, it remains undefined within the context of ecthr. to facilitate an examination of vulnerability and its application within the ecthr, it is crucial to establish a typology recognized by the court. several scholars have endeavored to effectively categorize vulnerability for this purpose @xcite . one notable study is conducted by @xcite , which provides a systematic and comprehensive examination of the concept of vulnerability under echr article 3. heri proposes a complete typology encompassing eight types: dependency, state control, victimization, migration, discrimination, reproductive health, unpopular views and intersections thereof. tab 1 gives a description for each type. ## dataset analysis tab 2 presents the key statistics of our dataset. vechr comprises a total of 1,070 documents, with an average of 4,765 tokens per case (σ = 4167). 788 and 282 cases fall under the article 3 and non-article 3 partitions, respectively. among all, 530 documents are considered as "nonvulnerable", meaning they are not labelled as any of the seven vulnerable types. in the vulnerablerelated cases, the average number of labels assigned per document is 1.54. we observe a strong label distribution imbalance within the dataset. the label "state control" dominates, accounting for 33% of the cases, while the least common label, "reproductive health", is present in only 3% of the cases. for more detailed statistics of our dataset, including details regarding the label imbalances in tab 6, please refer to app i. ## conclusion we present vechr, an ecthr dataset consisting of 1,070 cases for vulnerability type classification and 40 cases for token-level explanation. we also release a set of baseline results, revealing the challenges of achieving accuracy, explainability, and robustness in vulnerability classification. we hope that vechr and the associated tasks will provide a challenging and useful resource for legal nlp researchers to advance research on the analysis of vulnerability within ecthr jurisprudence, ultimately contributing to effective human rights protection.