Can A bstract M eaning R epresentation Facilitate Fair Legal Judgement Predictions? (2024)

Abstract: Legal judgment prediction encompasses the automated prediction of case outcomes by leveraging historical facts and opinions. While this approach holds the potential to enhance the efficiency of the legal system, it also raises critical concerns regarding the perpetuation of biases. Abstract Meaning Representation has shown promise as an intermediate text representation in various downstream NLP tasks due to its ability to capture semantically meaningful information in a graph-like structure. In this paper, we employ this ability of AMR in the legal judgement prediction task and assess to what extent it encodes biases, or conversely, abstracts away from them. Our study reveals that while AMR-based models exhibit worse overall performance than transformer-based models, they are less biased for attributes like age and defendant state compared to gender. By shedding light on these findings, this paper contributes to a more nuanced understanding of AMR’s potential benefits and limitations in legal NLP.

## introduction transformer-based language models such as bert, t5, and gpt have ushered in a new era in nlp. these language models have demonstrated exceptional proficiency in comprehending text with their non-trivial degree of knowledge in every field, propelling them to the forefront of various languagerelated domains @xcite . however, despite their impressive performance, language models still face challenges in dealing with contextdependent language, biases in data, and a lack of interpretability @xcite . such limitations make them unsuitable for domains like legal nlp, which have an abundance of complicated, lengthy, and contextual legal documents. thus, a system that can capture the intricate semantics of these documents is needed. semantic representation frameworks have proven to be a promising this paper conducts a theoretical analysis of abstract meaning representation, scrutinizing its potential in the realm of law. more concretely, it investigates the critical question of whether amr can help produce fair legal decisions and reports potential biases that may arise from its use. this evaluation helps us determine if amr is a suitable intermediate representation for legal judgement prediction. we conduct our experiments on the ecthr dataset (see section 4.1), a benchmark for legal judgement prediction which has been annotated with demographic and diversity labels. it proves to be a primary choice due to its inclusion of these labels for fairness evaluation. we utilize the macro f1 score as an evaluation metric for our experiments. ## related work while previous research has predominantly focused on amr parsing of legal documents @xcite @xcite , limited attention has been given to assessing amr's performance in legal tasks. a study by @xcite explores amr's ability to identify logical relationships in legal mcqa tasks, revealing challenges posed by amr parsing. in contrast, our work is the first to investigate whether amr representations capture social biases alongside linguistic information, emphasizing the need to scrutinize amr input representations for potential biases in legal judgement prediction tasks. research on fairness in machine learning models within the legal domain has also been limited. previous studies @xcite @xcite @xcite @xcite have highlighted racial and gender biases in the legal system and language models. more recently, @xcite introduced the fair-lex benchmark to assess the fairness of language models. in our study, we leverage one of these datasets to examine whether amr-based models can effectively mitigate bias, addressing the critical issue of bias reduction in legal language processing. 1 https://github.com/supritivĳay /amr-for-legal-ai. ## abstract meaning representation abstract meaning representation is a structured framework that utilizes graph-like structures to represent sentence meaning, ensuring interpretability for machines and humans. these graphs, conforming to rooted, directed, and acyclic properties, are independent of semantics, grounded in syntax, and annotated using penman notation for textual representation. for example, the sentence "mr. t was born in 1949 in country a. he robbed mr. j.", as shown in figure 1 . here, the sentence can be seen divided into two sub-sentences (snt1 and snt2). in snt1, the event of "being born" (bear-02) is associated with mr. t along with the : 𝑡𝑖𝑚𝑒 and : 𝑙𝑜𝑐𝑎𝑡𝑖𝑜𝑛 of birth. while in snt2, the event of "robbing" (rob-01) is described. here, amr can be seen establishing relationships between entities, connecting mr. t to both the birth and the act of robbery. ## conclusion in this paper, we explore the application of abstract meaning representation (amr) in predicting legal judgments. our analysis has revealed both the benefits and challenges associated with using amr in this context. while amrs offer the capability to capture the semantics of legal texts and enable automated analysis and decision-making, providing a promising avenue for fair judgement still remains ambiguous in domains like applicant gender. even so, it clearly demonstrates its efficacy in other group disparities like age and defendant state. however, due to their poor performance and low mf1 scores, we conclude that while amr-based models are fairer by design, they are unsuitable for ensuring fairness in the real world.