Leveraging Large Language Models for Learning Complex Legal Concepts through Storytelling (2024)

Abstract: Making legal knowledge accessible to non-experts is crucial for enhancing general legal literacy and encouraging civic participation in democracy. However, legal documents are often challenging to understand for people without legal backgrounds. In this paper, we present a novel application of large language models (LLMs) in legal education to help non-experts learn intricate legal concepts through storytelling, an effective pedagogical tool in conveying complex and abstract concepts. We also introduce a new dataset LegalStories, which consists of 294 complex legal doctrines, each accompanied by a story and a set of multiple-choice questions generated by LLMs. To construct the dataset, we experiment with various LLMs to generate legal stories explaining these concepts. Furthermore, we use an expert-in-the-loop approach to iteratively design multiple-choice questions. Then, we evaluate the effectiveness of storytelling with LLMs through randomized controlled trials (RCTs) with legal novices on 10 samples from the dataset. We find that LLM-generated stories enhance comprehension of legal concepts and interest in law among non-native speakers compared to only definitions. Moreover, stories consistently help participants relate legal concepts to their lives. Finally, we find that learning with stories shows a higher retention rate for non-native speakers in the follow-up assessment. Our work has strong implications for using LLMs in promoting teaching and learning in the legal field and beyond.

## introduction often individuals find themselves in certain highstakes situations where they have to educate themselves on novel concepts such as new policies before voting, mortgage terms when buying a house or legal principles relevant to an ongoing lawsuit. unfamiliar terms and nuanced use of language in these contexts can make it challenging for non-experts to make informed decisions, to have equal access to justice, or to participate in civic discourse and democracy. we present this work as a step towards enhancing general legal literacy, bridging the gap between non-experts and experts and promoting constructive and civic discourse. storytelling is an important medium to communicate science to non-experts @xcite and teach professional knowledge to beginners @xcite @xcite . in legal contexts, storytelling has been used extensively to teach abstract legal concepts such as ethics @xcite , and has proven effective at explaining complex legal concepts such as legal mediation to the general public @xcite . however, the scalable implementation of legal storytelling education is severely limited by the high costs associated with legal experts. large language models (llms) and their impressive text generation abilities have facilitated high-quality automated explanations and stories. recent efforts @xcite @xcite have leveraged llms to generate accessible explanations of scientific or medical concepts for diverse audiences. @xcite used gpt-4 to generate explanations for legal concepts from statutory provisions. however, to the best of our knowledge, previous work has not: (1) used llm-generated stories as a medium to explain complex concepts, especially in the under-explored legal domain, (2) generated and refined (via expert feedback) questions for the assessment of concept comprehension, nor (3) validated the effectiveness of llm-generated stories in enhancing comprehension among non-experts. in this work, we explore a novel application of llms that focuses on the use of generated stories and questions to facilitate the learning and assessment of legal concept understanding. we use a human-in-the-loop pipeline that combines llm and expert input to generate stories and multiplechoice questions. we loop in both prolific workers and legal experts foot_0 to ensure that the llmgenerated content is of high-quality. our pipeline presents a holistic approach to llms' application in the legal education domain, where both the learning intervention (stories) and assessment (reading comprehension questions) are generated and evaluated. by providing a reusable dataset and promising experiment results, our work has strong implications for the broader use of llms to enhance teaching and learning and to improve general legal literacy. our contributions are as follows: savelka et al. (2023) used gpt-4 to explain legal concepts in statutory provisions. however, none of the previous legal nlp work has combined llms and storytelling-a widely-used technique in education and communication-as a device to bridge legal experts and non-experts. storytelling in education and nlp stories, which we use interchangeably with the term narratives, are sequential depictions of actions and events (abbott, 2020). they are an effective technique for pedagogy @xcite @xcite . stories effectively illustrate complex concepts in various fields like math, science, and law, with law education heavily relying on "fact-patterns" @xcite . specifically, second-person narratives are known to elicit particularly strong emotional and aestheticreflexive involvement from readers @xcite , especially when combined with more advanced narrative techniques and emotive verbs @xcite . automatic story generation is a long-standing task in nlp. before llms, the best story generation models, even those using transformer architectures, struggled to create coherent stories with well-defined characters and story-lines @xcite . recent progress in llms such as chatgpt have opened up new possibilities with storytelling, showing significantly higher quality compared to other story generation models @xcite @xcite . however, little work has explored the use of llm-generated stories in education. @xcite experimented with llms to generate age-appropriate stories for children. our work is the first to use llm-generated stories to help non-experts in learning legal knowledge. educational question generation generating educational questions or quizzes is important for educators to increase engagement, test reading comprehension, and improve learners' knowledge retention @xcite . in the era of modern nlp, question generation (qg) was first tackled with seq2seq @xcite and transformer-based models @xcite . previous qg work tackled selecting question-worthy content @xcite , 2020; @xcite , modeling question types @xcite @xcite @xcite , or focused on specific interrogative words such as what, how, etc. @xcite . a subset of work focuses on the needs of students and teachers to tackle question generation to make them more accessible, adaptive, and educational @xcite @xcite @xcite @xcite . recently, llms have been widely used to generate questions @xcite @xcite @xcite @xcite in different domains such as language learning @xcite , commonsense knowledge @xcite , coding @xcite , and science @xcite . however, prior research primarily evaluated generated questions via automatic metrics or simple human ratings. differently from previous work, @xcite evaluated the usability of generated questions in a practical learning setting and showed that asking non-native learners to finish questions after reading helps them understand sci-ence texts. our work instead uses llm-generated questions as a tool to assess the effectiveness of llm-generated stories in legal concept learning. ## legalstories dataset in figure 1 , we present a pipeline and demonstrate how we apply it to curate a new dataset. it consists of three components: story generation, question generation, and expert critique. the following paragraphs illustrate how we generate stories and questions from legal doctrine definitions and refine questions with expert feedback. ## evaluation a two-fold evaluation is carried out as follows: (a) an evaluation to determine the quality of the generated stories relating to doctrines, and (b) an evaluation of the generated questions and answers, as well as their efficacy in assessing comprehension. ## conclusion in this work, we explore a novel application of llms in legal concept learning through storytelling. we use an expert-in-the-loop pipeline to create the legalstories dataset that contains educational stories and comprehension questions. moreover, we compare the performance of several benchmark llms including gpt-4, gpt-3.5, and llama 2 with automatic and human evaluations. while gpt-4 outperforms the others at generating legal stories and creating questions, it still exhibits certain reasoning errors, highlighting the need for human supervision when using llms for educational content development. finally, through rcts, we show that, among non-native speakers, learning with stories not only improves comprehension of legal concepts and interest but also leads to a higher retention rate in the follow-up assessment compared to learning with definitions alone. our study suggests considerable potential for using llms in advancing legal education and beyond.