Cross Examine: An Ensemble-based approach to leverage Large Language Models for Legal Text Analytics (2024)

Abstract: Legal documents are complex in nature, describing a course of argumentative reasoning that is followed to settle a case. Churning through large volumes of legal documents is a daily requirement for a large number of professionals who need access to the information embedded in them. Natural language processing methods that help in document summarization with key information components, insight extraction and question answering play a crucial role in legal text processing. Most of the existing document analysis systems use supervised machine learning, which require large volumes of annotated training data for every different application and are expensive to build. In this paper we propose a legal text analytics pipeline using Large Language Models (LLM), which can work with little or no training data. For document summarization, we propose an iterative pipeline using retrieval augmented generation to ensure that the generated text remains contextually relevant. For question answering, we propose a novel ontology-driven ensemble approach similar to cross-examination that exploits questioning and verification principles. A knowledge graph, created with the extracted information, stores the key entities and relationships reflecting the repository content structure. A new dataset is created with Indian court documents related to bail applications for cases filed under Protection of Children from Sexual Offences (POCSO) Act, 2012 an Indian law to protect children from sexual abuse and offences. Analysis of insights extracted from the answers reveal patterns of crime and social conditions leading to those crimes, which are important inputs for social scientists as well as legal system.

## introduction legal language is inherently complex @xcite , characterized by formality, precision, and 1 protection of children from sexual offences act (pocso), 2012 : indian law to protect children from sexual offences complexity along with use of specialized vocabulary. legal documents are often lengthy, with intricate reasoning about laws, acts, clauses, and provisions, with redundancy and repetition, as is necessary in the legal domain. legal professionals, who have to wade through large volumes of legal text daily, therefore look for text processing tools that can help them in searching through the documents, and retrieve relevant information efficiently. insights extracted from large collections of legal documents benefit different stakeholders like legal practitioners, clients, social scientists as well as law makers. consequently legal document summarization, sentence / paragraph labeling using classification models and question answering from legal documents have been popular applications of natural language processing(nlp) @xcite , @xcite . advances in the area of natural language processing(nlp) have inspired a large volume of work in the area of legal analytics across the world. legal document summarization, both extractive and abstractive, in different languages have been reported all over the world. legal data analytics is a relatively new area, but gaining rapid popularity. it may be noted that most of the earlier systems were developed using supervised machine learning methods, where the models were trained with large volumes of carefully annotated data, obtaining which is prohibitively expensive. besides, each system catered to a specific use case, for which it was trained, therefore requiring substantial rework for extension to other legal domains or jurisdictions. with the evolution of large language models (llms) @xcite , trained on massive volumes of heterogeneous data from a wide variety of sources, the domain of text processing is seeing a paradigm shift. applying them for legal text analytics is also being explored. however, one of the key challenges of working with these models is the restricted context length on which they can work, which is far less than a standard legal document. another challenge stems from the fact that llms are known to hallucinate, or generate text that may not be contextually relevant or correct, which is also not quite acceptable for legal text processing tasks. the current work was motivated by these challenges and sought to explore whether these challenges can be overcome or bypassed, thereby easing the tasks of legal document summarization and question answering for analytical insight generation. this paper proposes an llm-driven legal text processing pipeline that first generates contextually relevant summaries from long court proceedings, and subsequently uses the summaries to extract legal information to build a knowledge graph from a legal text repository. the knowledge graph is designed following a legal ontology design that stores the core legal concepts and relationships among them. the summarization process uses a "summary of summaries" approach along with retrieval-augmented generation (rag) @xcite and llms @xcite . legal information extraction from the summaries is realized using an ontology-driven llm-powered question-answering system that employs a novel ensemble-based approach. the ensemble approach was motivated by the method of "cross examination" that is used during legal trials to challenge the credibility and reliability of a witness's testimony, uncover inconsistencies, and present an alternative interpretation of the facts. this process is crucial in legal systems to ensure a thorough and fair examination of the evidence. in the current context, the term "cross-examination" is used metaphorically and the key idea is to follow a multi-pronged information retrieval over two phases. in the first phase, for a given type of information to be extracted from a legal document, a set of paraphrased questions are formulated using ontology definitions of concepts and relations. in the second phase, a set of verification questions are formulated with the retrieved answers. the above design of the ensemble of questions is aimed at establishing the validity of llm-generated answers by analyzing them from multiple perspectives. answer to a single question can be compared with its counterparts generated by the paraphrases, and also verified against the original document before it is accepted. our experiments also show that the quality of answers obtained from the summaries are often better than those obtained from the original document. based on the fact that llms work better on shorter contexts than longer ones, this result is as expected. besides the presentation of a novel workflow, contribution of this work also lies in creating a new dataset consisting of indian court proceedings related to protection of children from sexual offences (pocso) act, 2012. the pocso act, 2012 is a gender-neutral law that was passed to protect children from offences including sexual assault, harassment, threatening and child pornography. analyzing cases under the purview of pocso act can unravel insights about the nature of the crimes, social and economic circumstances related to the event of crime, existence of bias, if any, in the judicial system, and also uncover the dynamics of how such cases proceed @xcite . the knowledge graph contains all details extracted from the court proceedings related to bail applications from the accused. experiments and evaluations on this collection show promising and interesting results, thereby establishing the feasibility of the proposed methods in setting up a legal text analytics pipeline in a completely unsupervised way. the repository is publicly available. the rest of the paper is organized as follows. section 2 presents a review of earlier work in the area of legal text analytics. section 3 presents the details of the proposed pipeline for summary generation and question answering from the summaries. section 4 presents details of dataset creation, experimentation and results obtained. section 5 discusses the experiments and the results. section 6 concludes with a discussion of some limitations, which are addressed in section 7, followed by the dataset link in section 8. ## legal text analytics -review of earlier work automated summarization of legal documents has been an active area of research for quite some time. two types of summaries are prevalent-extractive and abstractive. extractive summaries contain a subset of sentences identified as important from the original document. abstractive summaries may contain new words and sentences, which are strewn together to convey the original content with a reduced size but without losing the original meaning. in @xcite , authors have emphasized the use of extractive summaries for legal documents to ensure that the characteristic of legal language is retained in the summary. abstractive summarization stores the essence of a document, but does not preserve exact sentential structures. masked language models like t5, bart, etc., were found to work well for summarization. in @xcite , a t5-based abstractive summary generation model was proposed for eu legal documents. in @xcite , a bart-based model, that could capture the argumentative structure of legal documents by integrating argument role labeling into the summarization process, was proposed. in feijo and moreira (2023) proposed an abstractive text summarization approach using an encoder-decoder architecture. most of the above approaches required a large training corpus to train the models. the models were not transferable and hence not usable in a context other than for which they were designed. to overcome the challenges of high-quality training data, a transfer learning based approach that exploits extractive and abstractive techniques simultaneously, was proposed in @xcite . though large language models (llms) can summarize content pretty well and are known to work with little or no supervision, legal document summarizing still pose a challenge since these are very large and often do not fit into acceptable context lengths. legal question answering and text analytics beyond summarization is emerging as an important area. martinez-gil (2023) presents results of a quantitative and qualitative survey carried out to document the existing challenges in the area, the primary one being the fact that the task is timeconsuming and error-prone. @xcite reports a study on the adoption of large language models(llms) by the legal community. they present a collaboratively constructed legal reasoning benchmark consisting of 162 tasks covering six different types of legal reasoning called legal bench. summarizing documents from indian court proceedings using nlp techniques is a relatively less explored area. recently, platforms like scc online (scc online, 2024), manupatra @xcite , and indian kanoon @xcite have started hosting vast repositories of digitized court proceedings with advanced search capabilities. @xcite presents transformerbased models for rhetorical role labeling to assign labels such as fact, argument, final judgement, etc., to sentences of a court case document. @xcite presents a detailed study on the readiness of general-purpose llms for abstractive summarization of legal documents. they propose a human-in-the-loop approach for obtaining functional summaries with llms. ## ontology-driven framework for legal document summarization and analytics using large language models though large language models (llms), trained over very large repositories are known to be good for general -purpose language generation tasks like summarization or question answering, performance of similar tasks over specialized domains can greatly benefit from the use of ontologies or knowledge graphs @xcite . this is true for all specialized domains like health, climate or legal repositories. a legal ontology is well structured framework that defines the relationships between various legal concepts, entities, principles and processes enabling a systematic understanding of law and legal domain. legal documents are an important part of the ontology @xcite . while some legal documents store information about the legal processes, statutes etc. and are more permanent in nature, legal documents arising out of legal proceedings contain details about a specific case, referring to other legal concepts. each legal proceeding usually gives rise to a number of legal documents of different types. figure 1 presents a portion of the legal ontology used for our work. this ontology is created from concepts and relations presented in @xcite . the ontology depicts the interplay between legal professionals, persons, documents, organisations, principles and process. for instance, it shows that accused, victim, lawyer, judge all are "legal persons", with various properties like gender, age etc. legal documents encompass a large category of documents like case files, bail applications, court proceedings, judgements, contracts etc. each has its own characteristic (not shown in the figure). the ontology plays a key role in information extraction from document summaries, explained in section 3.2. but before that, in the next subsection we present the mechanism for creating document summaries from legal documents using llms. though the ontology is not used for the task of summarization, it plays a key role in its evaluation, wherein it provides the list of key concepts and relations that should be present in the summary. ## dataset creation we now present the details of the dataset that has been created for this work. this dataset was created keeping in mind an important application of legal text mining, namely analysis of crimes against children. according to national crime records bureau (ncrb), india, 43.44% of pocso cases end in no convictions due to lack of evidence @xcite . it was also mentioned that only in about 6% of the cases involved an unfamiliar accused and victim pair. in almost 23% of cases, the victim and accused are known to each other, which includes an approximate estimate of 4% of cases where the accused is a family member. according to the ncrb report of 2022, out of 38,444 cases analyzed, 414 or 1% of the cases involved male victims, while the rest involved female victims. the results stated above were manually curated, and have not been updated for last three years. we belive that with proposed mechanisms, one can do these kind of analysis regularly in an automated way. to check the applicability and validity of the proposed framework for insight generation, a repository of 50 pocso bail applications filed after 2020 has been created. these were collected from two sites ecourts india (2024) and @xcite . the second site also contains human-generated summaries for these applications, along with human annotations for various sections of the documents, which have been used for evaluation purposes. the questions were designed to extract insights like those mentioned by ncrb. the full list of questions is presented in appendix 1 (10.1). ## experiments and results all experiments were done using the langchain @xcite , @xcite platform which facilitates retrieval-augmented generation. results are provided for llama 2 (touvron, 2023) and @xcite . the details of evaluation and results obtained are presented in the following subsections. all experiments were run thrice and average results are presented. ## conclusion in this paper, we have explored how llms can be leveraged to perform legal text analytics. we have proposed an efficient mechanism to generate summaries for legal documents using llms, with no further training. we have also proposed a mechanism to generate a knowledge graph from a repository of case documents, using across-examination figure 6: pocso:citation knowledge graph like technique of posing a set of questions and cross-verifying the answers. going ahead, we intend to build a completely automated pipeline for legal document analytics and summarization. the dataset has been shared in section 8. along with building a large knowledge base, one aspect of research will be focused on automated evaluation of llm-generated content. validating the answers through external causal frameworks is also being explored.