IL - TUR : Benchmark for I ndian Legal Text Understanding and Reasoning (2024)

Abstract: Legal systems worldwide are inundated with exponential growth in cases and documents. There is an imminent need to develop NLP and ML techniques for automatically processing and understanding legal documents to streamline the legal system. However, evaluating and comparing various NLP models designed specifically for the legal domain is challenging. This paper addresses this challenge by proposing : Benchmark for Indian Legal Text Understanding and Reasoning. contains monolingual (English, Hindi) and multi-lingual (9 Indian languages) domain-specific tasks that address different aspects of the legal system from the point of view of understanding and reasoning over Indian legal documents. We present baseline models (including LLM-based) for each task, outlining the gap between models and the ground truth. To foster further research in the legal domain, we create a leaderboard (available at: https://exploration-lab.github.io/IL-TUR/ ) where the research community can upload and compare legal text understanding systems.

## introduction besides several other purposes, legal systems have been established in various countries to ensure, at the very minimum, order and fairness in society and to safeguard fundamental human rights. however, legal systems worldwide struggle with exponentially growing legal cases in various courts. it is even more pronounced in populous countries; e.g., in india, there are about 50 million pending cases in multiple courts at various levels (district, state, federal) (national judicial data grid, 2023). such a massive backlog of cases goes against the why a separate benchmark for the legal domain? the legal text involves natural language but differs from the regular text used to train nlp models. 1) many of the terms used in legal documents are domain-specific. for example, some words used in everyday language have specialized meanings in legal parlance. the presence of a different lexicon posits a need for specialized nlp tools to handle legal texts. 2) legal documents are typically very long compared to regular texts. for example, the average length of a legal document from the supreme court of india (sci) is 4000 words @xcite . it poses a challenge for existing nlp models (e.g., llms) as the information is spread throughout the document and must be linked together for reasoning. moreover, many of the existing language models (e.g., bert (devlin et al., 2019)) have limitations on the length (512 tokens) of the input. it requires developing specialized models for processing and handling long legal documents. 3) legal documents are highly unstructured and sometimes noisy (for example, in the indian setting, most documents are typed manually in the courts and prone to grammatical mistakes and typos). the absence of structure in the documents makes extracting semantically relevant information from large chunks of text difficult. 4) the legal domain is further subdivided into specialized subdomains; for example, criminal law differs from civil law, and both differ from banking and insurance law. even though some fundamental legal principles are shared across various laws, models trained on a particular law (e.g., civil law) may not work on another (e.g., banking and insurance law). hence, domain adaptation is a challenge. 5) lastly, many existing state-of-the-art (sota) nlp models are black boxes; however, explainability is not a second-class citizen for the legal domain. for models to be widely usable by legal practitioners, these need to be explainable. due to the above reasons, a separate set of models/systems is required to process and understand legal documents. given the huge backlog of cases, nlp-based technologies could come to our rescue and help streamline the legal workflow. even a small technical intervention can have a considerable impact. hence, a benchmark is needed to promote the development of models in this area. in a nutshell, we make the following contributions: ## il-tur: legal-nlp benchmark ## conclusion and future directions this paper presented il-tur, a benchmark for indian legal text understanding and reasoning. the benchmark has eight tasks requiring different types of legal skills to solve. results indicate that the tasks are far from solved using state-of-the-art transformer-based models and llms. the list of tasks in il-tur is not exhaustive, and we plan to expand the list of tasks in the future; for example, we are working on developing foundational tasks like legal coreference resolution (l-coref) that are required for various applications such as information extraction and knowledge graph creation. although such tasks have been addressed well in general nlp, our initial experiments show that using sota transformer models (which have become part of standard nlp toolkits) do not perform well on legal texts. due to the usage of specialized terms, new models are needed for the legal domain. on the modeling side, in the future, we plan to develop one model that generalizes and works across all the tasks (e.g., mt5 (xue et al., 2020) and multitask adapters (pfeiffer et al., 2020)). overall, we hope that il-tur (along with its leaderboard) and its successive versions would create excitement in the legal-nlp community and lead to the development of new technologies that could benefit society immensely and facilitate fair access to justice, a fundamental human right.