D eep P arliament: A Legal domain Benchmark & Dataset for Parliament Bills Prediction (2022)

Abstract: This paper introduces DeepParliament, a legal domain Benchmark Dataset that gathers bill documents and metadata and performs various bill status classification tasks. The proposed dataset text covers a broad range of bills from 1986 to the present and contains richer information on parliament bill content. Data collection, detailed statistics and analyses are provided in the paper. Moreover, we experimented with different types of models ranging from RNN to pretrained and reported the results. We are proposing two new benchmarks: Binary and Multi-Class Bill Status classification. Models developed for bill documents and relevant supportive tasks may assist Members of Parliament (MPs), presidents, and other legal practitioners. It will help review or prioritise bills, thus speeding up the billing process, improving the quality of decisions and reducing the time consumption in both houses. Considering that the foundation of the country”s democracy is Parliament and state legislatures, we anticipate that our research will be an essential addition to the Legal NLP community. This work will be the first to present a Parliament bill prediction task. In order to improve the accessibility of legal AI resources and promote reproducibility, we have made our code and dataset publicly accessible at github.com/monk1337/DeepParliament.

## introduction in recent years, artificial intelligence(ai) based methods have been employed in the legal field for several uses and within many sub-areas. in the legal field the majority of resources are available in textual format (e.g., contracts, court decisions, patents, legal articles). therefore considerable efforts have been made at the intersection of law and natural language processing research. efforts can be witnessed in the various projects dealing with nlp applications in the legal domain and recently published scientific papers such as legal judgement prediction @xcite passed s 2016 y a bill to provide for, as a good governance, efficient, transparent, and targeted delivery of subsidies, benefits and services, the expenditure for which is incurred from the consolidated fund of india... ## dataset collection & preprocessing we constructed the deepparliament corpus from raw data collected from the official foot_2 & open website foot_3 which put together all the parliament bills from 1986 to the present. in addition to the raw data, additional metadata is also provided, i.e. the title, type of bill such as government or private, source of the bill, pdf url and status of the bill. we used pdfminer3 foot_4 to extract bill content from each pdf. some old pdfs are in image format; we applied an ocr system to convert images to text. the pdf content & metadata were converted into csv format and combined into a single dataset. next, we eliminate bill documents with a single token and duplicates. the cleaning pipeline involves removing the special characters, extra spaces etc. all bill documents in this dataset are specifically in the english language. ## dataset analysis the documents on the proposed dataset are considerable long and contain richer information on bill content. as described before, the dataset has been categorized into two settings: binary classification and multi-class classification. the most frequent category status is lapsed, which occupies 50.6%. fig 4 shows the percentage of each status type. lapsed, passed, and withdrawal is the dataset's top three common statuses. the proposed dataset text covers a broad range of bills, including government bill, private member bill, money bill, ordinary bill, financial bill and constitutional amendment bill. we used word cloud to visualize the top commonly occurring legal words in the dataset shown in fig 7 . we visualized the top sections mentioned in the entire dataset. a section is a specific provision of a legal code or body of laws, often laying out a specific legal obligation. most sections in the dataset come under the indian panel code and income tax act. fig. 2 shows the top indian act/law's section mentioned in the dataset. to understand the dataset better, we also visualize the bill status of the last top 20 years. fig. 3 shows the visualization. we can see that year 2015 and 2016 has the most significant failure ratio in the last 20 years, while in 2019, most bills were passed compared to other years. ## methods our study considers ten text classification models ranging from long-range rnn and cnn to transformer-based methods increasing recency and sophistication. ## experiments in this section, we evaluate the mentioned models on the proposed dataset, describe the executed experiments, and examine the results. ## results & discussion it is observed that there is a significant improvement in the model's performance when glove embedding is used as word vectors compared to other embeddings results. bilstm + glove performed best in sequential and convolutional models. cnn + glove gave the second-best results in this category. in the general domain of pre-trained models, transformer models outperform sequential & convolutional models. bert base performed best in both binary and multi-class settings while roberta base and albert base are a close second with better f1-score of all the models based in multi-class and binary settings, respectively. our assumption was that legal domain models would not perform well on the proposed dataset as india's legal systems are completely different. however, our assumption did not hold true. in a few settings, domain-specific models performed well compared to general domain models; this is likely because most words in the proposed dataset are legal domain-specific. the high frequency of unique, domain-specific terminologies appears in the dataset but not in the vocabulary of the transformer models trained on the general domain. 8.1 which portions of the bill contain the most useful information? legal documents are lengthy and include specialist terminology compared to conventional corpora used to train text classification and language models. we did not employ the longformer @xcite and reformer @xcite ) models explicitly designed for lengthy texts due to memory and gpu constraints. we initially experimented with the pre-trained models trained on general-purpose texts. we experimented with various portions of the documents, including starting tokens, middle tokens and end tokens, to overcome the restriction on the number of input tokens bert and other transformer models accept. among all the combinations of input tokens, we observed that the performance of the prediction algorithm improves when more tokens from the first and middle document sections are being used as input. this leads us to infer that the first and middle portion of the documents contains the most helpful information. the proposed dataset shares the quality of including many domain-specific words relevant to the law. when the dataset is limited, the models depend on prior knowledge utilizing the transfer learning. legal-bert base uncased produced the highest macro-averaged f1 score across first and last token settings under both the multi-class & binary classification categories. legal-bert's prior learning is more applicable to the proposed benchmarks. moreover, bert base performed well in the middle token setting in both classification categories. at the same time, legal-roberta base emerged as the second best performing model under the last token setting. figure 6 shows the visualization of macro-f1 scores of different models on the test dataset. we discovered that the best model had a significant advantage over the general domain model since it had been pre-trained in the same language and on data specific to the domain. ## conclusion in this work, deepparliament, a legal domain benchmark dataset, is presented, which requires a deeper domain and language understanding in the legal field. it covers a broad range of parliament bills from 1986 to the present and tests the reasoning abilities of a model. based on extensive quality experiments on different models, it is shown that the dataset is a challenge to the present state-ofthe-art methodologies and domain-specific models, with the best baseline obtaining just 59.79% accuracy. this dataset is anticipated to aid future studies in this field.