MAUD : An Expert-Annotated Legal NLP Dataset for Merger Agreement Understanding (2023)

Abstract: Reading comprehension of legal text can be a particularly challenging task due to the length and complexity of legal clauses and a shortage of expert-annotated datasets. To address this challenge, we introduce the Merger Agreement Understanding Dataset (MAUD), an expert-annotated reading comprehension dataset based on the American Bar Association’s 2021 Public Target Deal Points Study, with over 39,000 examples and over 47,000 total annotations. Our fine-tuned Transformer baselines show promising results, with models performing well above random on most questions. However, on a large subset of questions, there is still room for significant improvement. As the only expert-annotated merger agreement dataset, MAUD is valuable as a benchmark for both the legal profession and the NLP community.

## introduction while pretrained transformers @xcite have surpassed humans on reading comprehension tasks such as squad 2.0 @xcite and superglue @xcite , their accuracy in understanding real-world specialized legal texts remains underexplored. reading comprehension of legal text can be a particularly challenging natural language processing (nlp) task due to the length and complexity of legal clauses and the difficulty of collecting expertannotated datasets. to help address this challenge, we introduce the merger agreement understanding dataset (maud), a legal reading comprehension dataset curated under the supervision of highly specialized mergers-and-acquisitions (m&a) lawyers and used in the american bar association's 2021 public target deal points study ("aba study"). the dataset and code for maud can be found at github.com/theatticusproject/maud. public target company acquisitions are the most prominent business transactions, valued at hundreds of billions of dollars each year. merger agreements are the legal documents that enable these acquisitions, and key clauses in these merger agreements are called "deal points." lawyers working on the aba study perform contract review on merger agreements. in general, contract review is a two-step process. first, lawyers extract key legal clauses from the contract (a span extraction task). second, they interpret the meaning of these legal clauses (a reading comprehension task). in the aba study, the lawyers extract deal points from merger agreements, and for each deal point they answer a set of standardized multiplechoice questions. models trained on maud's expert-annotated data can learn to answer 92 reading comprehension questions from the 2021 aba study, given extracted deal point text from merger agreements. by answering these questions, models interpret the meaning of specialized legal language and categorize the different agreements being made by companies in the contract. span extraction and reading comprehension are both important and challenging tasks in legal contract review. a large-scale expert-annotated span extraction benchmark for contract review is already available in @xcite . however, to the best of our knowledge, there is no large-scale expert-annotated reading comprehension dataset for contract review or any other legal task in the english language. therefore in this short paper, we focus on the legal reading comprehension task. (appendix a.14 presents a preliminary benchmark for the extraction task for interested researchers.) annotating maud was a collective effort of over 10,000 hours by law students and experienced lawyers. prior to labeling, each law student attended 70-100 hours of training, including lectures and workshops from experienced m&a lawyers. outputs no business and operation of target ability to consummate transaction deal point answers: ☑ ☑ ☐ ☐ ☐ ☐ deal point question: fls (mae) applies to ## related work due to the high costs of contract review and the specialized skills it requires, understanding legal text has proven to be a ripe area for nlp research. ## conclusion maud is a large-scale expert-annotated dataset which facilitates nlp research on a specialized merger agreement review task, based on the american bar association's public target deal point study. maud can accelerate research towards specialized legal tasks like merger agreement review, while also serving as a benchmark for assessing nlp models in legal text understanding. finetuned llm baselines exhibit strong performance on some deal point categories, but there is significant room for improvement on the three hardest categories.