Towards Cross-Domain Transferability of Text Generation Models for Legal Text (2022)

Abstract: Legalese can often be filled with verbose domain-specific jargon which can make it challenging to understand and use for non-experts. Creating succinct summaries of legal documents often makes it easier for user comprehension. However, obtaining labeled data for every domain of legal text is challenging, which makes cross-domain transferability of text generation models for legal text, an important area of research. In this paper, we explore the ability of existing state-of-the-art T5 & BART-based summarization models to transfer across legal domains. We leverage publicly available datasets across four domains for this task, one of which is a new resource for summarizing privacy policies, that we curate and release for academic research. Our experiments demonstrate the low cross-domain transferability of these models, while also highlighting the benefits of combining different domains. Further, we compare the effectiveness of standard metrics for this task and illustrate the vast differences in their performance.

## introduction legalese is often perceived to be an expert language containing jargon-filled text, which makes it difficult for non-experts to comprehend @xcite @xcite . however, owing to recent regulations @xcite there is a shift in paradigm to make legal documents more accessible to nondomain experts. summarizing such documents is a vital step in this direction. a few examples include summarization over legislative bills @xcite @xcite and legal contracts like terms of service @xcite @xcite . however, obtaining annotated data for every domain of legal text for this task is * equal contribution expensive and often infeasible. thus, exploring the ability of text generation models to transfer across multiple legal domains is of importance, particularly for low resource domains for which knowledge transfer from domains with large amounts of annotated data could be beneficial. while there has been research on various tasks and aspects of legal text, such as summarization @xcite @xcite , question answering @xcite and title generation in privacy policies @xcite , transferability of generative models across legal domains has remained relatively understudied. in this work, we explore the cross-domain transferability of state-of-the-art text generation models across four distinctly different legal domains. we use standard summarization metrics to measure their degree of transferability. further, we compare the effectiveness of such metrics at capturing the summarization capability of these models, and demonstrate the differences thereof. further, since summarization datasets are not available in the privacy policy domain, we curate and release an annotated dataset for further research. ## datasets in order to study cross-domain transferability of generative models, we select four summarization datasets consisting of legal text of varying domains, each of which is described below. billsum: this dataset (kornilova and eidelman, 2019) consists of us congressional bills collected over a 25 year time-period (103 rd -115 th sessions of us congress) ranging from 1993-2018 & summarized by the respective legislative counsel. jrc-acquis (en): this dataset introduced by @xcite is composed of the contents, political objectives of treaties, legislation, declarations, etc. pertaining to the member states of the eu. we focus on the english subset of the corpus for this paper. the task here is to summarize the paragraphs of the documents using their titles. legal contracts: curated by @xcite , this dataset is composed of unilateral legal contracts such as terms of service, terms of use and licensing agreements. instead of summarizing the entire document as a whole, manually curated summaries of each section are provided. privacy policies: privacy policies are legal documents that disclose ways in which a company collects and manages their user data. each section of the privacy policy discusses various facets of user data management. while there has been work done to summarize sections of privacy policies @xcite , there is no open sourced dataset available for this task. privacy policies dataset creation: we leverage the ∼1m english language privacy policy dataset @xcite in order to create & release a dataset for section summarization. to that end, we sample a subset of 20k privacy policies at random, from which we randomly select 24k sections. the dataset created for section summarization consist @xmath0, @xmath1. for more details, please refer to appendix a.1. the statistics of the train/dev/test split for these datasets is shown in ## methodology & experiments we leverage pretrained seq2seq transformer-based text generation models such as bart @xcite and t5 @xcite for our experiments. in order to measure cross-domain transferability of generative models for the four domains discussed in section 2, we first conduct experiments in the single-domain setting, in which each seq2seq model is fine-tuned with @xmath2, @xmath3, and subsequently used to generate summaries for the test splits of each dataset, both in and out-of-domain. cross-domain performance of these models helps determine their transferability across different domains. further, in order to compare with a scenario in which the text generation model learns from all domains and is thereby able to incorporate the domain differences during generation, we propose a multi-domain setting, in which we fine-tune the model with training data of all domains, and generate summaries for each of the four datasets. for more details on experimental settings, please refer to appendix a.3. standard metrics such as rouge-l @xcite , meteor @xcite , bertscore @xcite and bartscore @xcite are reported to measure model performance. ## results in this section, we report quantitative ( ## conclusion & future work in this paper, we study the cross-domain transferability of neural text generation models across four different domains of legal text. we consider seq2seq model architectures such as bart & t5 and fine-tune them on datasets of specific do-mains. based on standard generation metrics such as rouge, meteor, bertscore & bartscore, we find such models to show a drop in performance for cross-domain settings. further, our experiments demonstrate the benefits of combining different domains to train models for such tasks. moreover, we observe some metrics to be more effective at capturing the differences in predicted and groundtruth summaries. we also curate & release a dataset on title generation for privacy policies for further research in this direction. in the future, we wish to explore text generation specific to legal text for low resource scenarios including zero and few-shot settings.