FEDLEGAL : The First Real-World Federated Learning Benchmark for Legal NLP (2023)

Abstract: The inevitable private information in legal data necessitates legal artificial intelligence to study privacy-preserving and decentralized learning methods. Federated learning (FL) has merged as a promising technique for multiple participants to collaboratively train a shared model while efficiently protecting the sensitive data of participants. However, to the best of our knowledge, there is no work on applying FL to legal NLP. To fill this gap, this paper presents the first real-world FL benchmark for legal NLP, coined FEDLEGAL, which comprises five legal NLP tasks and one privacy task based on the data from Chinese courts. Based on the extensive experiments on these datasets, our results show that FL faces new challenges in terms of real-world non-IID data. The benchmark also encourages researchers to investigate privacy protection using real-world data in the FL setting, as well as deploying models in resource-constrained scenarios. The code and datasets of FEDLEGAL are available here.

## introduction it has been noticed that learning, comprehending and properly using an ever-increasing huge amount of legal data is way beyond human capability of legal practitioners @xcite . since the majority of the data is text, such an "information crisis in law" is encouraging the research and development of legal natural language processing (nlp) techniques, to provide affordable legal services to both legal professionals and the general public @xcite . as the majority of those techniques are based on machine learning, they require training on centralized datasets. however, such approaches raise increasing privacy concerns of the public and impose risks of breaching data protection laws, such as the general data protection regulation (gdpr). to address the above concerns, federated learning (fl) is widely considered as a family of training algorithms to achieve a promising trade-off between information utility and privacy preservation, without sharing sensitive data of data owners @xcite . as depicted in figure 1 , those algorithms permit local machines of participants to coordinate with one or multiple servers to train a model in a decentralized and collaborative way while preserving data privacy. despite its rosy future, fl still faces open challenges due to the needs of coping with data heterogeneity @xcite , privacy attacks @xcite , and system inefficiency @xcite . in particular, differences between local data distributions of participants impose a special challenge when they are not independently and identically distributed (non-iid) @xcite . although this phenomenon is broadly observed in practice, almost all studies in this area rely on artificially partitioned non-iid datasets using heuris-the plaintiff, xie ×× (id card: 6127×××4503, famele) , made a lawsuit request to the court: 1. request to order the two defendants to return the loan principal of 1,331,173 yuan and overdue repayment interest; 2. the litigation costs of this case shall be borne by the two defendants. ## preliminaries this section starts with reviewing the concepts, problem formulations, and challenges of federated learning, followed by providing an overview of the lifecycle of the lawsuit in the chinese court system. ## fedlegal to facilitate the research on the incorporation of fl and legalai, we present the legal fl benchmark fedlegal with natural non-iid partitions and practical private information. fedlegal consists of six critical legal tasks which covers a broad range of task types, federated participant numbers, and natural non-iid data as shown in ## experiment in this section, we first show the performance of different fl algorithms on fedlegal (see section 4.2). to obtain a clear understanding of the practical challenges of fl in real-world applications, we conduct an in-depth investigation on fedle-gal, covering privacy leakage analysis (see section 4.3) and resource-constrained fl scenario (see section 4.4). ## conclusion this paper proposes the first real-world federated learning benchmark for legal nlp (fedlegal), which contains five nlp tasks and one privacy task. the benchmark features a large number of fl participants and natural non-iid data partitions. on this dataset, we conduct the extensive empirical study, including performance comparisons, privacy leakage, and resource-constrained analysis. the experimental results reveal that fl algorithms are effective for real-world applications but our benchmark poses new challenges on natural non-iid partitions. in addition, we build a lightweight and easy-to-extend codebase to facilitate fl research in the legal domain. we hope that fedlegal would facilitate the development of novel and practical fl algorithms for real-world legal applications.