Legal-Tech Open Diaries: Lesson learned on how to develop and deploy light-weight models in the era of humongous Language Models (2022)

Abstract: In the era of billion-parameter-sized Language Models (LMs), start-ups have to follow trends and adapt their technology accordingly. Nonetheless, there are open challenges since the development and deployment of large models comes with a need for high computational resources and has economical consequences. In this work, we follow the steps of the R&D group of a modern legal-tech start-up and present important insights on model development and deployment. We start from ground zero by pre-training multiple domain-specific multi-lingual LMs which are a better fit to contractual and regulatory text compared to the available alternatives (XLM-R). We present benchmark results of such models in a half-public half-private legal benchmark comprising 5 downstream tasks showing the impact of larger model size. Lastly, we examine the impact of a full-scale pipeline for model compression which includes: a) Parameter Pruning, b) Knowledge Distillation, and c) Quantization: The resulting models are much more efficient without sacrificing performance at large.

## introduction transformer-based languages models (lms) @xcite @xcite have stormed nlp benchmarks with state-of-the-art performance, while recently humongous billion-parameter-sized models @xcite @xcite have showcased impressive few-shot capabilities. in addition, multi-lingual lms @xcite have been also developed demonstrating exceptional results as well as impressive performance in zero-shot cross-lingual transfer. the legal nlp literature is also flourishing with the release of many new resources, including large legal corpora @xcite , benchmark datasets @xcite @xcite @xcite , and pre-trained legal-oriented language models @xcite . despite this impressive progress, the efficacy of differently-sized language models on legal nlp tasks and the importance of domain (legal) specificity are still understudied, while the effect of model compression techniques in model's performance and efficiency is ignored. in this work, we aim to shed light in all these directions following model development across three incremental steps in a pipelined approach: to do so, we initially develop 4 multi-lingual legaloriented language models (c-xlms). we benchmark their performance across 5 down-stream legal nlp tasks, comprising both publicly available and private datasets, covering both english and multilingual scenarios in several tasks types, i.e., document/sentence classification, natural language inference, and entity extraction. finally, we experiment with a full-scale pipeline for model compression which includes a) parameter pruning, b) knowledge distillation, and c) quantization to produce much more efficient (smaller and faster) models that can be effectively deployed in production. our work aims to provide guidelines to legaltech practitioners on model development (pretraining, fine-tuning, compression) bearing both performance and efficiency into consideration. our findings show that the impact of larger vs. smaller models, domain-specific vs. generic models and the efficacy of model compression techniques varies across tasks, but in general larger domainspecific models perform better. via full-scale model compression, we produce models with performance decrease by 2.3 p.p., while being approx. 42× smaller, and approx. 66× faster. we also find that fully compressed models outperform equally sized distilled or fine-tuned models. ## model specifications following @xcite , we pre-train from scratch legal domain specific transformerbased language models. our models are based on the roberta architecture @xcite , i.e., trained with the masked language modelling (mlm) objective, excluding the next sentence prediction (nsp) one used by bert @xcite . in addition, based on the industry needs and driven by the work of @xcite , our models are a multilingual one -usually referred as xlm in the literature-and supports ten languages in total (english, french, german, greek, spanish, italian, dutch, polish, portuguese, russian). we pre-train 4 variants of custom xlm models (c-xlm) starting from a large version with 24 transformer blocks (layers), each consisting of 1024 hidden units and 16 attention heads and continue by decreasing each time by a factor of 2 across all dimensions, i.e., blocks/layers, hidden units, and attention heads ( ## conclusions following model development across all three incremental steps of the examined pipelined approach, we make the following observations: to conclude, our guidelines to legaltech practitioners who aim to build effective, but also efficient models, can be summarized in four general points: