Enhancing Legal Expertise in Large Language Models through Composite Model Integration: The Development and Evaluation of Law-Neo (2024)

Abstract: Although large language models (LLMs) like ChatGPT have demonstrated considerable capabilities in general domains, they often lack proficiency in specialized fields. Enhancing a model’s performance in a specific domain, such as law, while maintaining low costs, has been a significant challenge. Existing methods, such as fine-tuning or building mixture of experts (MoE) models, often struggle to balance model parameters, training costs, and domain-specific performance. Inspired by composition to augment language models, we have developed Law-Neo, a novel model designed to enhance legal LLMs. This model significantly improves the model’s legal domain expertise at minimal training costs, while retaining the logical capabilities of a large-scale anchor model. Our Law-Neo model outperformed other models in comprehensive experiments on multiple legal task benchmarks, demonstrating the effectiveness of this approach.

## introduction large language models (llms) have shown significant capabilities, including commonsense and factual reasoning, world knowledge, and language generation. these abilities have been validated across various scientific fields such as finance, biochemistry, and medicine @xcite @xcite @xcite . however, the training cost escalates as the number of parameters in llm increases when enhancing model's domain-specific capabilities. this cost barrier is a significant challenge in developing domain-specific llms, such as those for the legal field. to address these challenges, we propose the development of a comprehensive llm-based legal assistance system. ## related work since the development of bert @xcite , significant efforts have been made to create language models (lms) specifically tailored for the legal domain. initially, these models were small and followed the paradigm of pre-training followed by downstream task fine-tuning. recent advancements have seen an increase in model size and the introduction of instruction fine-tuning, with evaluations extending across a broader spectrum of legal tasks. most existing legal llms are text-based, with a focus on chinese, english, or multi-language support @xcite . ## method this section focuses on law-neo as illustrated in fig. 1 . while our model comprises three main components-the domain knowledge model (qwen2-7b-legal), the anchor model (qwen2-72b-chat @xcite , and the parameter-merging block-this section will specifically discuss the domain knowledge model and the parametermerging block in detail. our approach assumes: (i) the model weights are fixed and unmodifiable, reflecting the high computational cost of training or fine-tuning large llms from scratch in production environments. pre-trained models are treated as fixed assets to ensure cost efficiency and stability. (ii) we can access model weights, perform forward and backward passes, and retrieve intermediate representations. this is feasible with many open-source llms, allowing us to use their parameters for inference and further training. (iii) we lack access to the original training data, hyperparameters, or training states, as open-source llms typically do not provide such information. ## experiments we evaluated the performance of lawbench @xcite and the unified qualification exam for legal professionals. additionally, we conducted benchmark testing for the ljp(legal judgment prediction) task. our primary focus, lies in ljp tasks utilizing fact-based articles from the cail2018 @xcite dataset. ## training overhead our training procedure was conducted on a gpu server equipped with eight 80gb a800 gpus and an intel xeon 8470 processor. the entire training process took approximately 19.24 hours. the comparative training regimen for chatlaw was conducted on a gpu server equipped with eight 80gb a100 gpus and two intel xeon 8358p processors. the entire training process was completed in approximately 23.14 hours, which exceeds our training duration by 3.9 hours. ## conclusion in this paper, we introduced law-neo, an innovative approach to enhancing large language models (llms) for the legal domain by leveraging the concept of composition to augment existing models. our methodology focused on integrating qwen2-72b-chat with a legal domain-specific model based on qwen2-7b. our comprehensive experiments, conducted on multiple legal benchmarks including lawbench, the unified qualification exam for legal professionals (uqelp), and cail2018, demonstrate the efficacy of our approach. the law-neo model outperformed several existing models, including general-purpose llms and specialized legal llms. our results indicate that integrating models through shared parameters can effectively enhance their specialized knowledge without sacrificing the foundational abilities of the base models.