Reformulating Domain Adaptation of Large Language Models as Adapt-Retrieve-Revise: A Case Study on C hinese Legal Domain (2024)

Abstract: While large language models (LLMs) like GPT-4 have recently demonstrated astonishing zero-shot capabilities in general domain tasks, they often generate content with hallucinations in specific domains such as Chinese law, hindering their application in these areas. This is typically due to the absence of training data that encompasses such a specific domain, preventing GPT-4 from acquiring in-domain knowledge. A pressing challenge is that it’s not plausible to continue training LLMs of the GPT-4’s scale on in-domain data.This paper introduces a simple yet effective domain adaptation framework for GPT-4 by reformulating generation as an adapt-retrieve-revise process. The initial step is to adapt an affordable 7B LLM to the Chinese legal domain by continuing learning in-domain data. When solving an in-domain task, we leverage the adapted LLM to generate a draft answer given a task query. Then, the draft answer will be used to retrieve supporting evidence candidates from an external in-domain knowledge base. Finally, the draft answer and retrieved evidence are concatenated into a whole prompt to let GPT-4 assess the evidence and revise the draft answer to generate the final answer. Our proposal combines the advantages of the efficiency of adapting a smaller 7B model with the evidence-assessing capability of GPT-4 and effectively prevents GPT-4 from generating hallucinatory content. In the zero-shot setting of four Chinese legal tasks, our method improves the average score by +33.6 points, compared to GPT-4 direct generation. When compared to two stronger retrieval-based baselines, our method outperforms them by +17.0 and +23.5.

## introduction recent large language models like gpt-4 bring remarkable improvements in various general domain nlp tasks @xcite @xcite @xcite . however, in specific domains such as the chinese legal domain, the performance of such general llms still lags considerably behind. we show a real example of chinese legalqa @xcite on the left of figure 1 , which requires the model to generate the corresponding legal provision (i.e., the law name and the clause index) and the rationale for the judgment, given a brief case description as the query. we initialize the research with a preliminary examination of utilizing gpt-4 to address the chinese legalqa task, which involves responding with a law clause relevant to a given query case. figure 1 reveals the extremely low performance (f1 12.0) of directly prompting the query case to ask gpt-4 to generate the corresponding law clause. though the generated answers are grammatically fluent, they often consist of non-logical content, factual mistakes, and fail to refer to the correct legal provision (also known as "hallucination"). for example, in the left answer of figure 2 , the direct generation of gpt-4 seems to retell the case description but fails to point out the corresponding clause. a potential reason is the insufficient chinese legal domain text during pretraining, leading to a lack of domain knowledge acquisition, and consequently generating hallucinatory content. for the llms with the scale of gpt-4, it's generally not feasible for researchers to do learningbased adaptation. the enormous model size could make the cost of continual learning extremely high, and meanwhile, the access functions are often limited by apis. therefore, recent work @xcite @xcite introduces retrieval-based methods that first use the given query to retrieve relevant evidence candidates from the external domain-specific knowledge base or the internet and then concatenate the query and the evidence candidates into the prompt. gpt-4 could implicitly validate the relevance between the query and the evidence, as well as the correctness of the evidence, before producing a generation. our replicated retrieval-based method improves the legalqa f1 from the 12.0 points of direct generation to 33.1 as shown in figure 1 . it indicates that even though gpt-4 may not generate domain content, it possesses sufficient evidence-assessing capacity to select the correct evidence from candidates. nevertheless, the retrieval module is limited by the capability of representation mapping from query to evidence and is also influenced by the domain issue, leading to a decline in search quality. gpt-4 still produces hallucinations in responses as the middle answer in figure 2 . on the other hand, with the rapid development of open llms led by llama @xcite , it becomes affordable to continually train an open llm tailored to your demands on sufficient indomain texts, resulting in a domain-adapted llm. we therefore conduct the second examing of continually training baichuan 7b (baichuan-inc, 2023), a chinese foundation model, on over 50b token chinese legal data. its performance (the green bar in figure 1 ) even surpasses the retrieval-based gpt-4 generation on chinese legalqa. hallucinations caused by the lack of domain knowledge are largely reduced but not completely solved. as shown in the right answer in figure 2 , adapted llm generates generally correct responses but still makes query: what are the penalties for parking violations under the traffic code? answer: ## methodology our adapt-retrieve-revise method consists of three steps. in the first step (section 2.1), we continually train a chinese pre-trained llm on the chinese legal domain corpora to derive a domain-adapted legal llm and given the query, the legal llm will generate the draft answer. in the second step (section 2.2), we use a sentence embedding model to produce embeddings for both the draft answer and each paragraph in the corresponding knowledge base, then evidence retrieval will be computed by the similarities between the answer embedding and the paragraph embeddings. in the third step (section 2.3), we concatenate the query, the draft answer, and the retrieved evidence in the prompt for gpt-4 to revise and produce the final response. figure 3 shows the overview of our method. in the following sections, we will introduce details of each step. ## experiment settings we conducted a series of experiments to compare our adapt-retrieve-revise method to the baselines of direct generation and retrieval-based generation on various chinese legal benchmarks. we show the model details and the task settings in this section. ## main experimental results we provide the main results as in we first observe the effectiveness of domain adaption. our 7b legal llm significantly beats gpt-4, and even compared with the retrieval-based generation of gpt-4, the 7b legal llm still outperforms on three tasks and has competitive results on the jec-qa task, indicating that our continual learning on chinese legal raw corpora shows a fast and effective domain adaptation on various legal tasks. however, related work (lawgpt and chatlaw) fails to benefit largely from the continual training due to the much less data used in training, and their base models are multilingual llama. then, considering the results of gpt-4 and the gpt-4 retrieval-based generation, we find that after providing evidence of related legal knowledge, gpt-4 can improve its responses significantly (+10.1 points). this indicates that the retrievalbased method is a proper way to reduce hallucinations caused by the lack of domain knowledge, and owing to the robust evidence-assessing capacity, gpt-4 can adapt to the chinese legal domain well with convincing evidence available. in our final experiment, using the draft answers generated by the 7b legal model for retrieval and revision, the performance significantly exceeded two query-based retrieval baselines by large margins of +17.0 and +23.5 points. it's worth noting that the improvement here comes from both the enhanced answer-based retrieval quality and the revision setup. in the subsequent ablation study, we will further examine the quantified improvement of the retrieval quality through an additional retrieving task. an interesting observation is that, by comparing the direct generation of the 7b legal model and the adapt-retrieve-revise method with the revision model as the legal 7b model, we find that with retrieved evidence, the revised answers seem to be no obvious difference from the direct generation. this indicates that the 7b legal llm shows almost zero evidence-assessing capacity. ## conclusions in this paper, we reformulate the zero-shot domain content generation of large language models as an adapt-retrieve-revise procedure. this approach combines the merits of efficient continual training of a smaller 7b llm for domain adaptation, robustly retrieving the supporting evidence from an external knowledge base, and effectively leveraging the evidence-assessing and revision capabilities of gpt-4. our proposal substantially enhances the zero-shot performance on the chinese legal tasks.