T ong G u: Mastering Classical C hinese Understanding with Knowledge-Grounded Large Language Models (2024)

Abstract: Classical Chinese is a gateway to the rich heritage and wisdom of ancient China, yet its complexities pose formidable comprehension barriers for most modern people without specialized knowledge. While Large Language Models (LLMs) have shown remarkable capabilities in Natural Language Processing (NLP), they struggle with Classical Chinese Understanding (CCU), especially in data-demanding and knowledge-intensive tasks. In response to this dilemma, we propose TongGu (mean understanding ancient and modern), the first CCU-specific LLM, underpinned by three core contributions. First, we construct a two-stage instruction-tuning dataset ACCN-INS derived from rich classical Chinese corpora, aiming to unlock the full CCU potential of LLMs. Second, we propose Redundancy-Aware Tuning (RAT) to prevent catastrophic forgetting, enabling TongGu to acquire new capabilities while preserving its foundational knowledge. Third, we present a CCU Retrieval-Augmented Generation (CCU-RAG) technique to reduce hallucinations based on knowledge-grounding. Extensive experiments across 24 diverse CCU tasks validate TongGu’s superior ability, underscoring the effectiveness of RAT and CCU-RAG. The model and dataset are available at https://github.com/SCUT-DLVCLab/TongGu-LLM .

## introduction classical chinese is a vital bridge in connecting the present with the wisdom of ancient china, illuminating insights into historical social life and cultural practices. however, the significant linguistic differences between classical and modern chinese, including vocabulary and syntax, render this invaluable heritage prohibitively challenging for non-experts to understand. recent advancements in large language models (llms) have demonstrated remarkable capability in addressing various natural language processing (nlp) tasks @xcite @xcite @xcite @xcite , prompting researchers to explore their capabilities in the specialized realm of classical chinese understanding (ccu). however, existing models, including general-purpose and preliminary ccu-specific llms (wptoux, 2023; xunzi-llm-of-chinese-classics, 2024), often struggle with tasks that require large-scale training data or extensive domain knowledge. this predicament primarily stems from two causes: the lack of dedicated instruction-tuning datasets capable of unleashing their full capabilities, and models' innate propensity to generate hallucinations when tackling knowledge-intensive tasks without sufficient factual grounding. to address these challenges, we present tonggu, a pioneering vertical domain llm as well as the most proficient ccu specialist. we commence with devising an automated pipeline to construct instruction data from classical chinese texts, resulting in accn-ins (short for ancient chinese instruction), the first publicly available ccu instruction dataset catering to diverse ccu tasks. subsequently, tonggu undergoes a two-stage instruction tuning, respectively for the optimization of data-hungry and data-efficient tasks. here, "datahungry" means that a substantial volume of data is required to attain satisfactory model performance, and "data-efficient" denotes that a small amount of data is sufficient to achieve desired outcomes. it is first fine-tuned on data-hungry tasks using largescale training data, such as classical to modern chinese translation, followed by a second stage of fine-tuning on data-efficient tasks such as punctuation with small-scale data. to prevent catastrophic forgetting during the two-stage fine-tuning, we propose redundancy-aware tuning (rat), a novel sparse fine-tuning (a.k.a. parameter-efficient fine-tuning (peft)) method that identifies and freezes the most crucial layers for the current task according to layer redundancy. rat effectively injects new capability to the model while preserving prior learnt knowledge, thereby ensuring the stability and retention of foundational knowledge in tonggu. in addition, we propose an efficient ccu retrieval-augmented generation (ccu-rag) method that significantly mitigates the propensity of hallucinations in knowledge-intensive tasks, further bolstering tonggu's performance. to summarize, our contributions are as follows: ## accn-ins dataset the complexity of manual annotation in classical chinese question answering (qa) tasks requires extensive human expertise, resulting in laborintensive processes. to alleviate the labor intensity, harnessing llm for automatic data annotation becomes a natural and efficient solution. nevertheless, llm harbors the propensity to inadvertently introduce inaccuracies during the data generation process. to address this issue, we present a semiautomated annotation method that combines classical chinese corpora using aligned llms, thereby producing reliable instructional data for specialized ccu tasks. firstly, we collected classical chinese corpora from multiple public sources, such as daizhige @xcite , textbooks, and examination papers. we then design a pipeline to transform these data into the instructional format, as depicted in figure 1 . labeled data. labeled data refers to the data equipped with well-curated labels, such as the dynasty and author of a poem. owing to its highly structured organization, a large amount of instruction data can be simply synthesized using instruction templates. specifically, for each task, we first provide 8 instruction examples handcrafted by human experts as in-context examples, prompting the aligned llm to generate a broader range of diverse instruction templates. finally, we can populate the structured data into these instruction templates to obtain instruction data in the qa format. unlabeled data. unlabeled data refers to unlabeled text segments, such as introductions to certain poets, where information like era, life experiences, and representative works are intermingled within the same text segment. we adopt a reading comprehension approach, treating the unlabeled text segments as reference materials and requiring the aligned llm to extract qa pairs from them. similarly, we use 8 human-written qa pairs as in-context examples. employing the proposed data generation pipeline and leveraging chatgpt @xcite as the aligned model, we initially generated a large corpus of classical chinese instruction data. to ensure the quality and accuracy of this generated data, we implemented a rigorous cleaning process. this process included several key steps: removing duplicate entries, standardizing punctuation marks to ensure consistency between chinese and english usage, and carefully reviewing the content for accuracy and appropriateness. after this meticulous verification and refinement process, we obtained a final dataset of 4,020,136 instances of high-quality classical chinese instruction data, among which 4,014,355 instances originated from structured text and 5,781 from unstructured text. in ## tonggu tonggu is a generalist llm specifically designed for classical chinese understanding (ccu), whose capability is built based on three core steps as shown in figure 3 . first, we perform incremental pre-training on tonggu on a mixed corpus of 4.6 billion tokens consisting of classical and modern chinese to enrich its ccu knowledge. second, tonggu is fine-tuned on millions of instruction data with the proposed peft method redundancy-aware tuning (rat), which not only enhances multi-task understanding proficiency but also ensures highly efficient instruction tuning. thirdly, we introduce ccu-rag, a task-specific retrievalaugmented generation (rag) mechanism to alleviate hallucinations in knowledge-intensive tasks. through these three steps, tonggu demonstrates its ability to handle 24 diverse ccu tasks effectively, making it a powerful tool for understanding classical chinese texts. ## experiments in our experiments, we focus on evaluating tonggu's capabilities in classical chinese understanding (ccu) across three dimensions: understanding, generation, and knowledge. ## conclusion in this paper, we introduce tonggu, a new stateof-the-art llm specifically for classical chinese understanding (ccu). our contributions include the development of the accn-ins dataset, which serves as the first publicly accessible ccu instruction dataset, and the introduction of innovative techniques such as redundancy-aware tuning (rat) and ccu-rag (retrieval-augmented generation). through extensive experiments and evaluations, we have demonstrated tonggu's superior performance in diverse ccu tasks, surpassing existing llms by a large margin in both knowledge-intensive and non-knowledge-intensive tasks. we believe that tonggu and the ac-ins dataset will serve as valuable resources for future endeavors in the ccu research community.