P hilo GPT : A Philology-Oriented Large Language Model for A ncient C hinese Manuscripts with Dunhuang as Case Study (2024)

Abstract: Philology, the study of ancient manuscripts, demands years of professional training in ex-tensive knowledge memorization and manual textual retrieval. Despite these requirements align closely with strengths of recent successful Large Language Models (LLMs), the scarcity of high-quality, specialized training data has hindered direct applications. To bridge this gap, we curated the PhiloCorpus-ZH, a rich collec-tion of ancient Chinese texts spanning a millen-nium with 30 diverse topics, including firsthand folk copies. This corpus facilitated the develop-ment of PhiloGPT, the first LLM tailored for discovering ancient Chinese manuscripts. To effectively tackle complex philological tasks like restoration, attribution, and linguistic anal-ysis, we introduced the PhiloCoP framework. Modeled on the analytical patterns of philol-ogists, PhiloCoP enhances LLM’s handling of historical linguistic peculiarities such as phonetic loans, polysemy, and syntactic inver-sions. We further integrated these tasks into the PhiloBenchmark, establishing a new standard for evaluating ancient Chinese LLMs address-ing philology tasks. Deploying PhiloGPT in practical scenarios has enabled Dunhuang spe-cialists to resolve philology tasks, such as iden-tifying duplication of copied text and assisting archaeologists with text completion, demon-strating its potential in real-world applications.

## introduction the dissemination of ancient human civilizations relied on various carriers, from inscriptions to manuscripts and the more recent printed texts @xcite . manuscripts, particularly popular from the 4-th to the 14-th century due to advancements in papermaking and its affordability, encompassed a broad array of content including official documents and folk copies on economics, literature, science, and agriculture. these * equal contribution. † corresponding authors. manuscripts are invaluable for understanding ancient societies, offering insights into both the official narratives and the daily lives of the common people. however, interpreting these vast and topicdiverse ancient manuscripts wich span over a millennium, presents a highly specialized challenge. it demands philologists to undergo years of professional training and engage in extensive literature reviews, manual transcription, and contextual analysis @xcite . aforementioned challenges render manuscript discovery a time-consuming and labor-intensive field of study. however, the requirements of philology align closely with the strengths of llms @xcite . the recent success of llms in a variety of natural language processing (nlp) tasks across diverse fields has highlighted their effectiveness @xcite @xcite . nevertheless, the potential of llms to interpret ancient languages like classical chinese remains largely untapped @xcite . this oversight is primarily due to three major constraints: (1) insufficient training corpus. despite the abundance of high-quality english, chinese, and multilingual llm training corpora @xcite , there remains a significant gap in the availability of a large and diverse corpus for classical chinese. (2) linguistic complexities of classical chinese. the substantial differences in linguistic peculiarities between ancient and modern chinese, such as words meaning shift and syntactic inversions, has complicate text interpretation. these phenomenons demand a framework that integrates philological insights to fully exploit the capabilities of llms. to address these hurdles and fully leverage the capabilities of llms in classical chinese studies, we compiled philocorpus-zh, an extensive collection of ancient manuscripts. this pioneering archive systematically categorizes thousands of documents across 30 varied topics, including previously unorganized firsthand folk copies, enriching the understanding of ancient linguistic practices. recognizing the profound linguistic disparities between classical and modern chinese, including grammatical evolution and syntactic variations, we developed the philocop framework. the essence of philocop lies in mimicking philologists internally interpreting ancient manuscripts, i.e., entity identification, context-implicit relation reasoning, and relation-aware transcription. entity identification extracts, differentiate and categorize entities based on their roles within the text. contextimplicit relation reasoning uncovers the underlying semantic connections and facilitate a form of cognitive mapping. relation-aware transcription performs lexical standardization by adjusting for linguistic shifts and anomalies inherent in classical chinese. eventually, philocop concludes with a synthesis prompt that requires the model to integrate its observations from the previous steps into a cohesive interpretation. furthermore, we propose philobenchmark to evaluate the efficacy of llms from the perspective of philological research requirements. it comprises 9 targeted tasks closely related to ancient document analysis, including manuscript restoration, attribution, and philology-oriented qa, along with content reasoning. building on all previous substantial resources, we developed philogpt, the first llm tailored to handle a variety of analyti-cal tasks for classical chinese. through extensive experiments, we demonstrate that philogpt, in comparison to general llms, excels in the discovery and analysis of ancient documents, such as manuscript text pieces conjugation and historical reasoning. philogpt is applying in dunhuang manuscript research, demonstrating potential in tasks, such as distinguishing original and copied texts, and resolving debate among missing characters in manuscripts. the explored real-world applications of philogpt showcase its versatility and value as an indispensable tool for scholars, accelerating discovery and fostering interdisciplinary collaboration in classical chinese studies. the highlights of the paper are: 1. diverse original ancient chinese corpus: we curated the philocorpus-zh, a collection of ancient chinese manuscripts. these original documents, complemented by related modern research works, were systematically organized into four majors adhering to bibliographic principles. language models for ancient texts. several noteworthy projects have successfully employed language models pre-trained on ancient language corpora for specialized tasks @xcite @xcite . for instance, @xcite utilized ancient greek inscriptions to support textual restoration, geographical and chronological attribution. similarly, hue's work @xcite involved training bert-like models on ancient korean documents to address challenges such as chronological attribution, topic classification, named entity recognition, and summary retrieval. @xcite harnessed annals for developing model that translate hanja, ancient korean, into contemporary korean and english. as for linguistically limited resources, @xcite masked language models have been used to predict missing tokens with oracc dataset. however, these earlier efforts, predating the recent breakthroughs in llms, relied heavily on the accessibility of ancient language training corpora @xcite @xcite . this resulted in limited resources, constraining the models to a narrow range of specific applications unsuitable for llm pre-training @xcite . in contrast, we introduce the first ancient chinese llm designed to tackle a broad spectrum of downstream tasks, leveraging the extensive and diverse corpus from the philocorpus-zh and philobenchmark. ## philogpt we detail the construction of philogpt. section 3.1 describing the acquisition and organization of philocorpus-zh, which serves as foundation for training. section 3.2 discuss the development of philobenchmark, which comprises various downstream tasks formulated to test the efficacy of llms in handling ancient chinese. in section 3.3, we explore the philocop framework, which is designed to enhance llm's ability to mimic philologists interpreting ancient texts. section 3.4 provides details with the implementation of philogpt. ## dunhuang manuscript case study taking a step further, we conducted an analysis using dunhuang manuscripts as a case study to explore two specialized tasks of philology research practice. our objective is to show that, following pre-training and instruction tuning, philogpt can provide researchers with enhanced flexibility to perform comprehensive text information mining. this approach not only facilitates deeper textual analysis but also leads to potential discoveries. ## conclusion in this paper, we introduce philogpt, the first domain-specific llm designed to tackle a range of tasks related to the discovery of ancient chinese manuscripts. we developed the large-scale philocorpus-zh, which encompasses diverse topics of original ancient texts and associated scholarly research. to align with the unique characteristics of these languages, we proposed the philo-cop framework. from a philology perspective, our philobenchmark serves as an evaluation standard for llms tailored to ancient chinese. extensive experiments and case studies with dunhuang manuscripts have demonstrated philogpt's potential for ancient document discovery. for future work, we may extend philogpt's capability based on more applications and feedback when applied in other manuscripts. we also consider incorporating retrieval-augmented generation (rag) with specific documents to mitigate hallucination problems. limitation: due to historical reasons, official documents are often better preserved and more complex than folk texts, resulting in some data bias and imbalance. we anticipate that the discovery of additional manuscripts, such as those recently unearthed in xinjiang, will help mitigate this issue. moreover, ancient documentary studies place a high premium on factual accuracy. thus, findings from our model still require secondary verification by philologists and must undergo peer review. additionally, considering that manuscripts often incorporate image modality, fine-tuning a multimodal llm that integrates text styles with document materials could enhance accuracy in attribution tasks. multi-modality input would provide scholars with greater flexibility when deploying llms.