C hinese Parataxis Graph( CPG ) Parsing Based on Large Language Models (2024)

Abstract: “This paper presents the work submitted for the 23rd China National Conference on Computational Linguistics(Evaluation Workshop)(CCL24-Eval), focusing on the Chinese Parataxis Graph (CPG) Parsing task. CPG represents Chinese natural language hierarchically through relational triplets, providing a consistent representation for linguistic units of varying levels. Our approach has used large-scale language models through full fine-tuning, achieving the result with F1 value at 71.6% in the contest and 74.76% after the contest. Furtehrmore, our team has proposed a combined model that integrates multiple LoRA fine-tuned medium-scale models after the contest. This approach is able to minimize the time and space consumption while keeping the performance of CPG construction task relatively high.”

## preliminaries in this task, we use the wenxin qianfan's large-scale model platform of baidu ai cloud and the ernie speed closed-source model for fine-tuning. the model version is ernie-speed-8k, released on february 5, 2024. it is a high-performance industry-level knowledge-enhanced large language model developed by baidu. this model has the following advantages for this task: chinese context: ernie speed is specially optimized and trained for the chinese context, allowing it to better understand and process chinese text. context handling: ernie speed has a high context length processing capability in inference scenarios, enabling it to better handle contextual dependencies in chinese text. lightweight design: for this task, we introduced the ernie-speed-8k version, which is a lightweight large language model with strong performance and high processing speed. efficient training: due to its lightweight design, ernie speed has a relatively short training time, improving model iteration efficiency. during the task, we also tried fine-tuning other large language models provided by the wenxin qianfan's large-scale model platform, such as baidu's lightweight large language model ernie lite and the qianfan-chinese-llama-2-13b-v1 large model, which is based on llama-2, with an expanded chinese vocabulary and enhanced pre-training and instruction fine-tuning using large-scale chinese-english data. although these models also achieved good performance, the final results were not as good as the ernie speed model. we believe this is mainly due to the parameter scale, so we ultimately chose the ernie speed model. ## single-llm task in this section, we mainly present the methods we implemented in the ccl24-task2 competition. the core process of fine-tuning large models includes prompt engineering and parameter adjustment. this section will focus on demonstrating the methods of prompt engineering and parameter adjustment. ## multi-llm ensembling and tuning in this section, we introduced some smaller-scale llms into the task. when using these smallerscale llms separately, they perform not as well as large-scale models. our approach can be mainly described as two parts: multi-model reinforcement circum-block and iu-block. through these two tools, we came up with a similar result compared with using large-scale llm. ## conclusion this paper proposes a novel method for constructing cpg, significantly improving the accuracy of their construction. by fine-tuning large language models, a model capable of constructing cpg with high accuracy was developed. considering the substantial time and space requirements of large language models with extensive parameters, this paper explores the use of smaller-scale models for this task. a system integrating multiple small-scale models and a fusion runtime system was constructed. this approach successfully used smaller-scale models to construct cpg, achieving similar results to larger models but with significant savings in both runtime and space utilization.