Construction of CFSP Model Based on Non-Finetuning Large Language Model (2024)

Abstract: “Chinese Frame Semantic Parsing (CFSP) is an important task in the field of Chinese Natural Language Processing(NLP). Its goal is to extract the frame semantic structure from the sentence and realize the deep understanding of the events or situations involved in the sentence. This paper mainly studies the application of Large Language Model (LLM) for reasoning through Prompt Engineering without fine-tuning the model, and completes three subtasks of Chinese Framework Semantic Parsing tasks: frame identification, argument Identification and role identification. This paper proposes a Retrieval Augmented Generation (RAG) method for target words, and constructs more refined sample Few-Shot method. We achieved the second place on the B rankings in the open track in the “CCL2024-Eval The Second Chinese Frame Semantic Parsing”competition*.”

## introduction chinese frame semantic parsing (cfsp) is a research method based on frame semantics (charles j. @xcite , which is based on the semantic representation and annotation of chinese framenet (cfn) @xcite @xcite , and achieves the purpose of semantic parsing by extracting the frame semantic structure of sentences @xcite . this method is of great significance for a series of downstream tasks such as reading comprehension @xcite @xcite ), text summarization @xcite , and relationship extraction @xcite . ## related work due to the late appearance of llm, the relatively novel architecture, and the rapid development and updating speed, there are few researches on the application of llm in the chinese frame semantic parsing tasks. yang @xcite represents the first attempt at leveraging large pre-trained language models (llm) for sparql generation to address chinese knowledge graph question answering. @xcite tested the effect of chatgpt on fi task in zero-shot and few-shot scenarios, and tried to design the cot to carry out multiple rounds of dialogue, guiding chatgpt to better complete the tasks of ai and ri, but the final performance was not ideal as well. ## other and future work in terms of model optimization, we analyze and discuss the performance of llm in chinese frame semantic parsing tasks. combined with the relevant research of @xcite , we believe that we can try to use baai general embedding (bge) @xcite for semantic embedding in subsequent research, and by using the complementary nature of different ranking algorithms, the reciprocal rank fusion (rrf) @xcite , which has excellent results in previous studies, can get better search results by mixing their ranking results, so as to obtain better data, and then improve the analytical ability of the model. in addition, we used the qwen1.5-7b model ‡ in the local validation phase to evaluate the first 1000 data sets in the test (b) set. using the above-mentioned statement processing method, we have achieved an accuracy rate of 11.2%, and it is estimated that the accuracy rate of more than 45% can be obtained on the whole test (b) set. it can be seen that after reducing the difficulty of the task, llm with smaller parameters can also perform well. in the future, we can try to use more open source llm combined with fine-tuning operations in the corresponding fields to achieve better performance. finally, due to the limitation of funds and competition time, we only used the gpt-3.5-turbo model which released a year ago for experiments. we believe that the use of more advanced llm (such as gpt-4o § ) will bring considerable performance improvement in the future. ## conclusion in this study, we have developed a hierarchical index retrieval augmented generation (rag) system based on target words to improve the efficiency and accuracy of chinese frame semantic parsing (cfsp). by leveraging target word information, our approach filters and indexes documents, which significantly enhances the performance of semantic parsing tasks. the key components of our system include data preparation, text tokenization using the hanlp tokenizer, index creation, data insertion into elasticsearch (es) indices, cluster configuration, and index population. our hierarchical indexing method ensures efficient and scalable retrieval, which is critical for handling large datasets in cfsp. the utilization of the bm25 retrieval algorithm further optimizes the quality of samples in few-shot scenarios, ensuring balanced and relevant data for the model. by refining the input and post-processing stages, we reduce the computational load on the model, allowing it to focus on semantic understanding rather than mapping relationships.