Leveraging LLM s for C hinese Frame Semantic Parsing (2024)

Abstract: “We participate in the open track of the Chinese frame semantic parsing (CFSP) task, i.e., CCL24Eval Task 1, and our submission ranks first. FSP is an important task in Natural Language Processing, aiming to extract the frame semantic structures from sentences, which can be divided into three subtasks, e.g., Frame Identification (FI), Argument Identification (AI), and Role Identification (RI). In this paper, we use the LLM Gemini 1.0 to evaluate the three subtasks of CFSP, and present the techniques and strategies we employed to enhance subtasks performance. For FI, we leverage mapping and similarity strategies to minimize the candidate frames for each target word, which can reduce the complexity of the LLM in identifying the appropriate frame. For AI and RI subtasks, we utilize the results from small models as auxiliary information and apply data augmentation, self-training, and model ensemble techniques on these small models to further enhance the performance of subtasks.”

## introduction chinese frame semantic parsing (cfsp) is a fine-grained semantic parsing method based on chinese framenet (cfn) @xcite , which is first proposed during ccl2003-eval @xcite . it aims to extract frame-semantic information from a sentence @xcite . previous works have shown that cfsp can help various downstream tasks, including text summarization @xcite , reading comprehension @xcite , relation extraction @xcite , etc. cfn, proposed by shanxi university, is constructed based on the theory of frame semantics, with english framenet @xcite as a reference and chinese realistic corpora as the data foundation @xcite . it is a structured representation of knowledge that establishes relationships between vocabulary and concepts through frames. in the ccl2023-eval cfsp task, part of the cfn data is released for the first time. compared to the previous evaluation, this evaluation adds information on constructional target words. constructional target words refer to the words within a construction that are considered central or core. these words play a crucial role in activating the meaning of the entire construction. a construction is a fixed expression unit in a language that has specific form and meaning @xcite . it can be a single word, a phrase, or even a sentence. for example, in the phrase "爱买不买(love to buy or not)", the construction is "爱+v+不+v (love + v + not +v)", where "v" represents a verb. ## methods prompting is a recently-mainstream for llm and llm-friendly evaluation method @xcite . the performance of a task is highly related to the information contained in the prompt. for this, we design appropriate prompts for the three subtasks. for fi subtask, we need to provide candidate frames for each target word in the prompt. to mitigate the issue of excessively long prompts, we obtain a small number of candidate frames for each word based on the mapping relationships between target words and frames in the given training and dev data. after providing the target word and its triggered frame in the prompt, we can parse the ai task using llm. similarly, after providing the target word and its corresponding arguments in the sentence in the prompt, we can parse the ri task. due to the relatively low performance of the llm in the ai and fi subtasks, we use small models to obtain fi and ai results, which are then used to assist ai and ri subtasks, respectively. to enhance the performance of the small models, we employ data augmentation techniques (including copying and generating data) for the fi task and self-training methods for the ai task. in both tasks, we adopt model fusion techniques (i.e., multiple model voting) to further improve the model performance. ## conclusion in this evaluation task, we construct appropriate prompts in gemini for the three subtasks of cfsp for parsing. for the fi task, we use the mapping between the target words and frames to obtain candidate frames for each target word. for the ai and ri subtasks, we leverage the results from the small model to enhance the prediction performance of the llm. ultimately, we achieve first place in the open track of the cfsp. however, our work still has some notable limitations. for instance, in the fi subtask, the limited candidate frames we provided may constrain the choices of the llm. on the closed track, the small model has achieved 72.82%, 86.97%, and 60.27% respectively for the three subtasks, indicating ample room for improvement for the large model in cfsp task. in the future, we can enhance the performance of the three subtasks by providing more examples or utilizing information provided by the framework, such as frame definitions and argument definitions. the output of the large model is diverse, and we can further enhance its performance by generating multiple prediction results and using voting mechanisms. c c l 2 0 2 4