C hinese Frame Semantic Parsing Evaluation (2024)

Abstract: “Chinese Frame-semantic Parsing (CFSP) aims to extract fine-grained frame-semantic structures from texts, which can provide fine-grained semantic information for natural language understanding models to enhance their abilities of semantic representations. Based on the CCL-23 CFSP evaluation task, we introduce construction grammar to expand the targets, as basic units activating frames in texts, from word-style to construction-style, and publish a more challenging CFSP evaluation task in CCL-2024. The evaluation dataset consists of 22,000 annotated examples involving nearly 695 frames. The evaluation task is divided into three subtasks: frame identification, argument identification, and role identification, involving two tracks: close track and open track. The evaluation task has attracted wide attention from both industry and academia, with a total of 1988 participating teams. As for the evaluation results, the team from China University of Petroleum won the first place in the closed track with the final score of 71.34, while the team frome Suzhou University won the first place in the open track with the final socre of 48.77. In this article, we reports the key information about the evaluation task, including key concepts, evaluation dataset, top-3 results and corresponding methods. More information about this task can be found on the website of the CCL-2024 CFSP evaluation task.”

## introduction frame semantic parsing (fsp) is a fine-grained semantic analysis task based on frame semantics @xcite , its aim is to extract frame semantic structures from sentences, thereby achieving indepth understanding of events or situations within the sentence. fsp plays a pivotal role in downstream tasks such as reading comprehension @xcite ), text summarization @xcite , and relation extraction @xcite . chinese framenet (cfn) @xcite is a semantic knowledge base for the chinese language, constructed on the theoretical basis of frame semantics and developed from chinese corpus materials, referring to the framenet (fn) of the university of california, berkeley. it comprises a frame library, a sentence corpus, and a lexical unit library. currently, it contains 1398 frames, involves 18360 lexical units, and more than 100 thousand annotated sentences. currently, in the chinese framenet dataset, the lexical units activating frames are solely single words. however, in certain sentences, individual target words are insufficient to fully illustrate complex semantic scenarios. take "爱买不买" as an example, it indicates that the speaker doesn't care or is uninterested in whether the counterparty wants to purchase something. traditional methods analyze this phrase by taking words as units, introducing verbs like "爱" and "买" as target words, activating scenarios of liking or purchasing, yet falling short of expressing the true meaning of the phrase. construction grammar was first proposed by @xcite . it advocates that language knowledge consists of fixed, meaningful units, which are referred to as constructions. these units can be as simple as words or phrases, or as complex as sentences or utterances. thus, the phrase "爱买不买" is an entity that expresses semantics, which triggers the emotion directed frame. to enhance the capability of chinese frame semantic parsing, and achieve a deeper understanding of language, we have expanded our data to include constructions as "target words" for semantic frame parsing. consequently, we have launched the second chinese semantic frame parsing evaluation. ## submit results during the evaluation period, a total of 1988 teams registered for the competition, and 29 of them made it into the rematch of the b-rank track. in the end, we chose to reproduce the results of a total of 10 teams from both tracks. the table lists the scores of 6 participating teams in detail (the scores are based on the reproduction results), and the ranks are based on the final scores. for tasks 2 and 3, the table lists the accuracy, recall rate and f1 value of each team. in the following text, we will refer to the team numbers in the table to represent different teams for ease of subsequent expression. in the closed track, even though each team proposed a variety of methods to improve performance, the scores of all teams eventually fluctuated around 71.2. this reflects the difficulty for models like bert to fully represent all fine-grained semantic information under the constraint of parameter scale. in the future, we are considering introducing larger models or attempting methods such as knowledge distillation. moreover, many teams did not handle annotation data with constructions as target words in a special way. we believe this also to be one of the reasons why it's hard to further improve the final results. at the same time, we noticed that many methods did not perform as expected on task 3, while they achieved better results on task 2. we believe this is related to task 3 involving a large number of c c l 2 0 2 4 semantic roles. clearly, the model can effectively identify the related arguments of the target words in the sentence. however, the current methods cannot effectively identify the semantic roles of each argument in the scene triggered by the target word. the experimental results on the open track show that the performance of llm does not stand out in the frame identification task. in this task, t eam.4 has relatively favorable results. they used word vector cosine similarity to pre-select part of the frames instead of choosing from all the frames. we suspect this phenomenon occurs because the large language models can't distinguish subtle differences among a large number of frames. meanwhile, t eam.4 also achieved the best result in task 3, suggesting that the accuracy of frame identification has a significant impact on role identification. ## method overview after analyzing the technical reports submitted by 6 participating teams and reproducing their model results, we have compiled the main methods used by the teams, in order to analyze the advantages each team has on different tasks. in the closed track, t eam.1 adopts the token-aware virtual adversarial training (ta-vat) method to improve the performance of the model. t eam.2 proposed the extraction method for span type data for the argument identification task, which achieved good results. t eam.3 achieved pretty results by using a large language model and data augmentation techniques. in the open track, t eam.4 reduce the number of candidate frames for each target words by using word vector cosine similarity. t eam.5 build a hierarchical index rag system based on target words. these methods effectively improved the performances of large models in the task of chinese frame semantic parsing. ## summary this evaluation task, based on previous tasks, introduces construction grammar, and increases the data with a construction as the target word. it focuses on sentences in chinese where some common semantic cores are expressed through a specific structure in the sentence. this enhances the capability of the frame semantic analysis and further enables a deeper understanding of language. this evaluation is of great significance for fine-grained semantic analysis, and it has also attracted a large number of teams from the academic and industrial sectors to register for the competition. due to the high difficulty of the evaluation task, fine-grained semantics, and the target word is no longer a single vocabulary. small models lack semantic understanding when facing a large number of frames, and are unable to cope with a large number of role types in role tagging. large models lack frame semantic knowledge and cannot distinguish between subtle semantic differences among a large number of frames. they also struggle to correctly identify argument roles in sentences. this reflects that there are still tremendous development prospects for this task. in general, this evaluation targets the deficiencies of existing models in fine-grained semantic analysis, using the chinese frame semantic parsing task to assess the model's scenario depiction capabilities. future evaluations could consider expanding the data coverage fields, adding more data with constructions as target words, covering more semantic scenarios, and evaluating the model's understanding of fine-grained semantic scenarios in a more comprehensive way, further promoting the development of the chinese frame net.