The Fourth C hinese A bstract M eaning R epresentation Parsing Evaluation (2024)

Abstract: “Abstract Meaning Representation has become a key research area in sentence-level semantic parsing within natural language processing. Substantial progress has been achieved in various NLP tasks using AMR. This paper presents the fourth Chinese Abstract Meaning Representation parsing evaluation, held during the technical evaluation task workshop at CCL 2024. The evaluation also introduced a new test set comprising Ancient Chinese sentences. Results indicated decent performance, with the top team achieving an F1 of 0.8382 in the open modality, surpassing the previous record at CoNLL 2020 by 3.30 percentage points under the MRP metric. However, current large language models perform poorly in AMR parsing of Ancient Chinese, highlighting the need for effective training strategies. The complex syntax and semantics of Ancient Chinese pose significant challenges. Additionally, optimizing transfer learning techniques to better apply knowledge from Chinese Mandarin to Ancient Chinese parsing is crucial. Only through continuous innovation and collaboration can significant advancements in both Ancient Chinese and Chinese Mandarin AMR parsing be achieved.”

## introduction with the growing maturity of morphological and syntactic analysis techniques, natural language processing (nlp) has advanced to the level of semantic analysis. sentence-level meaning parsing, in particular, has become central to semantic analysis research. to address the challenges of whole-sentence semantic representation and the domain-dependent nature of sentence semantic annotation, @xcite proposed a domain-independent whole-sentence semantic representation method called abstract meaning representation (amr). amr abstracts the meaning of a sentence using a single-rooted, acyclic, and directed graph, predicting the semantic structure of the targeted sentence. large-scale corpora have been constructed for amr, and several international conferences have been held to evaluate amr semantic parsing tasks. the conference of conll 2020 featured a cross-lingual track with five languages, and it was the first time chinese was included. however, parsing chinese using amr has its challenges due to significant syntactic and semantic differences between chinese mandarin and english. to address these issues, @xcite introduced several major changes to develop chinese abstract meaning representation (chinese amr, camr), enhancing its ability to parse chinese effectively. similar to amr, the camr corpus has begun to take shape and played an important role from conll 2020. ## evaluation task our evaluation task is to parse input sentences and output camr graphs of the targeted sentences using data from the camr corpus. it is noteworthy that the alignment of concepts and relations, as well as additional semantic role labels, have been incorporated into camr to better capture the unique characteristics of chinese. the evaluation task at conll 2020 did not utilize the alignment of concepts and relations. therefore, to address this issue, our previous camrp 2023 evaluation @xcite ©2024 china national conference on computational linguistics published under creative commons attribution 4.0 international license c c l 2 0 2 4 第二十三届中国计算语言学大会论文集，第160页-第171页，太原，中国，2024年7月25日至28日。 卷3：评测报告 (c) 2024 中国中文信息学会计算语言学专业委员会 adopted a newly designed metric named @xcite . this metric includes the alignment of concepts and relations, aiming to better evaluate the performance of automatic parsing. and for the sake of consistency and accuracy, align-smatch remains the main metric this year. overall, camrp 2024 is a follow-up and extension of camrp 2023, with key differences including the addition of a blind test set containing 2,177 ancient chinese sentences. this extension aims to further test and refine our parsing methods, ensuring robust performance across different eras and styles of chinese language. ## data set the camr corpus has been constructed through a collaboration between nanjing normal university and brandeis university since 2015 @xcite . over the past two years, we have utilized these datasets to evaluate the progression of camr parsing. to further assess the generalization performance of parsers, we have introduced a new dev set b and a blind test set (test c), consisting of 500 and 2,177 ancient chinese sentences, respectively. consequently, our evaluation includes one training set, two development sets, and three test sets, as shown in the train set, dev set a, and test a for camrp 2024 are sourced from camr v2.0, which is available through the linguistic data consortium (ldc) 1 . this dataset, derived from the chinese tree bank 8.0, comprises a total of 20,078 chinese sentences. consistently, the dataset includes training, development, and test sets, which have been proven to be of high quality in the evaluation tasks at camrp 2022 @xcite and camrp 2023. ## evaluation design in the spirit of innovation and comparison, camrp 2024 includes three evaluation metrics and two modalities. ## conclusion and future work this paper provides an overview of the fourth chinese abstract meaning representation parsing evaluation at ccl 2024. camrp 2024 continued with the align-smatch metric to better assess the parsing performance of each participating system. and it was the first time that ancient chinese amr parsing has been introduced into our evaluation series. three teams submitted their results, each presenting inspiring and motivating work. some teams advanced prior methods with creative approaches, while others thoroughly explored the capabilities of llms. notably, the gdufe team achieved a score of 0.8382 in the mrp metric, surpassing the best record from conll 2020 by 3.30 percentage points. decent progress has been made in both mandarin and ancient chinese amr parsing, marked by notable achievements and innovative methodologies. however, relation prediction and alignment remain challenging and act as bottlenecks in the development of chinese amr parsing. despite remarkable breakthroughs in some aspects, leveraging the power of llms and maximizing their potential in transfer learning towards ancient chinese amr parsing seem to be critical areas for further improvement. also, the complexity of semantic relation identification and alignment within amr structures necessitates focused attention and the development of innovative techniques. in our future endeavors, we are dedicated to advancing chinese amr parsing through comprehensive initiatives. this involves hosting evaluation tasks to facilitate the assessment and benchmarking of parsing models. additionally, we aim to construct and refine models specifically tailored to the complexities of both mandarin and ancient chinese, thereby propelling the field of semantic analysis forward. by focusing on relation prediction and alignment, we seek to address current challenges and enhance the performance and understanding of chinese amr parsing. through ongoing research, collaboration, and innovation, we aspire to develop robust and accurate parsing models, pushing the boundaries of semantic analysis further.