CEAMC : Corpus and Empirical Study of Argument Analysis in Education via LLM s (2024)

Abstract: This paper introduces the Chinese Essay Argument Mining Corpus (CEAMC), a manually annotated dataset designed for argument component classification on multiple levels of granularity. Existing argument component types in education remain simplistic and isolated, failing to encapsulate the complete argument information. Originating from authentic examination settings, CEAMC categorizes argument components into 4 coarse-grained and 10 fine-grained delineations, surpassing previous simple representations to capture the subtle nuances of argumentation in the real world, thus meeting the needs of complex and diverse argumentative scenarios. Our contributions include the development of CEAMC, the establishment of baselines for further research, and a thorough exploration of the performance of Large Language Models (LLMs) on CEAMC. The results indicate that our CEAMC can serve as a challenging benchmark for the development of argument analysis in education.

## introduction argument mining (am) aims to automatically identify and extract the structure of inference and reasoning expressed as arguments presented in natural language @xcite . due to its significance, it has been widely incorporated into various natural language processing (nlp) tasks, such as argument evaluation @xcite , fallacy detection @xcite ) and text generation @xcite . with the surge in argumentative texts and advancements in nlp technology, am has been developed in various domains, such as court decisions figure 1 : an excerpt from an argumentative essay in ceamc. @xcite , political debates @xcite , scientific literature @xcite , social web @xcite , and online comments @xcite . these efforts have introduced various annotation schemes and datasets in conjunction with domain specificity, significantly advancing argumentation research. however, existing datasets struggle to fulfill the needs for argument analysis in education. primarily, current research either focuses on high-quality argument scenarios, such as legal texts @xcite , and peer reviews @xcite , where the argumentative texts are logically rigorous, highly professional, and persuasive. al-ternatively, it targets online scenarios like social media @xcite and online writing @xcite , where argumentative texts tend to be more fragmented and colloquial. these corpora exhibit significant differences in argument quality, textual traits, and writing styles compared to argumentative essays in educational settings, necessitating datasets that can reflect the unique complexity and nature of educational writing. furthermore, there remains a considerable discrepancy between the argument studies conducted by nlp researchers and the analysis of argumentative essays by teachers. computational approaches typically simplify arguments into generic major claims, claims and premises @xcite , which fall short of reflecting the realities of educational argumentation. in fact, argumentative essays in education usually encompass a rich variety of argument types, which is crucial for gaining insight into argument structures and support strategies. lastly, the scarcity and limited diversity of chinese argument mining datasets have somewhat constrained advancements in this field. to address the shortcomings of existing research, we introduce the chinese essay argument mining corpus (ceamc). the corpus is derived from authentic high school examination scenarios, and as illustrated in figure 1 , each argumentative essay undergoes meticulous annotation. the ceamc addresses key limitations in prior work: firstly, it bridges the gap between current corpora in fulfilling the needs of argument analysis in education. considering the pivotal role of argumentation in k12 education, we have curated a corpus of argumentative essays from high school examination scenarios, covering a variety of topics, qualities, and rich argumentative information, which adequately reflects the complexity and uniqueness of educational argumentation scenarios, and can provide a more reliable basis for argumentation assessment and instruction. secondly, it overcomes the issue of simplified argument types prevalent in previous studies. by deeply integrating argument mining research with educational practice, it provides 4 coarse-grained and 10 fine-grained argument component types, which can adeptly capture the nuances of real-world argument texts and facilitate in-depth and comprehensive analysis. lastly, by providing a diverse dataset and comprehensive experimental analyses for chinese argument mining, ceamc stimulates progress in this area. our contributions are summarised as follows: ## corpus construction this section delineates the process of collection and annotation for the chinese essay argument mining corpus (ceamc), designed for extensive argument mining research. ## experiments having constructed ceamc, we conduct an empirical study to benchmark the performances of some existing methods on the task of argument component classification against our dataset. to address this task, we split our data as summarized in ## case study as shown in ## conclusion in this paper, we introduce the chinese essay argument mining corpus (ceamc), a richly annotated and comprehensive dataset designed to address the limitations in current argument mining research. our dataset integrates argument mining research with educational practice, encompassing 4 coarse-grained and 10 fine-grained argument types, thereby overcoming the simplicity and monotony of argument types in previous studies. we also conduct several baselines with existing mainstream methods on our dataset, and the results demonstrate the superiority of llms, confirming the scaling laws. further analysis indicates that while llms possess basic argument analysis capabilities, their inherent biases and hallucinations limit their developmental potential, also showcasing the significant differences between llms' understanding of argumentation and human interpretation. therefore, how to further unleash llms' argumentation skills in education and enhance their logical reasoning abilities remains to be explored.