Analyzing Culture-Specific Argument Structures in Learner Essays (2022)

Abstract: Language education has been shown to benefit from computational argumentation, for example, from methods that assess quality dimensions of language learners’ argumentative essays, such as their organization and argument strength. So far, however, little attention has been paid to cultural differences in learners’ argument structures originating from different origins and language capabilities. This paper extends prior studies of learner argumentation by analyzing differences in the argument structure of essays from culturally diverse learners. Based on the ICLE corpus containing essays written by English learners of 16 different mother tongues, we train natural language processing models to mine argumentative discourse units (ADUs) as well as to assess the essays’ quality in terms of organization and argument strength. The extracted ADUs and the predicted quality scores enable us to look into the similarities and differences of essay argumentation across different English learners. In particular, we analyze the ADUs from learners with different mother tongues, different levels of arguing proficiency, and different context cultures.

## introduction analyzing the argument structure of a text helps understand the individual points being made and the relationships between these points to identify the overall position that the writer supports @xcite . in practice, manual annotation of argument structure is a skilled work; the laborious and time-consuming process behind would make large-scale studies challenging. this is undoubtedly true for second-language writing research. especially studies investigating language learners' use of arguments in the essays usually need to determine the occurrence of individual argument components, such as @xcite and @xcite , see section §2 for details. research on computational argumentation has drawn increased interest in recent years, with argumentative writing support being one of the main envisioned applications @xcite . computational methods to automatically mine argumentative discourse units (adus) and the relations between these units enable various applications in the context of language education @xcite . argument mining has been performed effectively on persuasive learner essays @xcite , and argument quality assessment has been aided with claim generation @xcite . given the close connection between argument structure and text quality @xcite , argumentative learner essays have also been studied in terms of quality dimensions such as organization @xcite and argument strength @xcite . so far, however, little attention has been paid to the cultural diversity of language learners with respect to the different argument structures they form. cultural variation may originate from different geographical origins, mother tongues, societal systems, the ways people communicate in these systems, and many other aspects @xcite . some of these aspects may be easy to access, others barely. either way, culture is recognized known as a factor affecting the persuasiveness of arguments and the organization of ideas of language learners @xcite . at the same time, the extent to which culture is reflected in a given text may depend on the learner's level of language proficiency. bearing these points in mind, this paper goes beyond previous studies of learner argumentation, analyzing differences in the structure and quality of essay argumentation of culturally diverse learners. 1 to learn about cultural differences, we first build statistical and neural nlp models, following previous research, to classify adus in learner essays and to extract common structural argument patterns in terms of sequences of types of adus in a paragraph (hereafter, adu flows). moreover, in line with the a study of the impact of argument structure on text quality @xcite , we develop models to score the essays in terms of their organization and argument strength. first, a state-of-the-art approach (prakash and madabushi, 2020) is adapted for mining adus from english texts, trained on the 402 persuasive student essays of @xcite as well as on a corpus of reddit changemyview discussions @xcite . then, two scoring models are learned on 1000 essays from the icle corpus @xcite @xcite . the trained models are compared with two strong baselines, including the current state of the art on the the respective tasks (section §4), in order to get an idea of their reliability. the models then serve as the basis for the main analysis carried out in this paper. in particular, applying the trained models to the entire icle corpus, we contrast the most frequent adu flows that learners use depending on their cultural background, reflected in the author's first language, and in terms of whether that is a high or lowcontext language @xcite , as well as their proficiency levels, reflected in the scores of organization and argument strength (section §5). in addition, we analyze the macro-structure of the essays to classify essays into climactic/anti-climactic (suzuki, 2010) and horizontal/vertical @xcite . the results suggested that the most frequent adu flows and their macro-structures correlate with the cultural background and language proficiency of learners, revealing various patterns. for example, speakers of european languages tend to use similar adus flows, and among them, speakers of germanic language have even more similar adus flows. altogether, we make three contributions in this analysis-oriented paper: the code of our experiments is available at: https://github.com/webis-de/ argmining22-culture-arg . ## related work most research on language learners' argumentation competence investigates essays of a small number of esl learners in their own countries, such as @xcite and @xcite . @xcite study how korean students use toulmin elements in their english essays. the results show that korean students relied heavily on claim and data due to the korean culture-specific discourse. @xcite and @xcite analyze toulmin elements in chinese students in their english argumentative writings. the researchers find that chinese students mainly use data and subclaim but they barely use counterarguments and rebuttal to consider opposing views. in addition, influenced by chinese culture, taiwanese students prefer backing and modal besides data and claim @xcite . the study of @xcite on iranian graduate learners of english also shows that the students are prone to use data and claim the most. on the other hand, numerous studies @xcite @xcite @xcite have investigated the effects of culture on persuasive essays produced by native and non-native learners. for example, @xcite studies the differences in korean and american editorials while suzuki (2010) conducts a similar study that compares the arguments written by japanese and american. the results show that non-native students tend to transfer their first language rhetorical style into their english writing. particularly, non-native speakers tend to use climactic and vertical macro-structures while english speakers tend to use anti-climactic and horizontal macro-structures. these terms are elaborated in section §5.1. the above mentioned studies suggest the learners' argument structures would differ depending on their mother tongue backgrounds. language is the carrier of culture, and cultural features can be reflected in one's writing. hall (1976) suggests the categorization of cultures into high context versus low context cultures 2 in order to understand their basic differences in communication style and cultural issues. in fact, the communication styles of people from different cultures range from explicit to ambiguous @xcite @xcite . that means one culture is more or less high-context (or low-context) than the other. zou (2019) shows various cultures on a continuum, from where a tendency is observed that most northern european countries are low-context whereas asian countries are more high-context. similarly, senthamarai and chandran (2015) classifies north america and much of western europe are lowcontext while middle east, asia, africa, and south america are high-context. given that the "thought patterns" @xcite are expected as an integral part of their communication, the "cultural thought patterns" @xcite may affect the persuasiveness of arguments and organization of ideas @xcite . with a better understanding of how learners from different cultural groups write arguments, language teachers could help learners enhance the quality of argumentative writing. unfortunately, the impact of cultural differences on argument forms of learners from diverse mother tongue backgrounds is understudied. typically, such studies rely heavily on manual annotation of argument structures. it is a skilled work. the laborious and time-consuming process would make large-scale studies challenging. luckily this thorny issue could be addressed using argument mining technology. it has enabled a variety of applications @xcite . to achieve our goal, two main tasks are performed. mining argumentative discourse units 2 high and low-context will be discussed in section § 5.3. we build on the setting of @xcite , but analyze a large number of different learner populations. to the best of our knowledge, we are the very first paper aiming at providing an indepth analysis of adus produced by learners from various cultures, and revealing the differences and similarities of argument structures among different learner populations and proficiency levels from the perspective of computational argumentation. ## method this section presents the computational methods that we develop for identifying argumentative discourse units (adus) in student essays as well as for scoring quality dimensions of the essays. we discuss how the methods are trained and what features are used. figure 1a and b illustrate their usage. ## results we seek to apply adu identification and quality scoring in order to analyze the whole icle corpus with 6,085 essays in total. this section discusses the effectiveness of the trained models. ## analysis the methods we developed and evaluated in the previous sections mainly serve as a means to carry out the analysis presented in this section. in particular, we applied the methods to all essays from the icle corpus @xcite . based on their output, we analyze culture-specific argument structures in terms of what adu flows learners use depending on three cultural aspects: the learners' first language, their arguing proficiency, and their cultural context. for each aspect, we also discuss the macro structures used in different cultures. ## conclusion this study aims to advance the understanding of language learners' argumentation with respect to cultural differences. to investigate argument structures in learner essays, we have built models for adu identification and quality scoring, aiming at analyzing all icle essays. the results reveal differences and similarities of argument structures across english learners from different cultural backgrounds and proficiency levels. the empirical findings from this study make two significant contributions to educational applications. first, argumentation technology can be of effective assistance in reducing the manual annotation workload as well as in expanding the research scope. second, the analysis helps gain a comprehensive understanding of the argument structures produced by learners from different language backgrounds. it appears that culture would have a substantial influence on learners' argumentation patterns in terms of argument strength and organization. our preliminary findings could be a doorway to the intercultural understanding of language learners' argument structures. for example, future research could usefully explore appropriate instructional approaches to help learners from different cultural backgrounds.