Discourse Analysis of Argumentative Essays of E nglish Learners Based on CEFR Level (2023)

Abstract: In this paper, we investigate the relationship between the use of discourse relations and the CEFR-level of argumentative English learner essays. Using both the Rhetorical Structure Theory (RST) and the Penn Discourse TreeBank (PDTB) frameworks, we analyze essays from The International Corpus Network of Asian Learners (ICNALE), and the Corpus and Repository of Writing (CROW). Results show that the use of the RST relations of Explanation and Background, as well as the first-level PDTB sense of Contingency, are influenced by the English proficiency level of the writer.

## introduction in a world where over 7,000 languages are used, much research has focused on improving methods to teach and learn natural languages. in particular, the field of natural language processing (nlp) has a long history of developing tools to assist language learners and reduce learning barriers. previous works on surface linguistic features and language learning, such as @xcite , @xcite , and @xcite have shown significant difference in discourse usage across textual genre and simplicity level. to our knowledge, very few studies have investigated the relationship between discourse structures and language learning. corpus research on the use of discourse structures among different cefr levels can provide valuable insights into how well language learners are able to organize and convey their ideas in written or spoken language. such an analysis can also identify common patterns of language use that are particularly challenging for learners at different cefr levels, leading to the development of more effective teaching materials and strategies that target learners' specific needs @xcite , while simultaneously reducing the workload of human graders @xcite . findings can also inform the development of more reliable assessment tools that accurately measure learners' proficiency in the use of discourse structures. accurate assessment is essential for learners to identify their strengths and weaknesses and make informed decisions about their language learning goals and strategies. in this paper, we investigate the usage of discourse relations using the rhetorical structure theory @xcite and the penn discourse treebank @xcite frameworks to discover trends in their usage in argumentative english learners across various proficiency levels. results show that the rst relations of explanation and background are statistically used more often by writers with a lower cefr language level, and the use of the pdtb relation of contingency decreases as cefr level increases. ## datasets in order to analyze discourse structures across cefr levels, we aimed for texts long enough to have rich discourse structures. we used two datasets of argumentative essays: ic-nale @xcite and crow @xcite . we did not use the datasets of schmalz and brutti (december 2021) (see section 3) as these largely consist of short 2-3 sentence texts. the first dataset we used was the international corpus network of asian learners (ic-nale) @xcite . the icnale dataset used the ets mapping (see section 2.2) to convert toefl scores into cefr scores. the dataset contains essays from 5 cefr levels: a2, b1.1, b1.2, b2, and c2. in order to be compatible with the second dataset, we merged b1.1 and b1.2 instances to create a single b1 label. the second dataset we used was the corpus and repository of writing (crow) @xcite . for the sake of consistency in genre, we only used the argumentative papers from this dataset for comparison with the icnale dataset. the crow dataset is not labelled with cefr scores, but rather with toefl scores. for comparative purposes, we used the ets mapping on the crow dataset to determine the cefr score. ## discourse analysis in order to extract reliable discourse information from the datasets, we used two publicly-available discourse parsers from each framework. for rst, we used the @xcite and the heilman and sagae (2015) parsers. we chose these parsers because they use the same set of rst relations, and they achieve high performance for relation tagging. @xcite achieves an fscore of 57.4% on the rst-dt test set, @xcite achieves 59.7%, while human performance is 65.8% @xcite . for pdtb parsing, we used the @xcite and the @xcite parsers due to their high performance and availability. we parsed the icnale and the crow datasets (see section 4) with all four parsers using all 16 rst relations and the 4 level-1 pdtb relations. the outputs of both rst parsers and both pdtb parsers were then compared. in order to have significant statistics, we ignored any discourse relation that appeared in less than 10% of the documents. these included the rst relations of evaluation, summary, topic-comment and topic-change. this left us with the 12 most frequent relations: attribution, back-ground, contrast, cause, comparison, condition, elaboration, enablement, explanation, joint, manner-means, and temporal. all pdtb level 1 relations appeared in more than 10% of the documents, hence all were considered. we computed the average frequency of each rst and level 1 pdtb relation for each cefr label in the dataset: a2, b1, b2, and c2. to determine if there was a statistical difference in the usage of these relations across cefr levels, we ran a two-tailed t-test with a p-value of 0.95, comparing a2 against c2, b1 against c2, and b2 against c2. ## conclusion and future work in this paper, we investigated the use of discourse information in essays across language proficiency levels. a corpus analysis with state-of-the-art rst and pdtb parsers showed a relation between learner cefr level and the rst relations of ex-planation and background. using the mapping of pdtb and rst proposed by @xcite , we showed a decrease in use of con-tingency relations in one dataset at the c2 level. while discourse relations frequency would not be the sole factor for automatic cefr assessment tools, the findings of this analysis could serve as a feature for improving the accuracy of these classifications. future work could look for differences in discourse relations based on the first language of the english learner, while accounting for the learner's cefr level. the corpora used in this study provide the native language or country of origin of the learner. previous work has begun mapping pdtb-3.0 @xcite relations to rst rela-tions, such as @xcite , so future work could use inter-framework mapping with the updated pdtb. finally, future research could expand its focus beyond discourse analysis in argumentative texts and delve into discourse structures across various text genres, including narratives, academic papers, and conversational dialogues. notably, recent work has explored this avenue in the realm of spontaneous spoken dialogue @xcite . by extending the examination of discourse relations and connectives to diverse genres, a more comprehensive understanding of language learning can be achieved, shedding light on genre-specific discourse patterns.