Transfer Learning of Argument Mining in Student Essays (2024)

Abstract: This paper explores the transferability of a cross-prompt argument mining model trained on argumentative essays authored by native English-speaking learners (EN-L1) across educational contexts and languages. Specifically, the adaptability of a multilingual transformer model is assessed through its application to comparable argumentative essays authored by English-as-a-foreign-language learners (EN-L2) for context transfer, and a dataset composed of essays written by native German learners (DE) for both language and task transfer. To separate language effects from educational context effects, we also perform experiments on a machine-translated version of the German dataset (DE-MT). Our findings demonstrate that, even under zero-shot conditions, a model trained on native English speakers exhibits satisfactory performance on the EN-L2/DE datasets. Machine translation does not substantially enhance this performance, suggesting that distinct writing styles across educational contexts impact performance more than language differences.

## introduction argumentative writing is a central skill to succeed across school subjects @xcite and automated feedback is an effective way to foster writing skills @xcite . figure 1 shows an example of providing students with feedback by highlighting different argumentative elements, such as lead, position, claim and conclusion in their writing. such feedback offers guidance to students for enhancing the structure of their essays. however, training a dedicated feedback model for each new task could incur substantial costs. one approach to mitigate this expense is to transfer a pre-trained model to new datasets. while existing research highlights model transferability across different writing prompts @xcite , no research demonstrates whether employing english argument mining models across languages and different educational contexts yields consistent performance. such educational contexts for argumentative writing can be specified according to two dimensions: native vs. foreign language instruction on the one hand and independent vs. integrated writing tasks on the other hand. we first have a closer look at the differences between l1 and l2 writing. in l1 teaching contexts, the emphasis is primarily on content, whereas in l2, the focus is on language acquisition and structure. these differences are also reflected in distinct cognitive models and therefore writing styles @xcite . beyond obvious characteristics such as spelling and grammar errors in l2 writing, like the misspelled advetisments and the subject-verb disagreement exemplified in figure 1 , prior studies also unveiled that non-native english writers tend to craft shorter sentences and employ fewer hedges (e.g., probably, may) to moderate the strength of their claims, in contrast to native english speakers @xcite . moreover, l2 writers prefer a more straightforward argumentation structure and often avoid counter-arguments @xcite . as for the second dimension, in independent tasks, individuals are typically provided with a specific writing prompt and are required to formulate their essays based solely on their thoughts, experiences, and knowledge. for example, the independent writing prompt of en-l2 in figure 1 is: do you agree or disagree with the following statement? television advertising directed toward young children (aged two to five) should not be allowed. use specific reasons and examples to support your answer. in integrated tasks, writers are presented with one or more texts related to a particular topic and then asked to synthesize information from the provided texts and incorporate it into their writing. for in- ## data in our experiments, we work with three different datasets: persuade, mews, and darius. in the following, we go into details for each dataset, describe our label mapping as the basis for the transfer learning, and compare the sequencing of argumentative elements in each dataset. ## study 1: zero shot transfer for our first study, we adopt the sequence tagging architecture developed by @xcite , which pre-processes the annotated training data into tokens with inside-outside-beginning (iob) tags and uses them as the input to the pretrained longformer model @xcite for token classification. we trained two models on 90% of en-l1 data with the longformer foot_2 to transfer on en-l2 data and its multi-lingual variation xlm-r longformer foot_3 for the de data. after 10 epochs of training with a maximal length of 1024 tokens, the iob tags of tokens are post-processed into predictions for different argumentative elements. following the same schema as for the iaa evaluation, we evaluate our results also through the f1 score: all gold standards and predictions for a given argumentative element are compared. if the overlap between the gold standard and prediction in both directions is higher or equal to 0.5, the prediction is considered a true positive. if multiple matches exist, the match with the highest is taken. any unmatched ground standards are false negatives and any unmatched predictions are false positives. ## study 2: learning curve after having established zero-shot transfer performance, we investigate the potential of using a small amount of target domain training data to improve the performance of our argument mining model on the target test dataset. this process involves finetuning the model trained on en-l1 data using a portion of data from the target domain (en-l2 and de), allowing it to adapt its representations of the features in l2 and german argumentative essays. however, it is important to note that fine-tuning requires access to labeled data from the target domain. in a practical application scenario, when a teacher wants to fine-tune such a model for a new educational context or language, it is important to know how much data needs to be labeled, since human annotation effort is often a crucial factor. therefore, we perform a series of learning curve experiments, in which we systematically vary the amount of training data from target datasets. ## conclusion this paper explores the transferability of argumentmining models across different educational contexts and languages. through comprehensive analyses of various datasets, including those authored by native english speakers (en-l1), english as a foreign language learners (en-l2), and native german writers (de), as well as machine-translated german essays (de-mt), we answer rq 1 and show their differences in linguistic structures and argumentation styles. our experimental studies designed for rq 2 reveal that, under zero-shot conditions, models trained on en-l1 demonstrate satisfactory performance when directly applied to en-l2/de datasets. however, fine-tuning the model on target domain data does not increase the performance significantly, highlighting the challenges of transfer learning across different educational contexts and languages. notably, as the answer for rq 3, machine translation does not significantly enhance performance, indicating that differences in dataset characteristics stem less from language disparities, but more from distinct educational contexts.