Using Transfer Learning to Automatically Mark L 2 Writing Texts (2021)

Abstract: The use of transfer learning in Natural Language Processing (NLP) has grown over the last few years. Large, pre-trained neural networks based on the Transformer architecture are one example of this, achieving state-of-theart performance on several commonly used performance benchmarks, often when finetuned on a downstream task. Another form of transfer learning, Multitask Learning, has also been shown to improve performance in Natural Language Processing tasks and increase model robustness. This paper outlines preliminary findings of investigations into the impact of using pretrained language models alongside multitask fine-tuning to create an automated marking system of second language learners’ written English. Using multiple transformer models and multiple datasets, this study compares different combinations of models and tasks and evaluates their impact on the performance of an automated marking system This presentation is a snap-shot of work being conducted as part of my dissertation for the University of Wolverhampton’s Computational Linguistics Masters’ programme.

## introduction and related work human marking of learner productive skills is a costly part of the test development process both in terms of direct financial outlay to external markers and internal resource. assessors are paid for marking candidates' written test responses and attending training while managing the marking process requires recruiting, training, and monitoring assessors, along with ensuring they are paid for the correct number of responses marked. this cost also increases linearly, meaning that as more candidates enrol to take the test, the number of assessors increases, as does the resource required to manage this increased pool of assessors. it therefore does not scale effectively. further to this, there are also issues regarding reliability of assessors. despite efforts by test developers to ensure their assessors mark consistently to the given criteria, human markers often demonstrate inconsistencies as shown by research conducted into inter-rater and intra-rater reliability @xcite . one way of attempting to address this issue is to implement an automated marking system. this would make a significant difference to test developers who would be able to save significant resource while improving the reliability of their written assessments. many test developers already use automated marking in their assessment products, such as linguaskill @xcite from cambridge assessment, e-rater @xcite from ets, and pte academic (ins, 2019) from pearson. it is also an area which has attracted much research @xcite @xcite @xcite @xcite @xcite . this has meant different approaches have been used for automated essay marking and have employed a range of different algorithms. @xcite conceptualised automated marking as a classification problem, employing naive bayes; @xcite adopted an unsupervised clustering approach; and @xcite used a linear batch perceptron classifier. this approach could be well suited to classifying test takers on the cefr @xcite , given the discreet categories, although they are ordinal in nature. ## research questions the questions that this paper sets out to investigate are: ## task definition to address the first research question, several models were created in order to achieve suitable comparison. the first model provided a baseline against which the more sophisticated and computationally expensive techniques could be compared. this model was created from manually extracted features, and aimed to be a simplification of the model features described by @xcite by using 1-3 grams of tokens, pos and dependency tags, and distances to other grammatical relations. these features were extracted using spacy @xcite . the task was defined as a standard regression task unlike @xcite which used a rank preference approach. these features were then used to train a random forest @xcite . the second model used the bert base architecture @xcite , but with all the model's weights randomly initialized. the third and fourth used bert base @xcite and roberta base @xcite pretrained models. it was decided not to include an untrained version of roberta because of its architectural similarity to bert. the transformers library implementations of the models were used @xcite . all bert models used its uncased variant. each model was trained on a dataset of humanmarked written responses to test questions. see below for further information on the dataset used. ## discussion the transformer-based language models performed better than that which used manually extracted features across all the reported metrics. also, the two models which used pre-trained weights reported superior metrics to those which had not. these results sit in contrast to those found by @xcite who reported no performance improvement when using pretrained transformer models. one explanation for this might be the difference in domains. @xcite was fine-tuned and tested on five datasets from the asap competition (shermis, 2014) which, though similar in approach, did not mark language learners for their competence in english. this point highlights a critical difference in standard aes and marking the quality of a learners' english. marking discursive essays requires the assessor to focus on much higher-level textual features compared to that of marking an essay written as part of and esol exam. for example, the scoring rubric for asap question 1 asks the assessor to focus on whether the main idea(s) ... stand out, the text makes connections and shares insights, or whether it is clear, focused or interesting. this can be contrasted with the oxford test of english scoring rubric (oup, 2019) which focuses on lower-level aspects of the text, requiring responses to demonstrate a high degree of grammatical accuracy and a wide range of cohesive devices. understanding such high-level and potentially subjective aspects are likely to be more problematic for language models than issues of grammatical accuracy. one reason for the increased performance of roberta compared to that of bert is likely to be the most most obvious differences between the two pre-training approaches. roberta @xcite is a similar model to bert but with several key differences: the authors used more training data, they removed the next sentence prediction task, the token sequence length was increased, and the masking pattern applied was dynamically changed during training. this meant that the authors reported improved performance on bert across the same tasks, mirroring these preliminary results. regarding the performance of the mtl models, the vast majority of those which used mtl preformed better than those trained only on the score prediction task. however, the models trained on a single auxiliary task produced higher correlations more consistently than those trained on more than one task, though this trend was not true for the mae and rmse metrics, which, showed no obvious trend. figures 1 and 2 show scatter plots of the human mark against the automated marker. although the model shown in figure 1 performed better across the selected evaluation metrics, the model trained on all tasks produced predictions that were much more centralized and less erratic. this effect can also be seen in the average standard deviation of predictions for each true score (2.48 for all tasks and 2.56 for the cefr task alone). furthermore, when looking at the rmse for the predictions split by each human mark, the model trained on all tasks was more accurate for all true scores 10-19 but the model trained with only the cefr auxiliary task was more accurate for all true scores 0-9 and 20-21. this effect can also be seen in figures 1 and 2 where the cefr trained model made several predictions at the top end of the scale but the model trained on all tasks never predicted the highest possible score. one potential explanation for this effect is that the model trained on all tasks is not able to differentiate effectively between the higher scores and so clusters many predictions just below the top end of the grade scale in order to be close the true score most frequently. another explanation might be that as the number of tasks increases, more responsibility is being placed on the model to find a single representation that can be used to predict multiple outputs with only a single transformation in the task specific layer. by increasing the depth of the task specific heads with more transformer layers, this issue might improve, though would not remove the model's dependence on a single representation. ## conclusion preliminary findings in this research project have shown that pre-trained language models can perform better at automatically grading a learner's english than both traditional methods and the same models without pre-training. the picture regarding the mtl approach is less clear, with the impact of multiple tasks included in training producing opaque results when looking beyond the reported metrics. however, the study has shown that an mtl approach can benefit model performance, but that the impact of using multiple tasks can be unpredictable. there are several limitations with the study in its current form. firstly, no model hyperparameter search was conducted to produce the results presented here. this means that some models might perform much better if trained with other hyperparameter values. another limitation is that it has not explored other prediction methods. for example, as part of the multitask approach, a separate head could be trained to predict each of the three marking criteria as opposed to a total mark. another limitation and perhaps the most important, is that because the dataset is not available to the wider community, comparison of this approach over others is not possible. one solution to this would be to apply this approach to a freely available dataset such as the cambridge fce dataset @xcite .