On the Use of Bert for Automated Essay Scoring: Joint Learning of Multi-Scale Essay Representation (2022)

Abstract: In recent years, pre-trained models have become dominant in most natural language processing (NLP) tasks. However, in the area of Automated Essay Scoring (AES), pre-trained models such as BERT have not been properly used to outperform other deep learning models such as LSTM. In this paper, we introduce a novel multi-scale essay representation for BERT that can be jointly learned. We also employ multiple losses and transfer learning from out-of-domain essays to further improve the performance. Experiment results show that our approach derives much benefit from joint learning of multi-scale essay representation and obtains almost the state-of-the-art result among all deep learning models in the ASAP task. Our multi-scale essay representation also generalizes well to CommonLit Readability Prize data set, which suggests that the novel text representation proposed in this paper may be a new and effective choice for long-text tasks.

## introduction aes is a valuable task, which can promote the development of automated assessment and help teachers reduce the heavy burden of assessment. with the rise of online education in recent years, more and more researchers begin to pay attention to this field. aes systems typically consist of two modules, which are essay representation and essay scoring modules. the essay representation module extracts features to represent an essay and the essay scoring module rates the essay with the extracted features. when a teacher rates an essay, the scores are often affected by multiple signals from different granularity levels, such as token level, sentence level, paragraph level and etc. for example, the features may include the numbers of words, the essay structure, the master degree of vocabulary and syntactic complexity, etc. these features come from different scales of the essay. this inspires us to extract multi-scale features from the essays which represent multi-level characteristics of the essays. most of the deep neural networks aes systems use lstm or cnn. some researchers @xcite @xcite attempt to use bert @xcite in their aes systems but fail to outperform other deep neural networks methods @xcite . we believe previous approaches using bert for aes suffer from at least three limitations. first, the pre-trained models are usually trained on sentence-level, but fail to learn enough knowledge of essays. second, the aes training data is usually quite limited for direct fine-tuning of the pre-trained models in order to learn better representation of essays. last but not least, mean squared error (mse) is commonly used in the aes task as the loss function. however, the distribution of the sample population and the sorting properties between samples are also important issues to be considered when designing the loss functions as they imitate the psychological process of teachers rating essays. different optimizations can also bring diversity to the final overall score distribution and contribute to the effectiveness of ensemble learning. to address the aforementioned issues and limitations, we introduce joint learning of multi-scale essay representation into the aes task with bert, which outperforms the state-of-the-art deep learning models based on lstm @xcite . we propose to explicitly model more effective representations by extracting multiscale features as well as leveraging the knowledge learned from numerous sentence data. as the training data is limited, we also employ transfer learn-ing from out-of-domain essays which is inspired by @xcite . to introduce the diversity of essay scoring distribution, we combine two other loss functions with mse. when training our model with multiple losses and transfer learning using r-drop @xcite , we almost achieve the state-of-the-art result among all deep learning models. the source code of prediction module with a trained model for asap's prompt 8 is publicly available 3 . in summary, the contribution of this work is as follows: ## related work the dominant approaches in aes can be grouped into three categories: traditional aes, deep neural networks aes and pre-training aes. ## conclusion and future work in this paper, we propose a novel multi-scale essay representation approach based on pre-trained language model, and employ multiple losses and transfer learning for aes task. we almost obtain the state-of-the-art result among deep learning models. in addition, we show multi-scale representation has a significant advantage when dealing with long texts. one of the future directions could be exploring soft multi-scale representation. introducing linguistic knowledge to segment at a more reasonable scale may bring further improvement.