Multi-Stage Pre-training for Automated C hinese Essay Scoring (2020)

Abstract: This paper proposes a pre-training based automated Chinese essay scoring method. The method involves three components: weakly supervised pre-training, supervised cross- prompt fine-tuning and supervised target- prompt fine-tuning. An essay scorer is first pre- trained on a large essay dataset covering diverse topics and with coarse ratings, i.e., good and poor, which are used as a kind of weak supervision. The pre-trained essay scorer would be further fine-tuned on previously rated es- says from existing prompts, which have the same score range with the target prompt and provide extra supervision. At last, the scorer is fine-tuned on the target-prompt training data. The evaluation on four prompts shows that this method can improve a state-of-the-art neural essay scorer in terms of effectiveness and domain adaptation ability, while in-depth analysis also reveals its limitations..

## introduction automated essay scoring (aes) is an important educational application of natural language processing (nlp) @xcite . aes aims to automatically judge the quality of student essays, which can reduce teachers' burden on essay scoring and provide fast feedback to students. aes is usually viewed as a supervised learning problem. traditionally, aes systems are based on hand-crafted surface-level features @xcite @xcite . recently, neural network based representation learning has been applied and achieved superior performance compared with traditional methods @xcite @xcite @xcite . most of the proposed methods, no matter the feature based or the representation learning based ones, work in an in-domain setting that is to train a scorer for a specific prompt based on a set of example essays for this prompt, and use this scorer to rate more essays from the same prompt. this manner usually requires many rated examples to get acceptable performance. although cross-domain transferable essay scoring has gained more attention @xcite , the progress is still limited. the possible reason may be that the available corpora for essay scoring usually cover narrow topics on a small scale, and the topics, scoring criteria, and score ranges of different prompts often vary. since an aes system should have the ability to appreciate or criticize essays, supervised pre-training is necessary. intuitively, if a reader has read many rated essays from different prompts, she should be more experienced to judge the quality of an essay that responds to a new prompt. at least, she should require less guidance compared to a novice. in this work, we empirically evaluate a pretraining based method for aes. figure 1 illustrates the main framework. our method has three components, each of which incorporates different level supervision. the first component is weakly supervised pre-training. an essay scorer is pre-trained based on a large scale essay corpus. the corpus covers diverse topics and is prompt-free. the essays are collected from the web and have been rated by anonymous teachers. the essays' ratings are converted to binary coarse ratings: good and poor for the ease of weakly supervised pretraining. the second component is supervised cross-prompt pre-training / fine-tuning. this component aims to exploit the supervision from the training data of other prompts to pre-train or further fine-tune an essay scorer. the third component is supervised target-prompt fine-tuning. the pre-trained scorer would be fine-tuned on the training data for target prompts. since human ratings are expensive to be collected, we expect the essay scorer depends on the target-prompt training data the less the better. although there are public available datasets in english such as the asap dataset. 1 these datasets usually cover only a few topics making it difficult to find datasets for pre-training and fine-tuning. as a result, we collect datasets and conduct experiments for automated chinese essay scoring. we built a dataset with more than 85,000 essays written by junior and senior high school students for weakly supervised pre-training. we also collected nearly 4,000 essays in response to four prompts from senior high schools. these essays were carefully rated by teachers and are used for cross-prompt fine-tuning and evaluation. although the framework is straightforward, the evaluation demonstrates the effectiveness of the proposed method. to the best of our knowledge, we are the first to investigate multi-stage pre-training based aes. we conduct careful analysis to gain more insights about how the method works and its limitations. although our research focuses on chinese, the results and observations should be useful for aes in other languages as well. since 2016, neural network based aes systems become popular @xcite @xcite @xcite . these models obtained superior performance compared with traditional methods. however, most of these systems are promptspecific. new training data has to be annotated for training a new model for a new prompt. @xcite proposed domain adaptation as a solution to adapt an aes system from one initial prompt to another prompt based on bayesian linear ridge regression. @xcite demonstrated that the hierarchical cnn based model performs better in domain adaptation setting. @xcite ; @xcite also attempted to incorpo-rate external knowledge for readability assessment. however, these methods mostly focused on domain adaptation from one domain to another but did not explore external resources. ## conclusion in this paper, we presented a pre-training based approach to automated chinese essay scoring. our method investigates multi-stage pre-training and incorporates multi-level supervision, including the weak supervision from large scale coarse ratings, the supervision from rated essays from other prompts and the target-prompt training data. the experimental results show that the pretraining based approach is effective for aes in terms of both effectiveness and domain adaptation ability. we carefully analyze the effects of each component and find that: multi-stage pre-training improves the base model in general; the domain adaptation ability can be consistently improved; target-prompt fine-tuning is still indispensable but the required amount of training data can be largely reduced; weakly supervised pre-training and supervised transfer fine-tuning are both helpful. we also observe some phenomena but do not have good explanations. for example, the performance gain across prompts is inconsistent. when the pre-trained scorer can work best should be further studied. we suggest that the prompts' properties should be investigated more for applying aes. the proposed method has a limitation that it pays more attention to the score range that most essays are from, and may hurt the performance in other ranges. another limitation of the method is the dependence on pre-training dataset. the pre-training dataset used in this paper is still small compared with the data used for pre-training language models. larger pre-training dataset with supervised labels or self-supervised learning strategies could be explored. moreover, we are interested in understanding what features or traits of essays are captured by the deep models for scoring. we plan to investigate these in future.