Automatic scoring of short answers using justification cues estimated by BERT (2022)

Abstract: Automated scoring technology for short-answer questions has been attracting attention to improve the fairness of scoring and reduce the burden on the scorer. In general, a large amount of data is required to train an automated scoring model. The training data consists of the answer texts and the scoring data assigned to them. It may also include annotations indicating key word sequences. These data must be prepared manually, which is costly. Many previous studies have created models with large amounts of training data specific to each question. This paper aims to achieve equivalent performance with less training data by utilizing a BERT model that has been pre-trained on a large amount of general text data not necessarily related to short answer questions. On the RIKEN dataset, the proposed method reduces the training data from the 800 data required in the past to about 400 data, and still achieves scoring accuracy comparable to that of humans.

## introduction automatic short answer scoring (sas) system using natural language processing technology has several advantages, such as the immediate return of scoring results and the ability to submit answers from any location over networks. to realize such interactive learning, a lot of research has been done on asap-sas data. assuming amount of scored short answers are available as training data, semantic similarity @xcite or machine learning @xcite is used for the score prediction. also, as an attempt using deep learning, cnn and lstm have been configured on top of word embedding to predict the holistic score directly @xcite . unfortunately, predicting holistic scores directly from word sequences is not very promising because there are too big a leap between words and scores. with this background, riken center for aip provided sas dataset with analytic scores and annotations (justification cues) as well as holistic scores in public to help with research activities. the dataset includes sample responses from 2,100 students for each of the six readings comprehension test prompts @xcite . riken center also developed automatic scoring technology using deep learning for the dataset. @xcite proposed a bidirectional lstm model integrating a supervised attention mechanism estimating the justification cue for scoring @xcite . the model was evaluated with various sizes of training data. it is reported that approximately 800 training data per question are needed to achieve the same accuracy as humans. however, we know it is difficult to prepare 800 training data manually in actual schools. therefore, we consider using bert model @xcite ) pre-trained with a large amount of general text not necessarily related to short answer questions, so to reduce the amount of specific training data required. several research institutes provide pre-trained bert models. they are well-trained with huge general corpus and supposed to be fine-tuned with small amount of specific corpus. instead of using supervised attention in @xcite , this study uses bert to annotate word sequences as the justification cues. the justification identification model is created by fine-tuning one of the pre-trained bert models with a specific data set. ## riken dataset for short answer assessment our study compares the performance and the required data size with the previous method by @xcite . therefore, the same riken dataset was used. this data was obtained by annotating and scoring the answers to japanese writing questions in the mock examinations for high school students conducted by yoyogi seminar from 2014 to 2015 according to a predefined rubric. answers are approximately 50 to 70 characters long (in japanese). as shown in figure 1 , each answer is annotated with a "1" for words that contribute to the score and a "0" for words that do not. the overall score is calculated as the sum of the individual scores. in the example shown in figure 1 , there are four analytic criteria, a, b, c, and d, and annotations are assigned to each of them. each item is then scored using this annotation as a justification cue. the overall score is calculated by summing the item scores and subtracting points for spelling errors and bad sentence endings etc. ## experiments the following experiments were conducted to evaluate the performance of the proposed method. our experiments used the japanese pre-trained bert model published by the inui-suzuki laboratory at tohoku university (inui laboratory 2021). ## conclusion this paper proposed the combined model of justification prediction and analytic scoring model. it includes fine-tuning of pre-trained bert model that predicts justification cues (annotations), which are crucial for automatic scoring. bert embedding vectors of annotated words are subsequently passed to lightgbm model @xcite for scoring. the proposed model uses a bert model that has been pre-trained with a large corpus of text in a general domain. as shown in