Should You Fine-Tune BERT for Automated Essay Scoring? (2020)

Abstract: Most natural language processing research now recommends large Transformer-based models with fine-tuning for supervised classification tasks; older strategies like bag-of-words features and linear models have fallen out of favor. Here we investigate whether, in automated essay scoring (AES) research, deep neural models are an appropriate technological choice. We find that fine-tuning BERT produces similar performance to classical models at significant additional cost. We argue that while state-of-the-art strategies do match existing best results, they come with opportunity costs in computational resources. We conclude with a review of promising areas for research on student essays where the unique characteristics of Transformers may provide benefits over classical methods to justify the costs.

## introduction automated essay scoring (aes) mimics the judgment of educators evaluating the quality of student writing. originally used for summative purposes in standardized testing and the gre @xcite , these systems are now frequently found in classrooms @xcite , typically enabled by training data scored on reliable rubrics to give consistent and clear goals for writers @xcite . more broadly, the natural language processing (nlp) research community in recent years has been dominated by deep neural network research, in particular, the transformer architecture popularized by bert @xcite . these models use large volumes of existing text data to pre-train multilayer neural networks with context-sensitive meaning of, and relations between, words. the models, which often consist of over 100 million parameters, are then fine-tuned to a specific new labeled dataset and used for classification, generation, or structured prediction. research in aes, though, has tended to prioritize simpler models, usually multivariate regression using a small set of justifiable variables chosen by psychometricians @xcite . this produces models that retain direct mappings between variables and recognizable characteristics of writing, like coherence or lexical sophistication @xcite . in psychometrics more generally, this focus on features as valid "constructs" leans on a rigorous and well-defined set of principles @xcite . this approach is at odds with transformer-based research, and so our core question for this work is: for aes specifically, is a move to deep neural models worth the cost? the chief technical contribution of this work is to measure results for bert when fine-tuned for aes. in section 3 we describe an experimental setup with multiple levels of technical difficulty from bag-of-words models to fine-tuned transformers, and in section 5 we show that the approaches perform similarly. in aes, human interrater reliability creates a ceiling for scoring model accuracy. while transformers match state-of-theart accuracy, they do so with significant tradeoffs; we show that this includes a slowdown in training time of up to 100x. our data shows that these transformer models improve on n -gram baselines by no more than 5%. given this result, in section 6 we describe areas of contemporary research on transformers that show both promising early results and a potential alignment to educational pedagogy beyond reliable scoring. ## background in aes, student essays are scored either on a single holistic scale, or analytically following a rubric that breaks out subscores based on "traits." these scores are almost always integer-valued, and almost universally have fewer than 10 possible score points, though some research has used scales with as many as 60 points @xcite . in most contexts, students respond to "prompts," a specific writing activity with predefined content. work in natural language processing and speech evaluation has used advanced features like discourse coherence @xcite and argument extraction @xcite ; for proficient writers in professional settings, automated scaffolds like grammatical error detection and correction also exist @xcite . natural language processing has historically used n-gram bag-of-words features to predict labels for documents. these were the standard representation of text data for decades and are still in widespread use @xcite . in the last decade, the field moved to word embeddings, where words are represented not as a single feature but as dense vectors learned from large unsupervised corpora. while early approaches to dense representations using latent semantic analysis have been a major part of the literature on aes @xcite , these were corpus-specific representations. in contrast, recent work is general-purpose, resulting in offthe-shelf representations like glove @xcite . this allows similar words to have approximately similar representations, effectively managing lexical sparsity. but the greatest recent innovation has been contextual word embeddings, based on deep neural networks and in particular, transformers. rather than encoding a word's semantics as a static vector, these models adjust the representation of words based on their context in new documents. with multiple layers and sophisticated attention mechanisms @xcite , these newer models have outperformed the state-of-the-art on numerous tasks, and are currently the most accurate models on a very wide range of tasks @xcite . the most popular architecture, bert, produces a 768dimensional final embedding based on a network with over 100 million total parameters in 12 layers; pre-trained models are available for open source use @xcite . for document classification, bert is "finetuned" by adding a final layer at the end of the transformer architecture, with one output neuron per class label. when learning from a new set of labeled training data, bert evaluates the training set multiple times (each pass is called an epoch). a loss function, propagating backward to the network, allows the model to learn relationships between the class labels in the new data and the contextual meaning of the words in the text. a learning rate determines the amount of change to a model's parameters. extensive results have shown that careful control of the learning rate in a curriculum can produce an effective fine-tuning process @xcite . while remarkably effective, our community is only just beginning to identify exactly what is learned in this process; research in "bert-ology" is ongoing @xcite @xcite . these neural models are just starting to be used in machine learning for aes, especially as an intermediate representation for automated essay feedback @xcite . end-to-end neural aes models are in their infancy and have only seen exploratory studies like @xcite ; to our knowledge, no commercial vendor yet uses transformers as the representation for high-stakes automated scoring. ## nlp for automated essay scoring to date, there are no best practices on finetuning transformers for aes; in this section we present options. we begin with a classical baseline of traditional bag-of-words approaches and noncontextual word embeddings, used with naïve bayes and logistic regression classifiers, respectively. we then describe three curriculum learning options for fine-tuning bert using aes data based on broader best practices. we end with two approaches based on bert but without finetuning, with reduced hardware requirements. ## experiments to test the overall impact of fine-tuning in the aes domain, we use five datasets from the asap competition, jointly hosted by the hewlett foundation and kaggle.com @xcite . this set of essay prompts was the subject of intense pub-lic attention and scrutiny in 2012 and its public release has shaped the discourse on aes ever since. for our experiments, we use the original, deanonymized data from @xcite ; an anonymized version of these datasets is available online foot_0 . in all cases, human inter-rater reliability (irr) is an approximate upper bound on performance, but reliability above human irr is possible, as all models are trained on resolved scores that represent two scores plus a resolution process for disagreements between annotators. we focus our analysis on the five datasets that most closely resemble standard aes rubrics, discarding three datasets -prompts #1, 7, and 8with a scale of 10 or more possible points. results on these datasets are not representative of overall performance and can skew reported results due to rubric idiosyncracies, making comparison to other published work impossible (see for example @xcite , which groups 60-point and 4-point rubrics into a single dataset and therefore produced correlations that cannot be aligned to results from any other published work). prompts 2-6 are scored on smaller rubric scales with 4-6 points, and are thus generalizable to more aes contexts. note that nevertheless, each dataset has its own idiosyncracies; for instance, essays in dataset #5 were written by younger students in 7th and 8th grade, while dataset #4 contains writing from high school seniors; datasets #3 and 4 were responses to specific texts while others were open-ended; and scores in dataset #2 was actually scored on two separate traits, the second of which is often discarded in followup work (as it is here). our work here does not specifically isolate effects of these differences that would lead to discrepancies in performance or in modeling behavior. ## discussion for scoring essays with reliably scored, promptspecific training sets, both classical and neural approaches produce similar reliability, at approximately identical levels to human inter-rater reliability. there is a substantial increase in technical overhead required to implement transformers and fine-tune them to reach this performance, with minimal gain compared to baselines. the policy lesson for nlp researchers is that using deep learning for scoring alone is unlikely to be justifiable, given the slowdowns at both training and inference time, and the additional hardware requirements. for scoring, at least, transformer architectures are a hammer in search of a nail. but it's hardly the case that automated writing evaluation is limited to scoring. in this section we cover major open topics for technical researchers in aes to explore, focusing on areas where neural models have proven strengths above baselines in other domains. we prioritize three major areas: domain transfer, style, and fairness. in each we cite specific past work that indicates a plausible path forward for research.