Essay Quality Signals as Weak Supervision for Source-based Essay Scoring (2021)

Abstract: Human essay grading is a laborious task that can consume much time and effort. Automated Essay Scoring (AES) has thus been proposed as a fast and effective solution to the problem of grading student writing at scale. However, because AES typically uses supervised machine learning, a human-graded essay corpus is still required to train the AES model. Unfortunately, such a graded corpus often does not exist, so creating a corpus for machine learning can also be a laborious task. This paper presents an investigation of replacing the use of human-labeled essay grades when training an AES system with two automatically available but weaker signals of essay quality: word count and topic distribution similarity. Experiments using two source-based essay scoring (evidence score) corpora show that while weak supervision does not yield a competitive result when training a neural source-based AES model, it can be used to successfully extract Topical Components (TCs) from a source text, which are required by a supervised feature-based AES model. In particular, results show that feature-based AES performance is comparable with either automatically or manually constructed TCs.

## introduction essay grading is a laborious task that can consume much teacher time and effort, especially for classes with a large number of students. while human essay grading has high reliability and validity, human grading can also be biased and inconsistent over time. under such conditions, automatic essay scoring (aes) has been proposed as a fast and effective solution to the problem of grading student writing at scale, minimizing effort and assuring consistency. many aes systems have been developed to evaluate the content, structure, and quality of written essays via natural language processing (nlp) and machine-learning techniques. over the more than 50 year history of aes research, the majority of work has used feature-based models @xcite @xcite @xcite @xcite @xcite @xcite . however, these models require carefully designed hand-crafted features to represent essays. recently, neural network models have drawn increasing attention @xcite @xcite @xcite @xcite @xcite due to their powerful ability to generate essay representations automatically and to generate reliable essay score predictions. no matter whether a feature-based or neural network model is used, state-of-the-art aes systems are all supervised, which means that the model needs to be trained on a large number of humangraded essays. unfortunately, such a humangraded corpus often does not exist, and grading a corpus of essays is a laborious task. to address this problem, we investigate using a weakly supervised aes approach, where automatically available essay quality signals replace the use of human-labeled scores when training a state-of-the-art neural network model for source-based essay scoring. we define essay quality signals in this work as scores that reflect the quality of essays. the human-labeled holistic score is the most common essay quality signal, and is typically used for supervised machine learning. we instead explore the use of two automated essay quality signals, namely word count and topic distribution similarity between student essays and a source article, in a weakly supervised machine learning aes approach. our work focuses on the response-to-text assess-ment (rta) @xcite , a particular source-based essay writing task. in source-based writing, students read a source article before writing an essay that responds to a prompt related to the source article. in order to guide students on revising an rta essay after receiving feedback, an automated writing evaluation (awe) system for the rta @xcite was developed that generated formative feedback based on rubric-based aes features. these aes features were computed by recognizing when a set of manually created topical components (tcs) from the source text were used by students in their essays. the tcs represent a comprehensive list of topics related to evidence in the source article, which include: 1) important keywords indicating the set of topics (denoted as topic words), and 2) phrases representing specific examples for each topic (denoted as specific examples). the grading rubric encourages students to mention and elaborate topics and specific examples from the source article as much as possible in their essays. therefore, the rta aes and awe systems use rubric-based features based on the tcs in order to follow the grading rubric. previously, human effort was required to manually create tcs based on expert knowledge of the source text. to eliminate this effort, zhang and litman (2020) used the attention layer output of a co-attention neural network to automatically extract tcs. their experimental results showed that automated tcs performed equally well as manual tcs on a downstream, rubric-based aes task. however, to eliminate the human effort for creating tcs, they introduced a different human evidence score grading effort to train the co-attention neural network. unfortunately, we believe that the effort to grade student essays is more than the original effort to create the tcs. our goal is to remove both sources of human effort. to do so, we introduce two simple essay quality signals, word count and topic distribution similarity, which can be generated automatically and used as weak supervision for training the neural aes. although the learned aes model outperforms simple baselines, weak supervision is not enough to yield a state-of-the-art aes model. nonetheless, the proposed essay quality signals can be successfully used to generate tcs for a downstream rubricbased aes task. by using auto-generated essay quality signals, we can thus eliminate all human effort for generating tcs. we evaluate the generated tcs using a rubric-based aes requiring tcs, for two rta source articles. results show that our simple signals for training a neural aes to create tcs automatically do not degrade performance compared to using manually constructed tcs, whether for 1) representing essays as rubric-based features, 2) grading essays. ## related work the majority of research in the aes area uses supervised machine learning techniques that require a large number of human-graded essays for training. however, graded essay corpora are usually missing in real classroom scenarios, and annotating a corpus to train an aes model is laborintensive. a prior proposal to address this problem used an unsupervised-learning approach based on a voting algorithm @xcite . the area of short answer scoring has also faced a similar problem. @xcite presented a semi-supervised method to reduce the size of the required human-labeled corpus. meanwhile, @xcite proposed a graph-based lexico-semantic text matching for pattern identification. these works reduce human effort, but do not eliminate them. in contrast, our aes work fully replaces human graded evidence scores with essay quality signals that are easy to extract automatically and to use during training. although our results show that the signals are not effective for the aes task itself, they are useful for extracting topical components (tcs). previously, human expert effort was required to extract tcs. specifically, experts read through the source article and created lists of topic words and of specific examples that students were expected to use in their essays @xcite . in order to eliminate this human effort, three systems were later developed. an lda-based system @xcite used a lda topic model @xcite and turbotopic algorithm (blei and lafferty, 2009) for tc extraction. zhang and litman (2020) proposed another system based on the positionrank @xcite algorithm. while these two tc extraction systems did not require any human coding, they also did not match prior performance. the state-of-the-art system @xcite extracted tcs by exploiting the attention weights of a neural aes model. however, human grading effort was needed for model training. in our work, we replace hu- we believe that many predictive features used in the traditional feature-based aes systems can be useful signals for our weak supervision approach to tc extraction. for example, lengthbased features @xcite @xcite @xcite , prompt-relevant features @xcite , or semantic features @xcite all weakly relate to the quality of an essay's content. in this paper, we examine two such signals, word count and topic distribution similarity, and show that with these simple essay quality signals, human-labeled essay scores are unnecessary for tc extraction. ## corpora our work focuses on a specific source-based essay writing task named the response-to-text assessment (rta) @xcite . the corpora were all collected from upper elementary level classrooms. based on different articles that students read before writing their essays, two forms of the rta were developed. rt a m v p is based on an article from time for kids about an effort by the united nations to end poverty in a rural village in sauri, kenya, named the millennium villages project (mvp). rt a space is based on an article which discusses the importance of space exploration. after reading an rta article, there is a prompt that asks students to write an essay in response to the prompt. the prompt encourages students to use evidence from the source article to support their claims. all collected essays (2970 essays for rt a m v p and 2076 essays for rt a space ) were manually graded on a scale of 1 to 4 (low to high) in five dimensions: analysis, evidence, organization, style, and mugs (mechanics, usage, grammar, and spelling). each dimension evaluates one aspect of students' writing skills. we focus on the evidence dimension , which evaluates students' ability to find and use evidence from the source article to support their ideas. this dimension is also directly related to the final output of this work, topical components. ## prior aes systems for the rta two approaches to aes have been developed for the rta: 1) a feature-based supervised learning approach (with the features aligned to the evidence rubric criteria), which we refer to as aes rubric , and 2) a neural network approach, which we refer to as aes neural . aes rubric @xcite used a traditional supervised learning framework with hand-crafted, rubric-based features that require knowledge of tcs to compute. that is, a set of interpretable features were carefully designed to capture the relation between student essays and the two aspects of tcs described above (topic words and specific examples): number of pieces of evidence (npe): an integer feature based on the list of topic words for each topic that indicates the number of topics (semantically) mentioned in the essay. concentration (con): a binary feature that indicates if an essay elaborates on topics, again based on the list of topic words. specificity (spc): a vector of integer values indicating the number of specific examples (semantically) mentioned in the essay per topic. word count (woc): number of words. to support the needs of the awe system for the rta @xcite , the feature values and predicted evidence scores from aes rubric are passed to the awe system to select formative feed-source excerpt: today, yala sub-district hospital has medicine, free of charge, for all of the most common diseases. water is connected to the hospital, which also has a generator for electricity. bed nets are used in every sleeping site in sauri... essay prompt: the author provided one specific example of how the quality of life can be improved by the millennium villages project in sauri, kenya. based on the article, did the author provide a convincing argument that winning the fight against poverty is achievable in our lifetime? explain why or why not with 3-4 examples from the text to support your answer. essay: in my opinion i think that they will achieve it in lifetime. during the years threw 2004 and 2008 they made progress. people didn't have the money to buy the stuff in 2004. the hospital was packed with patients and they didn't have alot of treatment in 2004. in 2008 it changed the hospital had medicine, free of charge, and for all the common dieases. water was connected to the hospital and has a generator for electricity. everybody has net in their site. the hunger crisis has been addressed with fertilizer and seeds, as well as the tools needed to maintain the food. the school has no fees and they serve lunch. to me that's sounds like it is going achieve it in the lifetime. although aes rubric thus provides useful information for the awe system, in order to improve the stand-alone aes performance, aes neural @xcite was later developed. aes neural used a hierarchical neural network model with a self-attention mechanism in the phrase level and a co-attention mechanism in the sentence level. the self-attention layer captures the importance of each individual phrase, while the co-attention layer captures the relationship between the source article and the essay. in terms of the essay score prediction task, aes neural significantly outperforms aes rubric , and no human effort for either tc extraction or feature engineering is required. however, the essay representations created by aes neural cannot be directly used by the awe system, which depends on the rubric-based features to provide formative feedback in terms of the grading criteria. category 1 category 5 category 7 unpaved roads crops dying progress just four years tattered clothing not afford fertilizer irrigation medicine free charge bare feet outcome poor crops no midday meal lunch less than 1 dollar day lack fertilizer water kids go school now ... ... ## prior tc extraction methods to develop aes rubric , human expert effort was first required to manually extract tcs (t c manual ) in the form of two lists related to evidence in the source text: 1) a topic words list of important keywords that indicate the main set of article topics, and 2) a specific examples list that includes phrases representing specific examples for article topics. to replace the need for the human expert in creating such tcs, zhang and litman (2020) developed a method for tc extraction using aes neural . their algorithm was based on the observations that the co-attention layer on the sentence level assigned higher attention scores to important sentences, while the self-attention layer on the word (phrase) level assigned higher attention scores to important words (phrases). therefore, their system extracted important words from important sentences based on attention scores and used kmedoids to cluster all words. finally, it extracted tcs from each cluster. since human-labeled evidence scores of each essay were required for the neural network training, we denote this method by t c es . note that t c es replaced the human effort needed to extract tcs with the human effort needed to create the aes neural training supervision signal. ## experimental setup figure 1 shows an overview of usage of aes neural and four tc extraction systems to be evaluated. t c manual lets human experts extract tcs from each source article, and is thus the upper bound for evaluating the other (automated) tc extraction systems. t c es is our baseline automated model, which builds on aes neural and a clustering algorithm to extract tcs from student essays and the source article, using the gold-standard evidence score of each essay for aes neural training. t c wc and t c tds are methods proposed by this work that are instead based on weakly-supervised aes neural training. t c wc replaces evidence score with the word count of each essay, while t c tds uses topic distribution similarity with the number of topics. our experiments are designed to test two hypotheses related to the alternative aes and tc methods shown in figure 1 . the first hypothesis, denoted by h1, is that while weakly supervised training might not yield state-of-the-art, aes neural performance when evaluated as an end in itself, the use of automated essay quality signals nonetheless can outperform weak baselines such as random and majority score prediction. our second hypothesis, denoted by h2, is that weakly supervised training can nonetheless yield versions of aes neural that are still useful for automated tc extraction. all neural network models are built with tensorflow 2.2.0, and trained on an rtx 5000 gpu. ## results and discussion although both wc and tds underperform the human-generated evidence scores, tds constantly outperforms wc, despite the fact that wc has higher correlations with evidence score than tds (recall a very interesting finding is that both wc and tds underperform human score on aes neural task, while t c wc and t c tds help aes rubric reach an even higher qwk. this result shows that while learning using weak supervision is not enough for aes neural training, with post-processing the intermediate output, the neural predictions can still help to generate useful tcs for the aes rubric task. since word count is highly positively correlated with evidence score for both rt a m v p and rt a space , t c wc works well on average compared to t c tds . extrinsically, it outperforms t c manual on both corpora. it also matches t c es on rt a space , and has similar performance on rt a m v p . intrinsically, t c wc yields higher cor-relations for the npe feature when comparing to t c manual . although correlations for spc are worse than t c es , considering word count is the most intuitive signal without the needs of human effort, it performs surprisingly well. moving to topic distribution similarity, t c tds shows worse extrinsic performance on rt a space comparing to rt a m v p . to figure out the reason, we take a deep dive into the tcs generated by both methods. we consider that good automated tcs should cover topics in t c manual as many as possible. therefore, we manually label a topic for each of the manual topic words. for rt a m v p , t c tds has 4 related topics out of 5 (80%), while there are 10 related topics out of 16 (62.50%) for rt a space . obviously, t c tds preserves more related topics in rt a m v p . similarly, we also manually compare specific examples of both automated tcs with t c manual . for examples rather than keywords, t c tds has 16 out of 20 related categories (80%) for rt a m v p , while there are 11 out of 16 related categories (68.75%) for rt a space . t c tds again preserves more related categories in rt a m v p . we also observe that we can always find a better qwk using an automated tc method compared to t c manual ( a concern that might be raised about our work is that our essay quality signals have the potential to be gamed by students. for example, our signals could assign high scores to long essays with no relation to the prompt or to essays with replicated words of the source article. however, in our experiments, both essay quality signals are only used for training aes neural and extracting topical components. for experiments of h1, essay quality signals are used as the label for training the aes neural model instead of used as features during testing (when gaming would be expected). the trained aes neural is in fact tested on the human-graded evidence scores, not word count nor topic distribution similarity. for experiments of h2, essay quality signals are only used as the label for training aes neural and extracting topical components. aes rubric extracts features based on the topical components, and are then trained and tested on evidence scores. the purpose of this experiment is to show that the topical components extracted from the aes neural are useful for another downstream aes model, even though the qwk of the aes neural trained on essay quality signals is low. in sum, since essay quality signals are not used as predictive features in our system and only used as the label for training, we believe that our proposed method for topical components extraction is hard to be gamed -assuming adverserial essays only appear in the testing set. however, if such essays appear in the training data the risk of the manipulation of both aes neural and aes rubric based on word count, topic distribution similarity, or other essay quality signals is still possible @xcite . a future deep analysis of both aes models would be necessary to address this problem. another limitation of our research is that to date, our approach has only been tested on two rta corpora. this is because topical components have only been used in aes rubric until now. furthermore, although word count shows a high correlation with evidence scores in our rta corpora, this high correlation could easily disappear in tasks in which the prompt specifies the expected word count @xcite . similarly, topic distribution similarity might not a good essay quality signal if a prompt is not source-based, or does not expect similar topics in content. ## conclusion and future work this paper presented an investigation of replacing human-labeled evidence scores with other automated essay quality signals, such as word count and topic distribution similarity. these signals are easy to be calculated and integrated into existing systems in order to eliminate human effort. not surprisingly, these weak supervised signals are not enough for training a useable aes neural model. however, they still help generate tcs, which is required by aes rubric . we observe that even a simple signal like word count does not hurt the state-of-the-art baseline (t c es ). since there is no need for human effort, we believe that our work brings aes technology closer to being useful in real classroom scenarios. in our future work on using weak supervision for aes training, we would like to explore other possible essay quality signals beyond the two investigated here, again drawing on machine learning features used in prior work. it would also be interesting to examine whether our two existing signals might yield better aes results when training other types of neural algorithms and/or when applied to different datasets. moreover, the impact of data size for training on this model is worth exploration. finally, with respect to tc generation, because the specific examples are generated from clustering results, words in a specific example are not in readable orders. this leads to another interesting future investigation into making all examples in the specific examples list more human-understandable, although this would not affect the system performance due to the nature of the aes rubric .