Automated Topical Component Extraction Using Neural Network Attention Scores from Source-based Essay Scoring (2020)

Abstract: While automated essay scoring (AES) can reliably grade essays at scale, automated writing evaluation (AWE) additionally provides formative feedback to guide essay revision. However, a neural AES typically does not provide useful feature representations for supporting AWE. This paper presents a method for linking AWE and neural AES, by extracting Topical Components (TCs) representing evidence from a source text using the intermediate output of attention layers. We evaluate performance using a feature-based AES requiring TCs. Results show that performance is comparable whether using automatically or manually constructed TCs for 1) representing essays as rubric-based features, 2) grading essays.

## introduction automated essay scoring (aes) systems reliably grade essays at scale, while automated writing evaluation (awe) systems additionally provide formative feedback to guide revision. although neural networks currently generate state-of-the-art aes results @xcite @xcite @xcite @xcite , non-neural aes create feature representations more easily useable by awe @xcite @xcite @xcite . we believe that neural aes can also provide useful information for creating feature representations, e.g., by exploiting information in the intermediate layers. our work focuses on a particular source-based essay writing task called the response-to-text assessment (rta) @xcite . recently, an rta awe system @xcite was built by extracting rubric-based features related to the use of topical components (tcs) in an essay. however, manual expert effort was first required to create the tcs. for each source, the tcs consist of a comprehensive list of topics related to evidence which include: 1) important words indicating the set of evidence topics in the source, and 2) phrases representing specific examples for each topic that students need to find and use in their essays. to eliminate this expert effort, we propose a method for using the interpretable output of the attention layers of a neural aes for source-based essay writing, with the goal of extracting tcs. we evaluate this method by using the extracted tcs to support feature-based aes for two rta source texts. our results show that 1) the feature-based aes with tcs manually created by humans is matched by our neural method for generating tcs , and 2) the values of the rubric-based essay features based on automatic tcs are highly correlated with human evidence scores. ## related work three recent awe systems have used non-neural aes to provide rubric-specific feedback. @xcite developed an influence estimation process that used a logistic regresion aes to identify sentences needing feedback. @xcite presented a web-based tool that provides formative feedback on rhetorical moves in writing. @xcite used features created for a random forest aes to select feedback messages, although human effort was first needed to create tcs from a source text. we automatically extract tcs using neural aes, thereby eliminating this expert effort. others have also proposed methods for preprocessing source information external to an essay. content importance models for aes predict the parts of a source text that students should include when writing a summary @xcite source excerpt: today, yala sub-district hospital has medicine, free of charge, for all of the most common diseases. water is connected to the hospital, which also has a generator for electricity. bed nets are used in every sleeping site in sauri... essay prompt: the author provided one specific example of how the quality of life can be improved by the millennium villages project in sauri, kenya. based on the article, did the author provide a convincing argument that winning the fight against poverty is achievable in our lifetime? explain why or why not with 3-4 examples from the text to support your answer. essay: in my opinion i think that they will achieve it in lifetime. during the years threw 2004 and 2008 they made progress. people didnt have the money to buy the stuff in 2004. the hospital was packed with patients and they didnt have alot of treatment in 2004. in 2008 it changed the hospital had medicine, free of charge, and for all the common dieases. water was connected to the hospital and has a generator for electricity. everybody has net in their site. the hunger crisis has been addressed with fertilizer and seeds, as well as the tools needed to maintain the food. the school has no fees and they serve lunch. to me thats sounds like it is going achieve it in the lifetime. in the computer vision area, attention cropped images have been used for further image classification or object detection @xcite @xcite . in the nlp area, @xcite proposed to use a generator to find candidate rationale and these are passed through the encoder for prediction. our work is similar in spirit to this type of work. ## rta corpus and prior aes systems the essays in our corpus were written by students in grades 4 to 8 in response to two rta source texts @xcite : rt a m v p (2970 essays) and rt a space (2076 essays). to date, two approaches to aes have been proposed for the rta: aes rubric and aes neural . to support the needs of awe, aes rubric @xcite used a traditional supervised learning framework where rubric-motivated features were extracted from every essay before model training -number of pieces of evidence (npe) foot_0 , concentration (con), specificity (spc) foot_1 , word count (woc). the two aspects of tcs introduced in section 1 (topic words, specific example phrases) were used during feature extraction. motivated by improving stand-alone aes performance (i.e., when an interpretable model was not needed for subsequent awe), @xcite developed aes neural , a hierarchical neural model with the co-attention mechanism in the sentence level to capture the relationship between the essay and the source. neither feature engineering nor tc creation were needed before training. ## attention-based tc extraction: t c attn in this section we propose a method for extracting tcs based on the aes neural attention level outputs. since the self-attention and co-attention mechanisms were designed to capture sentence and phrase importance, we hypothesize that the attention scores can help determine if a sentence or phrase has important source-related information. to provide intuition, based on these observations, we first extract the output of two layers from the neural network: 1) the attn sent of each sentence, and 2) the output of the convolutional layer as the representation of the phrase with the highest attn phrase in each sentence (denoted by cnn phrase ). we also extract the plain text of the phrase with the highest attn phrase in each sentence (denoted by text phrase ). then, our t c attn method uses the extracted information in 3 main steps: 1) filtering out text phrase from sentences with low attn sent , 2) clustering all remaining text phrase based on cnn phrase , and 3) generating tcs from clusters. the first filtering step keeps all text phrase where the original sentences have attn sent higher than a threshold. the intuition is that lower attn sent indicates less source-related information. the second step clusters these text phrase based on their corresponding representations cnn phrase . we use k-medoids to cluster text phrase into m clusters, where m is the number of topics in the source text. then, for text phrase in each topic cluster, we use k-medoids to cluster them into n clusters, where n is the number of the specific example phrases we want to extract from each topic. the outputs of this step are m * n clusters. the third step uses the topic and example clustering to extract tcs. as noted earlier, tcs include two parts: topic words, and specific example phrases. since our method is data-driven and students introduce their vocabulary into the corpus, essay text is noisy. to make the tc output cleaner, we filter out words that are not in the source text. to obtain topic words, we combine all text phrase from each topic cluster to calculate the word frequency per topic. to make topics unique, we assign each word to the topic cluster in which it has the highest normalized word frequency. we then include the top k topic words based on their frequency in each topic cluster. to obtain example phrases, we combine all text phrase from each example cluster to calculate the word frequency per example, then include the top k example words based on their frequency in each example cluster. ## experimental setup and results figure 1 shows an overview of four tc extraction methods to be evaluated. t c manual (upper bound) uses a human expert to extract tcs from a source text. t c attn is our proposed method and automatically extracts tcs using both a source text and student essays. t c lda @xcite (baseline) builds on lda to extract tcs from student essays only, while t c pr (baseline) builds on positionrank @xcite to instead extract tcs from only the source text. since positionrank is not designed for tc extraction, we needed to further process its output to create t c pr . to extract topic words, we extract all keywords from the output. next, we map each word to a higher dimension with word embedding. lastly, we cluster all keywords using k-medoids into p r topic topics. to extract example phrases, we put them into only one topic and remove all redundant example phrases if they are subsets of other example phrases. we configure experiments to test two hypotheses: h1) the aes rubric model for scoring evidence @xcite will perform comparably when extracting features using either t c attn or t c manual , and will perform worse when using t c lda or t c pr ; h2) the correlation between the human evidence score and the feature values (npe and sum of spc features) 3 will be comparable when extracted using t c attn and t c manual , and will be stronger than when using t c lda and t c pr . the experiment for h1 tests the impact of using our proposed tc extraction method on the downstream aes rubric task, while the h2 experiment examines the impact on the essay representation itself. following @xcite , we stratify essay corpora: 40% for training word embeddings and extracting tcs, 20% for selecting the best embedding and parameters, and 40% for testing. we use the hyper-parameters from @xcite for neural training as shown in results for h1. h1 is supported by the results in ## conclusion and future work this paper proposes t c attn , a method for using the attention scores in a neural aes model to automatically extract the topical components of a source text. evaluations show the potential of t c attn for eliminating expert effort without degrading aes rubric performance or the feature representations themselves. t c attn outperforms baselines and generates comparable or even better results than a manual approach. although t c attn outperforms all baselines and requires no human effort on tc extraction, annotation of essay evidence scores is still needed. this leads to an interesting future investigation direction, which is training the aes neural using the gold standard that can be extracted automatically. one of our next steps is to investigate the impact of tc extraction methods on a corresponding awe system @xcite , which uses the feature values produced by aes rubric to generate formative feedback to guide essay revision. currently, the t c lda are trained on student essays, while the t c pr only works on the source article. however, t c attn uses both student essays and the source article for tc generation. it might be hard to say that the superior performance of t c attn is due to the neural architecture and attention scores rather than the richer training resources. therefore, a comparison between t c attn and a model that uses both student essays and the source article is needed.