Automated C hinese Essay Scoring from Multiple Traits (2022)

Abstract: Automatic Essay Scoring (AES) is the task of using the computer to evaluate the quality of essays automatically. Current research on AES focuses on scoring the overall quality or single trait of prompt-specific essays. However, the users not only expect to obtain the overall score but also the instant feedback from different traits to help their writing in the real world. Therefore, we first annotate a mutli-trait dataset ACEA including 1220 argumentative essays from four traits, i.e., essay organization, topic, logic, and language. And then we design a hierarchical multi-task trait scorer HMTS to evaluate the quality of writing by modeling these four traits. Moreover, we propose an inter-sequence attention mechanism to enhance information interaction between different tasks and design the trait-specific features for various tasks in AES. The experimental results on ACEA show that our HMTS can effectively score essays from multiple traits, outperforming several strong models.

## introduction automatic essay scoring (aes) is a task of using the computer to evaluate the quality of students' essays. an efficient aes system can bring many benefits to the field of education, including avoiding the influence of subjective factors on essay scoring, greatly reducing teachers' workload, and providing instant feedback for students' written essays. early work treated aes as a classification @xcite , regression @xcite , or ranking problem @xcite , addressing aes by supervised learning. recently, with the progress of deep learning, the end-to-end models based on neural networks (e.g., lstm-based models or the combination of lstms and cnns) have achieved impressive results (alikaniotis et al., 2016; taghipour * corresponding author @xcite @xcite . in consideration for more feedback in aes systems, some studies began to focus on scoring specific traits of essays, such as word choice, content @xcite , organization @xcite , thesis clarity @xcite , prompt adherence @xcite , argument persistent @xcite , style @xcite , etc. most of the above studies focused on scoring prompt-specific essays. as previous researchers had pointed out @xcite , it was difficult and expensive to access ample target-prompt essays in the real-world aes system. therefore, some researchers on english aes had began to explore cross-prompt aes @xcite @xcite . their methods utilized non-target-prompt essays to train models and score target-prompt essays. in reality, people usually judge the essay from different perspectives to give the final overall holistic score. apart from @xcite , few studies had combined the overall score with the feature-specific scores. therefore, to make aes more interpretable and verify whether the information learned in the traits contributes to the learning of the overall score, we use multi-task learning to evaluate the overall holistic score from multiple traits and the score of each trait. this paper focuses on scoring chinese essays and their traits. it is worth noting that most of the previous studies were conducted on the englishlanguage dataset asap 1 , which contains eight different collections of essays with specific prompts. unlike that, our chinese dataset is thematically diverse and does not contain prompts, so it is more challenging than the english task. there are some issues between english and chinese for the overall score and the traits scores. first, they have different expression problems. the 1 https://www.kaggle.com/c/asap-aes . most significant differences are that chinese is a parataxis language while english is hypotaxis. in the task of english aes, word usage is the most important factor in english expression @xcite . english text mainly forms semantic representation by combining the similarity between words. while chinese essays pay more attention to sentence-level and paragraph-level expression. second, chinese and english essays have different evaluation emphases that we should evaluate the essay quality from different perspectives. for example, for text expression, the evaluation criteria for english essays are using language correctly and high-level vocabulary, while the evaluation criteria for chinese essays are using beautiful sentences and smooth expressions. therefore, our task of scoring chinese essays and their traits exhibits two challenges. first, we need to select appropriate traits to express chinese essays and design the proper model to combine the different traits to prompt the performance of overall scoring. second, a good model is needed to fully express the deep semantics of chinese essays, which should have a good generalization ability to perform well in different trait tasks. in this paper, we design a hierarchical multitask trait scorer (hmts) for the task of automated chinese essay scoring. to cope with the first challenge, we created a dataset named "automatic chinese essay assessment (acea)" extended from the dataset used in @xcite , which contains 1220 essays, the overall score of each essay, and the scores of the four traits (i.e., topic, organization, logic, and language). and then we utilize the multi-task learning framework to learn the trait levels. specially, to better combine different trait expressions, we propose an inter-sequence attention mechanism, which can integrate the representations of different trait tasks. we believe that different trait tasks can complement each other. for example, an essay that scores well on logic will also score well on language performance. besides, since different tasks have different evaluation criteria, we also obtain trait-specific features to enhance the representation of the specific trait tasks. to cope with the second challenge, we utilize the hierarchical model to obtain the shared sentences and paragraph representation, and essay representation independent for each task. we use the xl-net as the sentence encoder, which is a pre-trained model trained on large-scale datasets and can get better text semantic representation. in a word, the contributions of this paper are as follows: in this section, we first introduce the promptspecific and cross-prompt holistic scoring methods and then describe the prompt-specific and multitask trait scoring methods. finally, we briefly introduced the task of chinese aes. ## acea dataset following the college entrance examination essay scoring standard, we selected four traits of topic, organization, logic, and language for our multi-task learning, which can help evaluate the level of essay quality. ## hmts model we employ a hierarchical model to learn the semantic representation of texts and reinforce the expression of each trait task through multi-task learning. for our hierarchical neural model, we utilize xl-net @xcite to learn the sentence expressions and lstm to learn the paragraph and essay expressions, which can simulate the coherence between the sentence and paragraph sequences. we also use the attention pooling to encode sentences, which aims to capture more relevant sentences that contribute to the paragraph semantic representation, thereby further expressing the quality of an article. for our multi-task framework, unlike the english multi-task aes designed by @xcite and @xcite , we choose different traits for chinese trait scoring, and our test dataset contains essays with multiple prompts rather than a single prompt. to share information between different traits more clearly, we propose the inter-sequence attention mechanism to help each trait pay attention to the information contained in the other traits. we also introduce the trait-specific features to represent the trait quality. our hmts model includes two stages and the details are shown in figure 1 . hmts includes two layers: 1) common layer: this layer aims to learn the useful low-level representations for all tasks, including the sentence and paragraph representation; 2) privates layer: this high-level layer is employed to learn more task-specific representations, including essay representation, the inter-sequence mechanism, and trait-specific features. high levels in multi-task architecture can represent more complex information @xcite . ## experimentation in this section, we first introduce the experimental settings and then report the experimental results. finally, we analyze the results on different aspects. ## conclusion we first annotate a multi-trait essay scoring dataset acea from the topic, organization, logic, and language aspects. and then we propose a multi-task learning framework hmts for the chinese aes task. moreover, we propose an inter-sequence attention mechanism to enhance information interaction between the different trait tasks and design the trait-specific features for various trait tasks. the experimental results show that our hmts can effectively score essays from multiple traits, outperforming several strong models. in the future, we will focus on incorporating the traits task into the overall task in a more effective framework.