Automatic Comment Generation for C hinese Student Narrative Essays (2022)

Abstract: Automatic essay evaluation can help reduce teachers’ workload and enable students to refine their works rapidly. Previous studies focus mainly on giving discrete scores for either the holistic quality orseveral distinct traits. However, real-world teachers usually provide detailed comments in natural language, which are more informative than single scores. In this paper, we present the comment generation task, which aims to generate commentsfor specified segments from given student narrative essays. To tackle this task, we propose a planning-based generation model, which first plans a sequence of keywords, and then expands these keywords into a complete comment. To improve the correctness and informativeness of generated comments, we adopt two following techniques: (1) training an error correction module to filter out incorrect keywords, and (2) recognizing fine-grained structured features from source essays to enrich the keywords. To support the evaluation of the task, we collect a human-written Chinese dataset, which contains 22,399 essay-comment pairs. Extensive experiments show that our model outperforms strong baselines significantly. Moreover, we exert explicit control on our model to generate comments to describe the strengths or weaknesses of inputs with a 91% success rate. We deploy the model at http://coai.cs.tsinghua.edu.cn/static/essayComment/ . A demo video is available at https://youtu.be/IuFVk8dUxbI . Our code and data are available at https://github.com/thu-coai/EssayCommentGen .

## introduction automatic essay evaluation is a useful educational application of natural language processing @xcite , which is beneficial for reducing teachers' workload and enabling students to improve writing title: ;<9=> (the lights of home) body: ... ?@abcdefgehijklmn%opq%r jlst%@xmath6,g9% dhi9%djklm9%dnopq 9%urstj.?@abcdefgeui (in the morning there are few people and it is better to go in the evening. in the evening, the lights come on. red, white, green and yellow colored lights are like blooming flowers, which are on 12 zodiac signs, plants, restaurants and famous buildings. all of them are so beautiful.) ... given an essay with a specified segment (separated from the rest using special @xmath7, the model should generate a sentence to comment the strengths or weaknesses of the segment. skills independently. prior studies focus mainly on automatic essay scoring (aes) in terms of either holistic scores @xcite or traitspecific scores @xcite . however, real-world teachers usually provide detailed comments in natural language, which are more informative so that students can know more about the strengths and weaknesses of their works. in this work, we present the first study on automatic comment generation, which requires generating a fluent comment in natural language to describe strengths or weaknesses for a specified segment from a given student essay, as exemplified in figure 1 . we only focus on narrative essays in this work, which comprise more than 90% of our originally collected essays. the challenges of the task mainly lie in the following three folds: (1) capturing the linguistic features of the essay, ranging from the wording, rhetorical methods (e.g., "simile" in the example) to discourse structures. (2) generating coherent comments to correctly reflect the strengths or weaknesses (e.g., the segment does not use idioms in the example) of the essay. (3) generating informative and diverse comments since generic comments such as "it is good" do not pro-vide any helpful guidance for students, and it is also expected to generate diverse comments for different essays. to tackle the problem, we propose a planningbased generation model @xcite , which first plans a sequence of keywords concerning specific writing skills such as "simile", and then expands the keywords into a coherent comment. on the one hand, planning helps build explicit connections between essays and underlying skills, which alleviates the degeneration issue that the model focuses on predicting common elements such as "more vivid" and tends to generate generic comments @xcite . on the other hand, we can exert direct control on the intermediate keywords to improve the correctness and informativeness of generated comments. specifically, we fine-tune bert @xcite to serve as an error correction module to filter out incorrect keywords, which is trained to discriminate matching relations between essays and keywords. moreover, we recognize structured features from source essays in terms of idioms, proverbs, quotes, descriptive and rhetorical methods using heuristic techniques or pretrained classifiers. then we combine these features with the predicted keywords. to control the type of generated comments, we insert a binary control code before the keywords and comments (0/1 for describing strengths/weaknesses). in the comment generation stage, we inject noise into the ground-truth keywords during training @xcite to alleviate the exposure bias issue introduced by planning @xcite . to support training and evaluation of the proposed task, we collect a chinese dataset that contains 22,399 essay-comment pairs. extensive experiments show that our model outperforms strong baselines in correctness, informativeness and diversity. furthermore, we build a website to enable realtime interaction with our deployed model, where a user can upload a chinese student essay and see comments along with recognized structured features for most paragraphs. ## related work automatic essay scoring there have been wide explorations for automatic essay scoring, including holistic essay scoring and trait-specific essay scoring @xcite . holistic essay scoring aims to assign an overall score for the essay. @xcite ; @xcite used lstm and @xcite ; @xcite used cnn to give a total score for the essay. @xcite utilized word embedding clusters and string kernels to achieve strong performances. @xcite jointly resolved the essay scoring task and the essay ranking task through fine-tuning the bert model. trait-specific essay scoring aims to assign different scores for different traits of an essay, such as thesis clarity @xcite , style @xcite and narrative quality @xcite . @xcite compared different trait-agnostic approaches to automatically score many different essay traits. however, all these works assign numeric scores for an essay, while we focus on generating a readable comment. essay assessment systems @xcite constructed a system named e-rater, which could provide numeric scores for different features such as grammar and style. lingglewrite @xcite focused on grammatical error correction and automatic essay scoring. the system most similar to ours is iflyea @xcite , which has grammar level analysis techniques and components for discourse and rhetoric analysis. it also integrates the fine-grained analysis to form a review for the whole essay using templates. however, our system is capable of generating diverse and natural comments without the usage of templates and could give comments for different segments of the essay. planning-based generation humans usually outline the overall framework before writing. many works have explored planning-based text generation, which first predicts an intermediate representation as a plan and then generates the complete text conditioned on the plan. the plan could be a series of keywords @xcite , an action sequence @xcite or a dense keyword distribution @xcite . @xcite progressively refined the produced domain-specific content keywords into complete passages in multiple stages. in this paper, we adapt planning to the automatic comment generation task and improve the correctness and informativeness by revising the intermediate keywords. ## method we formulate our task as follows: given an essay @xmath8, x 2 , • • • , x m ) with m tokens and a specified segment from x i to x j (the segment is separated from the rest using special tokens), the model should generate a comment @xmath9, y 2 , • • • , y n ) with n tokens for the segment. the comment either shows praise for the strengths of the segment, or gives advice to improve the weaknesses of the segment. to improve the correctness and informativeness of generated comments, we modify the keyword sequence by first filtering out incorrect keywords using the error correction module and then inserting structured features from the feature recognition module. then we feed the polished keyword sequence into the comment generator along with the original essay to get the final comment. during training, we train different modules separately. ## demonstration a screenshot of our demo website is shown in figure 3 . after entering the title and the body of the article, the user can submit a request and get the result after a few seconds. we comment on all paragraphs except those that contain less than 15 chinese characters and do not have any recognized structured features. also, we show recognized fine-grained structured features on the right. the sentences and keywords corresponding to these features are underlined and marked green. with these comments and fine-grained features, the user can fully understand the essay's strengths and weaknesses. besides the demo website, we also create a github repository at https://github. com/thu-coai/essaycommentgen , where users can freely use our code and data under the mit license. ## conclusion we present a planning-based model for a new task named essay comment generation, which first plans a sequence of keywords and then expands these keywords into a complete comment. furthermore, we utilize an error correction module and a feature recognition module to modify the generated keywords for improving the correctness and informativeness of final comments. we manually collect a new chinese dataset for this task. extensive experiments show that our model outperforms strong baselines. we have deployed our model online to help with the automatic essay evaluation. we expect our work to facilitate further research on this new task and benefit both teachers and students. !"#@xmath0a:oe•ž•,•'/'"'y )#@xmath15 0žf ¶•¸2opqrs/0 †?¹ºj5» † ¼½0a()¾¿à¾¿áy)#@xmath2é!"#@xmath3/0êop[\ë!"#@xmath4/ 0 †ønùú ûüýþßàáßâãä åaeýçèááßâãä éêýãëìßä ``ab íê¨aeîï/ ae¿¡ðñò5 óbýþãôßâàãõä ¦ †,²³: ˜™x´() ••,µ@xmath5/0 †ønùú "#$%&ú 'x(a)* ,/0 †w+ ')*îï( …,-,.// y0úû)1 mü235 4bý5ß67âßþþä figure 5: an example of the essay with comments annotated by human. the source essay is on the left side and the commented segments are bolded and underlined. the segments along with the comments are shown on the right. comments could point out the strengths or weaknesses of the segments.