Feedback comment generation using predicted grammatical terms (2023)

Abstract: The purpose of feedback comment generation is to provide useful feedback comments for a wide range of errors in learners’ essays from a language learning perspective. Since it is difficult to obtain appropriate comments at a practical level with rule-based or retrieval- based methods, we explore neural-based gen- erative methods with pre-trained models. We further assume the effectiveness of consider- ing grammatical terms in generating feedback comments. Specifically, this paper proposes T5-based models using predicted grammati- cal terms, submitted to FCG GenChal, and presents their results. By using correct gram- matical terms, our model could improve the BLEU score by 19.0 points, compared with the baseline T5 without grammatical terms on the development dataset. Furthermore, by using predicted grammatical terms, our model could improve the manual evaluation score by 2.33 points, compared with the baseline T5 without grammatical terms on the test dataset.

## introduction feedback comment generation (fcg) for writing studies is a task to generate explanations of why ungrammatical sentences written by language learners are incorrect and on what basis a correction was made. related research has been mainly conducted on the basis of a dataset published by nagata (2019). however, the accuracy, which is based on manual evaluation (me), so far is insufficient for real-world use. one reason is that the data in the dataset are not necessarily sufficient to cover all error types and feedback comments. since annotating feedback comments would require specialized knowledge in writing, constructing larger datasets is time-consuming and labour-intensive. fcg genchal (nagata et al., 2021) targets the generation of feedback comments for prepositional errors. an example feedback comment for the prepositional error is shown in the following. target sentence: i agree on the idea. feedback comment: «agree on» indicates that more than one person agrees on a certain matter. <@xmath0. here, the words marked @xmath1. underlined words are the target word with an error that the feedback comment is generated for. to achieve this task, methods using an encoder-decoder model, which generates feedback comments from scratch, are considered effective because they can deal with various learners' error types even in unsophisticated settings. therefore, it is a good idea to develop the encoder-decoder model for generating feedback comments. state-of-the-art (sota) results have been achieved in various natural language processing tasks by using pre-trained models. however, there has been no previous work on fcg using pretrained models. a logical concern is what level of accuracy the sota pre-trained models can achieve in fcg. in grammatical error correction, which is highly related to fcg, methods using t5 (raffel et al., 2020) achieve good results. thus, in this work, we also utilize t5 for fcg. the system can more easily generate a feedback comment when some words in the target feedback comment sentence to be generated are known. one of such clue words for better generating feedback comments might be grammatical terms (gts), since commonly used gts are limited, and it might be possible to predict and use them to generate feedback comments. however, no research has focused on gts for fcg. therefore, we propose the following procedure for generating feedback comments in this study. first, gts to be used in the feedback comment are selected. as several gts are used in a feedback comment, this becomes a multi-label classification task. next, the selected gts are used to generate the feedback comment. to predict gts, we use roberta (liu et al., 2019) because roberta often achieves better accuracy than other pre-trained models in the multilabel classification task. t5 is then used to generate feedback comments since it can be used for text-totext tasks. the contributions of this research are therefore as follows: ## related work grammatical error correction is closely related to the fcg task. rothe et al. (2021) have achieved a high accuracy in grammatical error correction by using a pre-trained generative language model, t5. this suggests that fcg could also be handled by t5. a survey (hanawa et al., 2021) of the methods used in the fcg task investigated three methods: retrieval-based, retrieve-and-edit, and simple generation. the survey shows that the simple generation method performs best in generating feedback comments for prepositional errors and the retrievalbased method alone cannot cope with various errors present in the training examples in generating feedback comments. in generating feedback comments using a generative model, prompting the model with the predicted gts corresponding to the target error is likely to guide the direction for the generation. however, to the best of our knowledge, there have been no studies taking such an approach to fcg. ## feedback comment generation in fcg, t5 is used as the base model to predict ŷ given x and t . the input sequence to t5 is given in the following format: here, a special token "fbc:", which stands for a feedback comment, is used as a prompt to train the t5 model. the target span s is marked by square brackets. predicted gts for s are listed after s with another special token "<@xmath2. data instances sent toks. com toks. 5 experiments ## case study we investigated whether our model could generate correct feedback comments with the predicted gts in the development dataset. ## conclusion we explored neural methods for fcg using pretrained models. in this study, we showed predicting the gts and using them in generating feedback comments can be useful for feedback comment generation with t5. the results also suggested that further improvement in grammatical term prediction would be beneficial for fcg.