Identifying Feedback Types to Augment Feedback Comment Generation (2023)

Abstract: In the context of language learning, feedback comment generation is the task of generating hints or explanatory notes for learner texts that help understand why a part of text is erroneous. This paper presents our approach to the Feedback Comment Generation Shared Task, collocated with the 16th International Natural Language Generation Conference (INLG 2023). The approach augments the generation of feedback comments by a self-supervised identification of feedback types in a multitasklearning setting. Within the shared task, other approaches performed more effective, yet the combined modeling of feedback type classification and feedback comment generation is superior to performing eedback generation only.

## introduction several studies have dealt with identifying and correcting grammatical errors to help language learners improve their writing skills @xcite @xcite . however, these approaches do not provide learners with a rationale for why a piece of text is erroneous. to help learners better understand and adapt the underlying writing rules, @xcite introduced the task of feedback comment generation: given a learner text in which some span is known to be erroneous, automatically generate a comment containing helpful hints and explanations. specifically, the comment should prompt the learner to come up with a solution rather than pointing out an error (grammatical error detection) or correcting it (grammatical error correction). towards this end, the feedback comment generation shared task @xcite at the 16th international natural language generation conference (inlg 2023) has provided a corpus of erroneous english sentences written by non-native learners of english. each sentence comes with a feedback comment that is targeted towards a given position of the sentence. the focus is on errors related to the use of prepositions in order to restrict the extensive task of generating feedback to a manageable setting. the generated comments are supposed to explain to the writer why the text part in question is erroneous, possibly with related writing rules. one exemplary instance of the task looks as follows: input text "they can help their father or mother about money that we must use in the university too." feedback comment "«about» is not the @xmath0. look up the use of @xmath1." as our contribution to the shared task, we present an approach that relies on multitasklearning to simultaneously (a) classify the type of the target feedback for the given erroneous input sentence and (b) generate an appropriate feedback comment of this type. since no feedback type labels are given in the data, we tackle the type classification in a self-supervised manner. in particular, we apply an unsupervised clustering based on tf-idf vector representations of the feedback comments. each cluster is assumed to represent one feedback type. we then learn a mapping from input texts to feedback types. the rationale is that an explicit distinction between different types of feedback may help to generate targeted feedback comments per type and, hence, more diverse comments for different types. overall, the generated feedback comments may then better match the input text by exploiting the feedback patterns per comment type. our evaluation results in the shared task suggest that the combined modeling of feedback type classification and feedback comment generation is superior to performing feedback generation only. our approach improves over sequence-to-sequence baselines in automatic and manual evaluation. early research often targeted only on one common error type, such as incorrect article usage @xcite , preposition and determiner usage @xcite , singular and plural usage @xcite , or false friends @xcite . more recent work proposed approaches to detecting @xcite and correcting @xcite @xcite grammatical errors in general using largescale neural networks, including transformer-based language models. some works go beyond grammar to assess argumentative structures in learner texts @xcite @xcite . @xcite proposed the usefulness of rewriting language learner texts not only to correct errors but also to improve the fluency and naturalness of a text. the task of feedback comment generation, as proposed by @xcite , goes beyond detecting and correcting errors in that it includes to provide explanations for why some text part is erroneous. with this, language learners can better understand and adapt the underlying writing rules. @xcite compared a neural retrieval-based method to a sequence-to-sequence model and a hybrid of these two that edits retrieved feedback comments. they found that the sequence-to-sequence model works best in a setting with few feedback variations, for example, concerning preposition use only. at the same time, the hybrid approach seems most promising for general feedback generation. ## task and data this section summarizes the feedback comment generation shared task as well the data provided as part of the task. ## approach we now present our approach to feedback comment generation. its core idea is to classify the type of feedback to be given and to generate an according feedback comment simultaneously. ## evaluation this section reports on our experiments with joint feedback type classification and feedback comment generation before presenting the evaluation methods and results. ## conclusion this paper has described our approach to the feedback generation shared task collocated with the 16th international natural language generation conference @xcite . the key idea of our approach is to jointly model the classification of feedback types and the generation of feedback comments in order to exploit found patterns per comment type during the generation. our experiments suggest that the generation quality improves by modeling both tasks together. we also observed open issues, though, that indicate a wrong integration of parts of the input into the generated output. a refined control of the interaction of input and output may resolve such issues in future work.