Error syntax aware augmentation of feedback comment generation dataset (2023)

Abstract: This paper presents a solution to the GenChal 2022 shared task dedicated to feedback comment generation for writing learning. In terms of this task given a text with an error and a span of the error, a system generates an explanatory note that helps the writer (language learner) to improve their writing skills. Our solution is based on fine-tuning the T5 model on the initial dataset augmented according to syntactical dependencies of the words located within indicated error span. The solution of our team ‘nigula’ obtained second place according to manual evaluation by the organizers.

## introduction feedback comment generation for language learners is the task of generating an explanatory note that helps the writers (language learners) improve their writing skills for text with error and a span of this error. in terms of genchall2022 1 , the target language is english, but this task applies to any language. moreover, this task is mostly designed not for such cases as simple typos or misspellings (something which can be easily detected by standard grammar-error detection systems) but for erroneous, unnatural, or problematic words. see detecting the aforementioned misuse of words is not enough to prevent the same error in the future. it is important to provide some explanation and reference to the error-specific rule and give a direct hint on which correct word should be used in this particular case. while any language follows certain rules which can be encoded manually to obtain the solution relying on rules and templates, in our work we try to use the benefit of the availability of parallel data and explore the limits of modern seq2seq models with the minimal number of rules and manual labor involved. the contributions of our work are as follows: ## related works the first attempts to provide feedback on a particular error type were based on rules @xcite . in morgado da @xcite authors used the english resource grammar parser to analyze the learner's sentence. if the parser fails to process a sentence, this sentence is supposed to have an error, and, so-called, mal-rules are used to detect the particular type of error. if the mal-rule works, the user is provided with a mal-rule-specific comment. the steps towards the usage of more modern approaches, such as neural networks, have also been performed in this task. in @xcite , the authors use a combination of basic machine learning approaches to detect errors and rules to provide feedback on some common types of errors. @xcite showed that a neural retrieval-based method can be effective in preposition feedback comment generation. @xcite proposed a method that provides context-sensitive explanations using grammatical error correction and templates. @xcite suggested methods for automatically choosing feedback templates based on learning history. in @xcite , the sentence with an error is first corrected with ## task description in this section, we introduce the formal definition of the task and the dataset provided for it. ## our method description in this section, we introduce the main steps for training the model used for the final submission. these steps are also shown in figure 1 . ## results the results of the system output are scored automatically and manually. automatic and manual scoring compares the system's outputs with manually created feedback comments. the automatic approach uses the bleu score. in terms of manual scoring, a system output is regarded as appropriate if it contains information similar to the reference and does not contain information that is irrelevant to the offset; it may contain information that the reference does not contain as long as it is relevant to the offset. if these conditions are met, the output is labeled as correct. the task definition (see section 3.1) allows systems to @xmath0. that is why the final score is calculated as precision, recall, and f1-score. we do not make any filtering of the generated feedback, thus there is no case when our system @xmath1, so all metrics are equal. refer to it is also worth mentioning that the approach of using double brackets as a signal of the exact location of the error span to train the model worked well. to be more precise, the system always generates different feedback for similar sentences with different error spans and 12 out of 20 sentences in the test set (similar sentences with different slots were presented by pairs, so in total there are ten pairs of such samples) were scored as correct by organizers. examples of such sentences can be found at ## future work there are several possible ways to improve the proposed data augmentation approach, that we leave for future work. in our approach, we use the large language model to generate a new text using the prompt that contains a grammar error. the error could generally affect the quality of the generated text, which is why it could be interesting to first automatically correct the error in the clipped sentence, use the language model to generate a new version of the sentence, and then replace the corrected word with the erroneous word in the new sentence. another promising direction of the improvement of the augmentation approach is to apply changes not only to the right part of the error span but also to the left part. this could be done, for example by filling the masks placed on the position of random words that are not syntactically related to the words within the error span. the amount of data generated for our experiments was based on a "the good the better" basis. however, it is also worth studying the relation between the amount of augmented data and the improvement in the quality of the trained model. ## conclusion in this paper, we present our solution for genchal 2022 shared task dedicated to feedback comments generation to improve the english language learning experience. our solution uses the error span based preprocessing of the learner's text, augmentation of the dataset by clipping of the learner's text w.r.t syntactic dependency to the words within the error span, and then the inference of large language model, using clipped text as a prompt, and finally training large t5-based model with both initial and augmented version of the dataset. our solution took second place in this competition according to manual evaluation by organizers. the model and code of our experiments are open-sourced. we also share the track of unsuccessful experiments and general ideas about alternative approaches to this task to prepare the ground for future researchers.