E r AC on D : Error Annotated Conversational Dialog Dataset for Grammatical Error Correction (2022)

Abstract: Currently available grammatical error correction (GEC) datasets are compiled using essays or other long-form text written by language learners, limiting the applicability of these datasets to other domains such as informal writing and conversational dialog. In this paper, we present a novel GEC dataset consisting of parallel original and corrected utterances drawn from open-domain chatbot conversations; this dataset is, to our knowledge, the first GEC dataset targeted to a human-machine conversational setting. We also present a detailed annotation scheme which ranks errors by perceived impact on comprehension, making our dataset more representative of real-world language learning applications. To demonstrate the utility of the dataset, we use our annotated data to fine-tune a state-of-the-art GEC model. Experimental results show the effectiveness of our data in improving GEC model performance in a conversational scenario.

## introduction in recent years, both researchers and businesses have attempted to build effective educational chatbots to help language learners improve their conversational skills in a second language (primarily english) @xcite . however, many such systems, such as genietutor plus @xcite , use rule-based dialog engines, and thus do not take advantage of recent developments in dialog generation using transformer models, which have vastly improved the quality of modern chatbots @xcite . extant dialog systems for conversational language learning can be broadly classified into two types. in the first type, the chatbot serves as a teacher and repeatedly asks the user questions to test acquisition of specific words, syntax, and other pedagogical targets. in the second type, the chatbot serves as a conversational partner, encouraging users to chat with it and, in some cases, providing corrective feedback to learners @xcite . it is this latter type we hope to improve using our proposed dataset. grammatical error correction (gec) models are needed to generate appropriate corrective feedback for this second type of educational chatbot. however, nearly all current gec datasets focus on written essays, a domain which differs markedly from conversational speech in both syntax and style. as a result, datasets drawn from written sources, such as student essays, produce poor results when applied to dialog @xcite . there currently exists one dataset of errorannotated conversational utterances by english second language learners on which researchers can train and evaluate conversational gec models, the teacher-student chatroom corpus @xcite ; this data is generated in the context of human-human interaction, specifically interactions between a teacher and a second language learner student. however, no similar conversational dataset focuses on the human-machine conversational setting. in this work we seek to address this gap in the available data by developing a highquality, error-annotated dataset of learner dialog collected from an online educational chatbot. 1 to appropriately annotate our data for language learning applications, we introduce a 3-level grammatical error classification structure in order to categorize errors based on severity. our motivation for this error classification structure is to give users the opportunity to first focus on improving their most serious grammatical errors. to demonstrate the utility of the proposed dataset, we fine-tune and evaluate a state-of-the-art gec model using our newly developed dataset. ## related work as with many nlp tasks, the current state-of-theart in grammatical error correction involves using large transformer-based language models such as bert @xcite , roberta @xcite , and xlnet @xcite . to evaluate the utility of our dataset, we use omelianchuk et al. (2020)'s gector model, which reframes gec as a sequence labelling task rather than a monolingual machine translation task. gector achieves sota results on the test corpus used for the bea 2019 shared task on grammatical error correction @xcite . other promising supervised gec models include those of @xcite and @xcite , who achieve sota results on the jfleg @xcite and @xcite gec datasets, respectively. both models combine innovative synthetic data generation methods with large pretrained transformer language models. recent work related to the development of datasets for grammatical error correction include @xcite who presents a dataset of native and non-native english writing. trinh and rozovskaya (2021) proposes a new parallel dataset of russian student writing. these datasets add to the growing number of gec datasets available to the research community. however, as previously mentioned, no gec dataset that contains chatbotbased human-machine conversational data, in english or any other language, is currently available. we seek to begin closing this gap with the present research. ## dataset statistics as shown in ## grammar error correction model to demonstrate the utility of our proposed dataset in improving gec for the open-domain dialog setting, we use the eracond dataset to finetune a state-of-the-art gec model, gector @xcite , which we then test on held-out dialog data. our results show that era-cond is useful for adapting gec models to opendomain dialog. ## conclusions and future work we provide the first high-quality, fine-grained error-correction conversation dataset between english second language learner and an educational chatbot. to demonstrate the utility of our dataset, we train and evaluate a sota gec model on the dataset, resulting in a significant improvement in overall model performance for conversational setting. this project lays the groundwork for future work on conversational grammatical error correction (such as adding other dialog domains and incorporating information about the native languages of users) and customized educational dialog system for second language learners.