Overview of NLPTEA -2020 Shared Task for C hinese Grammatical Error Diagnosis (2020)

Abstract: This paper presents the NLPTEA 2020 shared task for Chinese Grammatical Error Diagnosis (CGED) which seeks to identify grammatical error types, their range of occurrence and recommended corrections within sentences written by learners of Chinese as a foreign language. We describe the task definition, data preparation, performance metrics, and evaluation results. Of the 30 teams registered for this shared task, 17 teams developed the system and submitted a total of 43 runs. System performances achieved a significant progress, reaching F1 of 91% in detection level, 40% in position level and 28% in correction level. All data sets with gold standards and scoring scripts are made publicly available to researchers.

## introduction automated grammar checking for learners of english as a foreign language has achieved obvious progress. helping our own (hoo) is a series of shared tasks in correcting textual errors @xcite . the shared tasks at conll 2013 and 2014 focused on grammatical error correction, increasing the visibility of educational application research in the nlp community @xcite . many of these learning technologies focus on learners of english as a foreign language (efl), while relatively few grammar checking applications have been developed to support chinese as a foreign language (cfl) learners. those applications which do exist rely on a range of techniques, such as statistical learning @xcite @xcite , rule-based analysis @xcite , neuro network modelling @xcite and hybrid methods @xcite . in response to the limited availability of cfl learner data for machine learning and linguistic analysis, the icce-2014 workshop on natural language processing techniques for educational applications (nlp-tea) organized a shared task on diagnosing grammatical errors for cfl @xcite . a second version of this shared task in nlp-tea was collocated with the acl-ijcnlp-2015 @xcite , coling-2016 @xcite . its name was fixed from then on: chinese grammatical error diagnosis (cged). as a part of ijcnlp 2017, the shared task was organized @xcite . in conjunction with nlp-tea workshop in acl 2018, cged was organized again @xcite . the main purpose of these shared tasks is to provide a common setting so that researchers who approach the tasks using different linguistic factors and computational techniques can compare their results. such technical evaluations allow researchers to exchange their experiences to advance the field and eventually develop optimal solutions to this shared task. the rest of this paper is organized as follows. section 2 describes the task in detail. section 3 introduces the constructed data sets. section 4 proposes evaluation metrics. section 5 reports the results of the participants' approaches. conclusions are finally drawn in section 6. ## task description the goal of this shared task is to develop nlp techniques to automatically diagnose (and furtherly correct) grammatical errors in chinese sentences written by cfl learners. such errors are defined as pads: redundant words (denoted as a capital "r"), missing words ("m"), word selection errors ("s"), and word ordering errors ("w"). the input sentence may contain one or more such errors. the developed system should indicate which error types are embedded in the given unit (containing 1 to 5 sentences) and the position at which they occur. each input unit is given a unique number "sid". if the inputs contain no grammatical errors, the system should return: "sid, correct". if an input unit contains the grammatical errors, the output format should include four items "sid, start_off, end_off, error_type", where start_off and end_off respectively denote the positions of starting and ending character at which the grammatical error occurs, and error_type should be one of the defined errors: "r", "m", "s", and "w". each character or punctuation mark occupies 1 space for counting positions. example sentences and corresponding notes are shown as ## data sets the learner corpora used in our shared task were taken from the writing section of the hsk (pinyin of hanyu shuiping kaoshi, test of chinese level) @xcite . native chinese speakers were trained to manually annotate grammatical errors and provide corrections corresponding to each error. the data were then split into two mutually exclusive sets as follows. in addition to the data sets provided, participating research teams were allowed to use other public data for system development and implementation. use of other data should be specified in the final system report. test set: this set consists of testing units used for evaluating system performance. ## performance metrics the criteria for judging correctness are determined at three levels as follows. besides the traditional criteria in the past share tasks, correction-level was introduced to cged since 2018. the following metrics are measured at all levels with the help of the confusion matrix. ## evaluation results in cged 2020, the implementation of pre-trained model like bert achieved significant improvement in many tracks. the "standard pipe-line" bilstm+crf in cged2017 and 2018 is replaced. hybrid methods based on pre-trained model were proposed by most of the teams. resnet, graph convolution network and data argumentation appeared for the first time in the solutions. the rethinking the data construction (including pseudo data generation) and feature selection did not attract the attention of the participants. however, the balance of the fpr and other track did not progress a lot. the rough merging strategies implemented in hybrid methods and the over generation of generation models may lead the drop in fpr. from organizers' perspectives, a good system should have a high f1 score and a low false positive rate. in summary, none of the submitted systems provided a comprehensive superior performance using different metrics, indicating the difficulty of developing systems for effective grammatical error diagnosis, especially in cfl contexts. it is worth noting that in the track of detection, the performance over 0.9 is close to the application of actual scene. in the highly focused track of position and correction, variant teams lead the ranks, unlike the past cgeds. it's a very exciting phenomena indicating the attraction the task increased quickly. ## conclusion this study describes the nlp-tea 2020 shared task for chinese grammatical error diagnosis, including task design, data preparation, performance metrics, and evaluation results. regardless of actual performance, all submissions contribute to the common effort to develop chinese grammatical error diagnosis system, and the individual reports in the proceedings provide useful insights into computer-assisted language learning for cfl learners. we hope the data sets collected and annotated for this shared task can facilitate and expedite future development in this research area. therefore, all data sets with gold standards and scoring scripts are publicly available online at http://www.cged.science .