Leveraging Denoised A bstract M eaning R epresentation for Grammatical Error Correction (2023)

Abstract: Grammatical Error Correction (GEC) is the task of correcting errorful sentences into grammatically correct, semantically consistent, and coherent sentences. Popular GEC models either use large-scale synthetic corpora or use a large number of human-designed rules. The former is costly to train, while the latter requires quite a lot of human expertise. In recent years, AMR, a semantic representation framework, has been widely used by many natural language tasks due to its completeness and flexibility. A non-negligible concern is that AMRs of grammatically incorrect sentences may not be exactly reliable. In this paper, we propose the AMR-GEC, a seq-to-seq model that incorporates denoised AMR as additional knowledge. Specifically, We design a semantic aggregated GEC model and explore denoising methods to get AMRs more reliable. Experiments on the BEA-2019 shared task and the CoNLL-2014 shared task have shown that AMR-GEC performs comparably to a set of strong baselines with a large number of synthetic data. Compared with the T5 model with synthetic data, AMR-GEC can reduce the training time by 32% while inference time is comparable. To the best of our knowledge, we are the first to incorporate AMR for grammatical error correction.

## introduction nowadays, high performance of grammatical error correction model mainly depends on data augmentation @xcite @xcite @xcite . according to the type of additional information, grammatical error correction models can be divided into dataenhanced models and knowledge-enhanced models. data-enhanced models require millions of synthetic data, which is obtained by back-translation or directly adding noise. training on these synthetic datasets is very time-consuming, which is unacceptable in some application scenarios. knowledgeenhanced model is to artificially design a large number of grammatical rule templates, and add the templates as external knowledge to gec model. this external knowledge is language-dependent and it requires the intervention of human grammar experts. abstract meaning representation (amr) is a type of rooted, labeled graph which contains semantic structures with fine-grained node and edge types. amr breaks through the limitations of the traditional syntax tree structure and supports reentrancy. figure 1 is a graph of sentence "i don't want to go to school on sunday.". in amr, :arg0 is typically the agent, :arg1 is typically the patient, and other arguments do not have standard definitions and may vary with the verb being annotated. negative meaning is denoted as "-". special keywords such as entity types, quantities and logical conjunctions are supported by amr. amr obtains a simple representation from natural language sentence and it is suitable for gec as extra knowledge. a non-negligible concern is that amrs of errorful sentences may not be exactly reliable. if these amrs with errors are directly introduced into the gec model as additional information, it may confuse the model. we use a pre-trained amr parser to predict amr of erroneous sentences and corrected sentences separately on the bea-19 development set. if two amrs are completely consistent, we assume that the amr of errorful sentences is reliable. after statistical analysis, we found that about half of the graphs are reliable. we designed a denoising semantic aggregated grammatical error correction model. specifically, we added a graph aggregation encoder based on a sequence-to-sequence model. the graph encoder aims to update the representation of the sequence encoder by amr semantic structure. besides, we designed two mask strategies to reduce the dependence on the model graph information. we designed these mask strategies by granularity: node/edge level mask and subgraph level mask. experiments have proved that the denoising semantic aggregated grammatical error correction model significantly improved the error correction accuracy. ## model we add a graph encoder based on transformer to aggregate denoised semantic information. the architecture of amr-gec is shown on figure 2 . ## denoising function masked language modeling (mlm) is a classic pre-trained model modeling method. the task of mlm is to mask some tokens with a special token mask and train the model to recover them. this allows the model to handle both the left and right context of the masked token. mlm can divided into five types: single word masking, phrase making, random span masking, entity masking, whole word masking. referring to @xcite , we use the mask strategy on amr. we used two ways to add masks: node/edge level mask and sub-graph level mask. node/edge level mask refers to mapping the nodes/edges in the amr graph using a noise function to generate a graph with noise. sub-graph level mask means randomly removing subgraphs and replacing them with a mask label. ## conclusion in this paper, we propose a denoising semantic aggregated grammatical error correction model, amr-gec, leveraging amr as external knowledge to the gec. we believe it gives a strong baseline for incorporating amr in gec.