UMRS pell: Unifying the Detection and Correction Parts of Pre-trained Models towards C hinese Missing, Redundant, and Spelling Correction (2023)

Abstract: Chinese Spelling Correction (CSC) is the task of detecting and correcting misspelled charac- ters in Chinese texts. As an important step for various downstream tasks, CSC confronts two challenges: 1) Character-level errors consist not only of spelling errors but also of missing and redundant ones that cause variable length between input and output texts, for which most CSC methods could not handle well because of the consistence length of texts required by their inherent detection-correction framework. Con- sequently, the two errors are considered out- side the scope and left to future work, despite the fact that they are widely found and bound to CSC task in Chinese industrial scenario, such as Automatic Speech Recognition (ASR) and Optical Character Recognition (OCR). 2) Most existing CSC methods focus on either detector or corrector and train different mod- els for each one, respectively, leading to in- sufficiency of parameters sharing. To address these issues, we propose a novel model UMR- Spell to learn detection and correction parts together at the same time from a multi-task learning perspective by using a detection trans- mission self-attention matrix, and flexibly deal with both missing, redundant, and spelling er- rors through re-tagging rules. Furthermore, we build a new dataset ECMR-2023 containing five kinds of character-level errors to enrich the CSC task closer to real-world applications. Ex- periments on both SIGHAN benchmarks and ECMR-2023 demonstrate the significant effec- tiveness of UMRSpell over previous represen- tative baselines.

## introduction recent decades have witnessed the comprehensive development of one important neural language processing (nlp) task: chinese spelling correction (csc), which focus on detecting and correcting spelling errors in texts @xcite . the † equal contributions. * corresponding author. early work on csc includes but not limited to pipeline strategy @xcite , traditional language model @xcite , and sequenceto-sequence learning @xcite . with the growth of the deep learning techniques and the insight for the task itself, the subsequent studies gradually summarize and focus on two crucial parts of csc, defined as detection part and correction part. generally, a csc system derives representations (pronunciation, pinyin, glyph, shape, strokes, and so on) for characters from audio and visual modalities to locate the misspelled ones in its detection part, and then output the text without spelling errors in its correction part @xcite @xcite . currently, since the pre-trained masked language models with attention mechanism @xcite achieve impressive performance in many nlp domains, they are also introduced into csc @xcite @xcite . despite its development, there are still two weaknesses in csc that bring bottleneck in performance: firstly, in real world scenario, especially for asr and ocr tasks, there is not only misspelled case but also missing and redundant errors happening in character-level preprocessing. missing characters mean a lack of characters needed to be inserted into the identified position, while redundant characters mean useless or repeated characters needed to be deleted @xcite . most existing csc literatures @xcite @xcite @xcite consider the issue belonging to another similar but more complicated task called chinese grammatical error correction (cgec) @xcite @xcite and leave them in the future work. 1 the key point is that dealing with such errors might change the length of the input text, while csc methods based on detection-correction framework could not handle it well due to inconsistency between inputs and outputs of their correction-part @xcite . secondly, detection-correction framework-based methods usually concentrate on how to utilize more useful features in either of two parts and might pretrain or fine-tune different language models for each part, respectively, leading to insufficiency of information sharing between detection and correction parts. in this paper, we proposed a novel pre-trained model, abbreviated as umrspell, to unify detection and correction parts for chinese missing, redundant, and spelling correction. umrspell considers each of detection and correction parts as a sub-task and pre-trains both parts based on one backbone together. inspired by a multi-task idea in multilingual learning @xcite , we modify the back-translation masked language modeling (btmlm) objective to obtain mutual information from the concatenated input texts. moreover, to solve the variable text length issue caused by missing or redundant errors, we apply several methods during the training and predication processes, such as symmetric concatenation, sequence re-tagging rules, and post consistence check. additionally, an automatically optimized weighted parameter λ is used for the loss calculation, instead of manually selection. finally, we construct the extended csc dataset with missing and redundant errors (ecmr-2023) in an unsupervised way with the help of a self-selected phonetic and morphological chinese character similarity dictionary. our contributions are concluded as follows: ## related work early work on csc follows the pipeline of error identification, candidate generation and selection, or selects candidates in an unsupervised way from a confusion set. then methods utilizing traditional language models or considering sequenceto-sequence structure are also proposed @xcite @xcite @xcite . reviewing the studies in recent years, we further summarize csc methods into seven types shown in figure 1 , mainly depending on which part of detection-correction framework they primarily focus on: inheriting ideas from earlier work, methods of type-1 concern dataset and confusion set. automatic corpus generation proposes a hybrid way to generate labeled spelling errors @xcite . confusionset-guided pointer networks utilizes the confusion set for guiding the character generation @xcite . methods of type-2 discuss about how to combine embeddings from multi-modal features better for the subsequent detection or correction models. mlm-phonetics is pre-trained to provide embeddings including pronunciation and pinyin of characters @xcite . phmospell derives and integrates pinyin and glyph representations to a language model by an adaptive gating mechanism @xcite . realise captures and selectively mixing the semantic, phonetic and graphic information of characters @xcite . methods from both type-3 and type-4 introduce useful features of characters into general detection process. the difference between them lies in the order in which they guide and affect correction models. being a typical method of type-3, soft-masked bert (smbert) designs a soft masking process after detection part, calculating the weighted sum of the input and [mask] embeddings weighted by the error probabilities to mask the likely errors in the sequence @xcite . alignment-agnostic model adds a modification logic unit following detection network to reformulate the sequence to support missing and redundant cases @xcite . as a type-4 method, spellgcn builds a graph over the characters and maps it into a set of interdependent detection classifiers that are applied to the representations extracted by bert @xcite . dynamic connected networks generates the candidate characters via a pinyin-enhanced generator and model the dependencies between characters through attention mechanism @xcite . mdcspell captures features of characters and fuses the hidden states of corrector with that of detector to minimize the misleading impact from the misspelled ones @xcite . the output length of detector from type-4 needs to be consistent with that of corrector, while methods of type-3 might avoid this issue by adding a unit to re-tag the text from detector before corrector. methods of type-5 directly handle complicated features in their correction part. chunk-based model designed a decoding part and generates all possible chunk candidates for the partially decoded correction @xcite . craspell forces correction model to yield similar outputs based on original contexts and constructed noisy ones @xcite . methods of type-6 introduce prior knowledge and feature information into either detection or correction parts by training large language models. faspell consists of a denoising auto-encoder and a decoder, and fine-tunes the masked language model in novel ways @xcite . spellbert fuses pinyin and radical features through a relational graph convolutional network and train them with a 4-layer bert @xcite . plome is a task-specific language model that jointly learns semantics and misspelled knowledge according to the confusion set based on specially designed masking strategy on bert @xcite . recently, methods focus on the learning strategy, such as curriculum learning @xcite and adversarial learning @xcite , make progress and are categorized in type-7. according to above taxonomy, the proposed umrspell could be regarded as a combination between type-3 and type-6. ## methodology in this section, we first formulate the task, then dive into the structure of umrspell, introducing its training and prediction processes. furthermore, we present how to built ecmr-2023. ## conclusion in this paper, we propose umrspell to learn detection and correction parts together with a detection transmission self-attention matrix, and flexibly deal with chinese missing, redundant, and spelling errors through re-tagging rules. we further construct a dataset ecmr-2023 containing five kinds of spelling errors to enrich the existing csc task. experiments on both sighan benchmarks and ecmr-2023 demonstrate the significant effectiveness of umrspell.