G ated T ransformer for R obust D e-noised S equence-to- S equence M odelling (2021)

Abstract: Robust sequence-to-sequence modelling is an essential task in the real world where the inputs are often noisy. Both user-generated and machine generated inputs contain various kinds of noises in the form of spelling mistakes, grammatical errors, character recognition errors, all of which impact downstream tasks and affect interpretability of texts. In this work, we devise a novel sequence-to-sequence architecture for detecting and correcting different real world and artificial noises (adversarial attacks) from English texts. Towards that we propose a modified Transformer-based encoder-decoder architecture that uses a gating mechanism to detect types of corrections required and accordingly corrects texts. Experimental results show that our gated architecture with pre-trained language models perform significantly better that the non-gated counterparts and other state-of-the-art error correction models in correcting spelling and grammatical errors. Extrinsic evaluation of our model on Machine Translation (MT) and Summarization tasks show the competitive performance of the model against other generative sequence-to-sequence models under noisy inputs.

## introduction noisy texts are very common in user-generated texts that appear abundant in various social media platforms like short message service (sms), online chat, email, blogs, wikis etc. these kind of texts may contain spelling errors, abbreviations, non-standard terminology, false starts to name a few. most of the nlp models assume the data to be linguistically correct and semantically coherent. thus, noisy texts pose a serious threat in ensuring accurate predictions and practicality of any nlp system in real-life applications. automatic noise correction from texts is thus crucial in * the author was employed at optum global advantage, india during the entire work. to devise a robust technique for noise removal and corrected target generation, we propose gated-trans (g-trans), a gated transformer sequence-to-sequence model. our model uses a mask gate based on a pre-trained transformer encoder to detect noises within texts, and a pre-trained transformer decoder to generate noise-free target sequence. the decoder uses a copy gate to determine whether to copy an output token directly from the input, a generate gate to generate new output token for a masked (contextually incorrect) token and a skip gate to skip tokens that are contextually irrelevant. we evaluate our model on real-life noisy texts generated from ocr engines, as well as artificial augmentation based noises @xcite ) that replicate the real-life user generated noises. further, our extrinsic evaluation on the noisy machine translation (mt) and summarization tasks shows the robustness of our model on translating noisy texts into correct generated target. contribution: we contribute to the existing body of noise removal task in several ways. ## related work in this work we adopt the definition of noise in text from @xcite as any kind of difference between the surface form of a coded representation of the text and the correct text. as more and more noisy text data being generated in various social communication media, removing noises from these texts have become an increasingly important task. existing methodologies for noise removal from texts can be divided into two categories -classifier based approaches and statistical machine translation (smt) based approaches. traditional classifier-based approaches @xcite and finetuned smt based methods @xcite @xcite are not generalized enough to correct different types of noises from texts @xcite and require huge parallel corpora. thus in smt-based approaches usage of language models pre-trained on monolingual corpora has become very popular. etoori et al. @xcite propose a seq2seq-based deep learning model to perform spell correction automatically in resourcescare languages like hindi and telegu. krishna et al. @xcite propose a post-ocr text correction approach based on seq2seq model for digitising texts using romanised sanskrit, the lack of resources has made them use ocr models trained for other languages written in roman. wang et al. use confusion-set-guided pointer networks @xcite , a novel seq2seq model for the task of chinese spell correction (csc). researchers have recently tried using large pretrained transformer language models to capture semantic understanding of texts accurately and achieve extremely competitive performance across various nlp tasks including spell correction. hong et al. @xcite have recently proposed bert @xcite based seq2seq model for the task of csc. however as pointed out by @xcite , vanilla bert is difficult to use for spelling correction, as it is primarily a pre-trained masked language model (mlm). the authors in @xcite have proposed a large pre-trained transformer model, bart, which is a denoising sequence-to-sequence language model. @xcite propose a bert based encoder-decoder model to correcting texts with edit operations. similar transformer models have been successfully used in grammatical error correction (gec) systems. for example, authors in @xcite employs a transformer encoder to design a simple and efficient gec sequence tagger called gector. in @xcite a pre-trained masked language model like bert is effectively incorporated into an encoder-decoder model in a gec system. people have also investigated the quality of output produced by gec systems, -researchers proposes a neural approach @xcite to automatically estimate the quality of gec-produced sentences which do not use any hand-crafted features. noises can be introduced into texts in two phases, pre-tokenization and post-tokenization. bart uses masking, deletion and infilling for introducing noises in pre-training. similarly, t5 @xcite is another pre-trained language model that uses fill-in-the-blank-style denoising objective for pre-training. however, in both bart and t5 the noises are added only after the tokenization of texts. so, denoising raw noisy texts is assumed to be more difficult than denoising noisy token sequence. further, none of the existing pre-trained language models are evaluated on noisy generation tasks with corrupted source texts. to overcome these limitations, in this work we introduce a robust transformer-based model that can detect and remove noises from texts injected during pre-tokenization and generate correct target texts flexibly. further, using different gating strategies, our model can understand the different kinds of induced and natural noises and act differently under various scenarios, which is not offered by bart, t5 or other pre-trained denoising language models. ## the proposed model in this section we formally describe our model gated-trans that consists of a transformer based gated-encoder and a gated-decoder. to showcase the efficacy of our proposed gating mechanism over the existing pre-trained language models, in this work we use the pre-trained bert, bart and t5 models as our backbone to initialize the encoder and decoder layers. in the subsequent discussion we assume each noisy source text is tokenized into m subword tokens represented as a sequence of vectors x 1:@xmath0, x 2 , . . . , x m }, where each x i , 1 ≤ @xmath1. given a sequence of tokens x 1:m the goal is to transform it into another sequence of subword tokens y 1:@xmath2, y 2 , . . . , y n }, whereas the length of target vectors n is unknown apriori and depends on the input sequence. the denoising process in bart and t5 is done after tokenization. so, an incorrect token (subword) is replaced with another token. on the other hand, in our gated-transformer, we use the raw, noisy text, tokenize them and feed them onto the sequence-to-sequence model. this task is assumed to be more challenging, as an incorrect word can be tokenized into multiple incorrect tokens (subword) after tokenization. hence, there may not be a one-to-one correspondence between the input and the output. also, as it is difficult to establish an one-to-one correspondence between source and target text, contrary to chinese spell correction task, we adhere to word-level and subwordlevel text denoising. in this work we adapt byte pair encoding (bpe) @xcite to convert both noisy input and the correct target into sequences of subwords. in the following subsections we describe each of the constituent modules of our model in greater detail (also see figure 1 for the architecture of the proposed model). ## experiments and results in this section we describe our experimental efforts on both intrinsic and extrinsic evaluations and subsequently report the results. ## result analysis in this section we analyze the performance of gated-trans both quantitatively and qualitatively. we perform a statistical test of significance to statistically validate the effectiveness of gating mechanism in transformer based seq2seq models. we conduct one-tailed welch's t-test (welch, 1947) on the intrinsic and extrinsic results of each pre-trained transformer model and the gated versions and reject the null hypothesis with a p-value of 0.01. this indicates that the improvement by our gated model is not random and is attributed by the architectural novelty. next we establish the relationships between gated-conditional probabilities and the volume of noise and overall length of the text. in figure 2a , we present the linear relationship between the wrr score and the gating probability values. one can notice that the masking probability a girl cricketer recovers having recovered from a head injury, caused by a cricket ball during an interstate women's match in adelaide on saturday, nesta moon, 17, wicketkeeperkeeper of the victorian team, was discharged from the adelaide hospital after detention for observation. target: girl cricketer recovers having recovered from a head injury, caused by a cricket ball during an interstate women's match in adelaide on saturday, nesta moon, 17, wicket-keeper of the victorian team, was discharged from the adelaide hospital after detention for observation. ## conclusion in this work, we propose gated-trans -a novel end-to-end seq2seq transformer model with conditional gates for robust generation. the gatedunit in our sequential model is efficient for effectively removing noises from texts which has been demonstrated by its superior performance over other competitive baselines. we also observe the performance improvement that gated-trans is achieved in downstream tasks like machine translation and summarization. further, we showcase the noise-invariant nature of gated-trans , which is capable of removing not only ocr induced noises but also synthetically infused noises, highlighting its potential efficacy towards dealing with (a) masking probability on noisy source (b) copy probability on a sample target figure 3: we highlight high masking and copy probability for different word tokens for a denoise instance a range of adversarial attacks on texts. further, our model allows more flexible text generation suitable for different purposes just by opting different gates out. looking forward, we would like to explore and handle noises which specifically occur in ocr and speech-to-text conversion processes by leveraging information from multiple modalities. yukun zhu, ryan kiros, rich zemel, ruslan salakhutdinov, raquel urtasun, antonio torralba, and sanja fidler. 2015. aligning books and movies: towards story-like visual explanations by watching movies and reading books. in proceedings of the ieee international conference on computer vision, pages 19-27.