Neural Multi-task Text Normalization and Sanitization with Pointer-Generator (2020)

Abstract: Text normalization and sanitization are intrinsic components of Natural Language Inferences. In Information Retrieval or Dialogue Generation, normalization of user queries or utterances enhances linguistic understanding by translating non-canonical text to its canonical form, on which many state-of-the-art language models are trained. On the other hand, text sanitization removes sensitive information to guarantee user privacy and anonymity. Existing approaches to normalization and sanitization mainly rely on hand-crafted heuristics and syntactic features of individual tokens while disregarding the linguistic context. Moreover, such context-unaware solutions cannot dynamically determine whether out-of-vocab tokens are misspelt or are entity names. In this work, we formulate text normalization and sanitization as a multi-task text generation approach and propose a neural hybrid pointer-generator network based on multi-head attention. Its generator effectively captures linguistic context during normalization and sanitization while its pointer dynamically preserves the entities that are generally missing in the vocabulary. Experiments show that our generation approach outperforms both token-based text normalization and sanitization, while the hybrid pointer-generator improves the generator-only baseline in terms of BLEU4 score, and classical attentional pointer networks in terms of pointing accuracy.

## introduction early natural language processing (nlp) faced the long-standing limitation of human language understanding, mainly due to linguistic morphology or the wide variance of word forms. therefore, a crucial requirement to obtain outstanding performance for modern nlp systems is the availability of "standardized" textual data @xcite . standardizing or normalizing textual data reduces the domain complexity, hence improves the generalization of the learned model. however, there are challenges to automatic text normalization. natural language is by nature evolving, e.g. urban dictionary foot_0 is a crowdsourced online dictionary for slang words and phrases not typically found in a standard dictionary, but used in an informal setting such as text messages or social media posts. moreover, abbreviations and emojis allow humans to express rich and informative content with few characters, but troubles machine understanding. finally, humans are prone to spelling errors while writing or typing. due to the reasons mentioned above, developers have designed pre-processing techniques to normalise textual data, including spell correction, tokenisation, stemming, lemmatization and partof-speech tagging. during the years, multiple libraries have been proposed to facilitate such preprocessing steps: e.g. nltk @xcite , spacy foot_1 or stanford core nlp @xcite . however, as textual domains vary greatly from medical records, legal documents to social media posts, there is no single solution or a fixed set of preprocessing steps for text normalization. thus, up to date, defining a pre-processing pipeline remains an art form which requires a significant engineering effort. while researchers can define hard-policies to eliminate all noisy textual data, they also considerably reduce the amount of information available to the model, thus limit its performance. such pruning approach appears problematic in the industry where engineers tackle domain-specific problems are given a relatively limited noisy textual dataset. enterprises also have to comply with multiple policies concerning privacy. thus, they are re- ## related work the introduction of word embeddings @xcite @xcite has produced a gigantic leap forward for most nlp-related task. traditional problems such as vector sparsity and word interaction were solved by a simple, yet effective, methodology that exploits a large corpus rather than a sophisticated algorithm. however, such methods are limited by the challenge of inferring embeddings for words unobserved at training time, i.e. out-of-vocabulary (oov). such scenarios are common in many socialmedia related applications where the input text is generated in real-time. thus, the user's malformed language might affect downstream performance @xcite . another solution is to include all the misspelling words in the training dataset or to impose similar embeddings for all n-character variations of a canonical word. this, would not scale well due to the sheer amount of such non-canonical terms; thus researchers have studied the spelling correction problem since long time @xcite . however, traditional approaches are based on a word-per-word basis; which has shown acceptable results when applied to formal languages. there have been many robust approaches to token-level spelling correction and lemmatization. the pioneering work done by han and baldwin demonstrated that micro-phonetic similarity could provide valuable insight to correct the spelling in an informal context, as many of these relaxed spellings are often based on the word's phonetic, e.g thr for there or d for the. monoise (van der goot, 2019a) generates feature-engineered n-character candidates for a misspelt word not found in the vocabulary and ranks them using a random forest classifier. however, to accurately identify misspelt words, let alone normalizing them, optimal approaches need to consider the whole contextual semantics rather than the word-level morphology. for example, the utterance can i speak to a reel person? is not misspelt at word-level as every word is a valid english word. however, if we consider sentence-level semantics, reel should be normal-ized into real. to factor in such contextual signals, recent advancements in nlp has considered these sequential nature of a written language as well as the long-term dependencies present in sentences. thus, the research community has proposed different methodologies to perform micro-text normalisation based on deep learning @xcite @xcite . while we address the problem of text normalisation in the nlp context, it has also been adopted as a key component for speech applications @xcite . pointer network was first proposed to solve geometric problems where the size of the output classes is a variable not conforming to the fixed multi-label classification of traditional seq2seq learning @xcite . pointer network becomes widely adopted in many nlp tasks including machine translation @xcite , abstractive summarization @xcite and language modeling @xcite as it aids accurate reproduction of factual details such as unseen proper nouns commonly treated as oovs. however, existing works formulate the pointing operation as a single position classification task that returns one word (token) position in the encoding sequence to be copied to the decoding sequence. such formulation is no longer suitable for our char-to-word strategy. furthermore, with the recent state-of-the-art in seq2seq learning introduced by the transformer architecture, there has not been a comprehensive comparison between different attention strategies, i.e. the classical attention mechanisms @xcite and multihead attention @xcite on this pointing objective. finally, none of the previous research considered the joint privacy-preserving issue, which is common in commercial nlis such as virtual agents for customer services. to the best of our knowledge, @xcite is the first model that attempted to solve the sanitization problem at a semantic level, without using a rule-based approach @xcite . however, the former approaches are based on manually defined policies that are application and context-specific or are limited to named entities; thus are not generalizable across domains and applications. ## experiments we conducted 3 experiments to verify the effectiveness of our proposed model. 1) for improved joint normalization and sanitization, we compare our context-aware model with: 1.1) a traditional tokenlevel lemmatizer and spelling corrector, and 1.2) a lstm w2w encoder-decoder model. 2) for improved normalization of proper nouns, we compare our multi-head attentional pointer-generator with 2.1) a generator-only and a pointer-only baseline, and 2.2) the traditional attention encoder-decoder model. 3) finally, to address the utility of text normalization we evaluate the performance's improvement obtained on a text classification task with or without text normalization. the seq2seq transformer architecture we used has 4 attention heads and 5 layers with 100 hidden units. the maximum number input characters and output words are 600 and 300 respectively. during evaluation we maintain a beam size of 3. we determine the correct positions for the pointer network by matching any output word to its character list if the characters appear consecutively in the input character sequence, and noting the start and end position of that character list. words whose characters are not found consecutively are assign a start and end position of 0 (the beginning of the sequence). we fix the start and end position to the nearest left and right space respectively in the input character sequence to select a complete word. we use the pointer output instead of the generator output whenever the predicted probability for generation is less than 0.6. ## conclusion we addressed the importance of context awareness in joint normalization and sanitization. we verified our c2w transformer-based model's quality over context-unaware word-level lemmatizer and traditional w2w seq-to-seq model at both macroscopic and microscopic level. moreover, we tackled the limitation of representing and producing oovs during generation with a pointer-generator that learns to copy the relevant text segments from the source input to the translated output. experiments at both macroscopic and microscopic level verified improved normalization and sanitization fluency previously limited by oovs. our formulation of text normalization as a learning-to-translate problem avoids the tedious engineering of domain specific preprocessing heuristics for textual data. the proposal of pointergenerator is highly generalizable to other nlp tasks such as summarization or machine translation.