Sequence-to-Sequence Lexical Normalization with Multilingual Transformers (2021)

Abstract: Current benchmark tasks for natural language processing contain text that is qualitatively different from the text used in informal day to day digital communication. This discrepancy has led to severe performance degradation of state-of-the-art NLP models when fine-tuned on real-world data. One way to resolve this issue is through lexical normalization, which is the process of transforming non-standard text, usually from social media, into a more standardized form. In this work, we propose a sentence-level sequence-to-sequence model based on mBART, which frames the problem as a machine translation problem. As the noisy text is a pervasive problem across languages, not just English, we leverage the multi-lingual pre-training of mBART to fine-tune it to our data. While current approaches mainly operate at the word or subword level, we argue that this approach is straightforward from a technical standpoint and builds upon existing pre-trained transformer networks. Our results show that while word-level, intrinsic, performance evaluation is behind other methods, our model improves performance on extrinsic, downstream tasks through normalization compared to models operating on raw, unprocessed, social media text.

## introduction social media is a pervasive part of our modern lives and provides us with a rich source of information and insight into human behaviour. usergenerated content has been a valuable resource for the research community, especially in the form of text, but it is notoriously noisy and non-standard. models that operate on social media posts go beyond marketing and advertisement applications, and have the potential to impact real human lives through, for instance, detecting loneliness @xcite , stress @xcite , life satisfaction @xcite , suicidal ideation @xcite , and mental health problems such as depression @xcite @xcite and ptsd @xcite . outside of a formal setting, users communicate freely in text form, resorting to abbreviations, slang or plain spelling mistakes or typos. @xcite further explored bad language on social media, in the sense of language that defies our expectation of good spelling, vocabulary and syntax. he identified several underlying factors for the cause of non-standard text: user illiteracy, length limits imposed by social media sites (i.e. twitter), text input affordances (i.e. standard mobile keyboards or predictive entry), pragmatics (emoticons/emoji, abbreviations and expressive lengthening), and a social component. @xcite further explored the latter, concluding that some types of non-standard text have strong social meaning, and normalization could induce a loss of meaning. however, it is well known that for most benchmark tasks, noisy/non-standard text has proven to be a real problem to nlp models, such as bert @xcite , trained on clean or curated data, but fine-tuned on tasks with noisy and inconsistent format. to overcome this predicament, eisenstein (2013) proposes two possible approaches: either domain adaptation or normalization. while domain adaptation is not specific to natural language processing, text normalization and cleaning have always been a central part of any modern text processing pipeline. text normalization is the process of adapting an input text to a more standard form. it has proven to be effective in increasing performance on tasks such as pos tagging (van der goot and çetinoglu, 2021), dependency parsing (van der goot, 2019a) and sentiment analysis @xcite . naturally, most text normalization pipelines are based on supervised models, which require carefully annotated data. however, annotating a large corpus of text in multiple languages is often cumbersome and expensive, and some approaches rely on synthetically generating corrupted text @xcite . commonly, approaches are based on word-level normalization. one of the most prominent methods is monoise @xcite , in which the text correction pipeline is similar to a classic ranked retrieval. however, monoise operates at the individual word and uses a spelling correction module and a word embedding module. while word embeddings can be made to account for a specific sentence context, it is mostly discarded. different from current methods, we aim to perform text normalization at a sentence level. this approach has several advantages, compared to word or subword methods: i) it can be naturally framed as a sequence-to-sequence type problem, ii) it is more straightforward, as it requires only one module, as opposed to a multi-stage pipeline (i.e. complex candidate generation and ranking), and iii) the same model can be trained on multiple languages at the same time, without increasing in size and computational processing. in this edition of the workshop on noisy usergenerated text (w-nut), organizers propose the shared task of multilingual lexical normalization foot_0 , in which participants are required to perform lexical normalization on 12 different languages @xcite . as such, we use the state-of-the-art multilingual sequence-to-sequence transformer model mbart @xcite and fine-tune it for our task. mbart is one of the first models that can be fine-tuned simultaneously on multiple languages without performance loss. we show that framing text normalization as a neural machine translation problem is a viable method for text normalization, improving performance on extrinsic, downstream tasks compared to models that operate on raw, unprocessed social media text. we made the code publicly available on github. foot_1 ## related work the best model, from jin (2015), generated candidates from the most similar canonical forms from the training data evaluated with the jaccard index. a random forest classifier was used to predict the suitable canonical form from all the candidates using features such as support and confidence, string similarity, and part of speech tags. the model was a constrained system, suggesting that the quality of the proposed model is more important than using additional data and tools. other approaches were based on conditional random fields (crf) @xcite @xcite and recurrent neural networks (rnn) @xcite among others. notably, monoise (van der goot and van noord, 2017) has long been considered state-of-the-art in lexical normalization. monoise is a normalization model using spelling correction and word embeddings for candidate generation and a feature-based random forest classifier for candidate ranking. it is a modular normalization system easily reusable and adaptable (van der goot and van noord, 2017). the model was at the beginning developed only for english text. still, then it was later expanded for multi-lingual lexical normalization covering languages such as dutch, spanish, turkish, slovenian, croatian and serbian (van der goot, 2019b). the lexical normalization task can also be formulated as a machine translation (mt) task. the noisy user-generated content is the source language, and the canonical form is the target language. @xcite compare the mt approaches for lexical normalization, focusing on statistical neural translation (smt) and neural machine translation (nmt) and obtaining better results using the smt method. furthermore, the authors show that the smt approach works better in a low-resource setting than an nmt approach which requires a lot of data. with the rise in popularity of pre-trained language models for natural language understanding and natural language generation, their ability to perform lexical normalization was also studied. by transforming the task into a token prediction one, @xcite demonstrate that a bert model can be used as a lexical normalization model in low resource settings. current methods for lexical normalization attempt to normalize at the character-level @xcite , syllable-level @xcite @xcite or sentence-level @xcite . @xcite propose an encoder-decoder approach for text normalization. we propose to make use of the latest transformer models that are capable of multilingual translation in a sequence to sequence manner, namely mbart @xcite . however, we do not perform translation between languages, but instead, we use mbart as a denoising autoencoder, i.e. translating from bad english to good english. this way, we take the whole sentence into consideration when correcting the text. moreover, this method is more straightforward and can scale to multiple languages without increasing computational demands. ## data and evaluation we further describe the dataset for this task and evaluation procedures. multilexnorm dataset the data provided by the organizers includes texts from 12 languages: croatian, danish, dutch, english, german, italian, serbian, slovenian, spanish, turkish and codeswitched data for indonesian-english and turkish-german, as seen in ## results we further showcase the results of the pretrained mbart models fine-tuned on the available data: firstly, we kept the transformer encoder fixed and trained only the decoder, and secondly, we kept the decoder fixed and trained the encoder. during this fine-tuning process, we trained a single model for all languages. further, for the codalab submission, we fine-tuned multiple models, one for each language, in the "fixed encoder" regime. intrinsic evaluation however, since our method is sentence-based, perfect alignment between words is cumbersome, with many cases of mismatch between punctuation. moreover, merging or splitting words for normalization is also not taken into account in the post-processing phase. extrinsic evaluation for extrinsic evaluation, we showcase the results for our best model in moreover, we also evaluated the effect of lexical normalization on two other tasks -sentiment analysis on the smile dataset and offensive language identification on olid ( however, as the organizers have pointed out, there are inconsistencies in the training and testing data annotations. in some cases, some words are not normalized (i.e. jajajajaj / w/ / n) even though they were clearly lengthenings or contractions. despite this, in some of these cases, our model was able to provide correct normalizations. there also appears to be no correlation between training dataset size and final normalization performance. for example, in the case of croatian, even though the dataset is the second largest, the performance is lower than for other languages. thus, the lower performance in some languages may be a cause of the complexity of the language; for english, our model obtained the best results. ## conclusions in this work, we presented a method to perform lexical normalization by fine-tuning a multilingual machine translation model on pairs of noisy and normalized sentences from various languages. we employed mbart, as it is currently the state-ofthe-art in transformer-based multilingual machine translation, allowing us to fine-tune on all available languages simultaneously. furthermore, we used mbart as a denoising autoencoder and tuned it in a supervised fashion. as opposed to current two-stage methods for word candidate generation and ranking, our approach is more straightforward. moreover, it scales to multiple languages without increasing computational demand (i.e. not increasing vocabulary size, increasing search space and others). evaluation results show that our method, even though it lacks behind current methods on intrinsic, word-level evaluation, improves performance on downstream tasks. for future work, we aim to develop our method for better post-processing of the output and increasing augmentation levels -i.e. injecting more noise in the form of spelling mistakes, backwards translations etc. moreover, since our method is supervised, the quality and quantity of training data play an essential role in the final performance. in this regard, we aim to explore ways to take into account inconsistent annotations.