Towards a Better Understanding of Noise in Natural Language Processing (2021)

Abstract: In this paper, we propose a definition and taxonomy of various types of non-standard textual content – generally referred to as “noise” – in Natural Language Processing (NLP). While data pre-processing is undoubtedly important in NLP, especially when dealing with user-generated content, a broader understanding of different sources of noise and how to deal with them is an aspect that has been largely neglected. We provide a comprehensive list of potential sources of noise, categorise and describe them, and show the impact of a subset of standard pre-processing strategies on different tasks. Our main goal is to raise awareness of non-standard content – which should not always be considered as “noise” – and of the need for careful, task-dependent pre-processing. This is an alternative to blanket, all-encompassing solutions generally applied by researchers through “standard” pre-processing pipelines. The intention is for this categorisation to serve as a point of reference to support NLP researchers in devising strategies to clean, normalise or embrace non-standard content.

## introduction in natural language processing (nlp), "noise" as a concept is not well understood and often described as hard to define @xcite . in this paper, we attempt to outline what different types of noise mean in order to support nlp researchers in devising strategies to clean, normalise or embrace unexpected content at either training or inference time. the term "noise" has been generally used to cover both harmful and meaningful types of nonstandard content. we however believe that a distinction should be made between content that should * equal contribution be removed/normalised because it harms nlp systems -henceforth harmful noise -and content that should be kept to improve the performance of such systems -henceforth useful noise foot_0 . to clarify further, the usual approach to handling "noise" is to attempt to clean or normalise the data as much as possible, but this content can be as important as any other elements in the data because it carries meaning or intentions. in text classification applications, for example, some types of noise should be kept and taken into account, e.g. certain punctuation patterns in sentiment analysis. furthermore, in generation applications, noise often needs to be transferred to the output, such as emojis in machine translation to preserve sentiment. moreover, in tasks such as correction of second language learners' essays, errors such as lack of cohesion and incorrect punctuation need to be retained. handling noise in nlp has been attracting significant attention especially with the widespread availability of data from social media platforms such as twitter, reddit, among others, that created not only new opportunities but also new and different needs @xcite . text found on these platforms, as well as other user-generated materials, is full of non-standard content @xcite which causes problems to nlp systemstypically trained on clean data -and these fail to handle them correctly @xcite @xcite . dealing with non-standard data has been targeted as a research direction in some areas, for example, machine translation (mt), with methods that aim to be robust to unexpected noise @xcite @xcite . however, work has been mainly limited to a few sources of artificially injected noise types. in this paper, we start by providing an overview of previous work addressing this issue (section 2); to then propose a definition of noise including cases that needs to be cleaned/normalised (harmful noise) and other cases that needs to be kept (useful noise), and present a comprehensive taxonomy of types of noise (section 3); and to finally demonstrate the impact of addressing noise in a task-based fashion through experimenting with three tasks, where we compare filtering out harmful noise and keeping useful noise versus cleaning/normalising or keeping all types of noise (section 4). our focus is on linguistic noise rather than other types of noise (such as numeric mislabelling). therefore, our definitions and categorisation broadly cover (i) text classification/regression tasks, where the noise is in the input text, e.g. sentiment analysis; and (ii) text-to-text tasks, where the noise is in the input text and -at training time -can also be in the output text, e.g. machine translation. ## related work noise in nlp has typically been defined in context of a specific dataset, intermediate or end-task, rather than in general. for example, noise is limited to sentence pairs "not being parallel" in parallel corpus filtering @xcite . @xcite describe their task as lexical normalisation of "ill-formed words", where the definition is narrowed down to "instances of typos, ad hoc abbreviations, unconventional spellings, phonetic substitutions and other causes of lexical deviation found on twitter and sms messages". @xcite define text normalisation where noise becomes "any kind of difference in the surface form of an electronic text from the intended, correct or original text". for an end task where the focus is on improving the performance of a certain task, however, the definition of noise becomes very specific, serving one purpose. for example, noise is referred to as the task of removing unconventional casing by truecasing the data when the aim is to get better vocabulary generalisation. very often such a blanket, one-size-fits-all pre-processing pipeline is applied without enough consideration to other possible sources of noise, or which sources should actually be kept in the data. our work is driven by the lack of studies aimed at defining and providing a comprehensive categorisation of noise types in a way that reflects a better practice and shows sensitivity towards such content. the few exceptions limit their categories to noise found in specific text types. @xcite provide a general overview of noise types in sms, emails and online chats, while eisenstein (2013) focuses on identifying types of what is called bad language and their possible causes. most of the previous studies on handling noise have focused on mt. while some studies have revealed that training with noise increases the robustness of systems towards noisy data, others have indicated that the quality of their systems degraded. this can be attributed to the differences in noise types and their potential impact on the task. for instance, @xcite have showed that training mt models on noisy parallel data (such as misaligned sentences or wrong language) leads to weaker systems. however, when models are trained on more specific types of noise such as spelling, profanity, grammar and emoticons, they have been shown to perform better on similar types of noise @xcite @xcite @xcite . we note that most of the latter work aims to adapt mt systems built on clean data to noisy test settings by artificially injecting errors to the training set. this is a different problem to the one we discuss in this paper where both training and test sets are taken from the same larger population with naturally occurring noise. these studies show that different types of noise have different effects in mt and other nlp tasks. however, the types of noise addressed and their effect vary considerably and a consistent definition has not been provided. noise is better defined and understood in other related fields. in automatic speech recognition, a distinction is made between what is considered as noise (e.g. environmental noise, reverberation) and other speech variations (e.g. accent, speaking style and rate, emotional states, speech impairments). noise is defined as "any unwanted disturbances superposed upon the intended speech signal" @xcite , e.g. by additive noise (e.g. background noise, traffic noise) and convolutional noise (e.g. transmission channel distortions, room reverberation, microphone filtering) @xcite . in optical character recognition, noise refers to "the error in the pixel value or an unwanted bit pattern, which do not have any significance in the output", where the unwanted bit patterns are introduced by uneven writing surface or poor quality of the data acquisition device and include textured or coloured background @xcite . different handwriting however is not seen as noise which needs removing but as a problem which needs to be addressed. similar to what has been done in these areas, this paper is an attempt to clearly define and categorise noise in nlp in a way that is not constrained to any particular application, task or dataset. while noise is mainly found in non-standard text, the taxonomy is agnostic to the type of text. ## a taxonomy of noise we intend to move away from the general labelling of noise as "errors", "difficult data", "ill-formed words", "contamination", "disfluent language" and "corruption" to a more specific one that reflects the current practice which have shifted from denoising to training with noisy text. we therefore look at noise through the same lens as in other related fields where a distinction should be made between unwanted constructs that occur in the text (generally unintentionally), and other (generally intentional) constructs that do not adhere to the conventions and rules of a given language to serve a purpose. based on this distinction, we define harmful noise as any unwanted disturbances that are either harmful or useless or both: harmful in that it affects the intended meaning of the text and/or the performance of the nlp system; and useless in that their existence does not serve a purpose. useful noise, on the contrary, covers wanted content that serves a purpose and carries meaning that is important for the task and/or for the performance of the nlp system. our definition does not includes cases where a special language is used in a cryptical fashion, i.e. hiding some codes/messages in the text so that only a particular addressee understands the true message. it is also worth noting that it depends on the nlp application under consideration, where harmful noise for one system could be useful noise for another one. our taxonomy of noise, displayed in figure 1 , focuses on applications that take sentences as input, as in most nlp tasks. from this work, we exclude types of noise that are related to errors in parallel corpora with text in both input and output such as misalignments, incorrect output, untranslated sentences, among others. the taxonomy is divided into several types and sub-types of noise that are limited to those naturally occurring, pro-duced by humans. machine-generated errors, e.g. stemming from back-translation or intentionally injected, are seen as a way to mimic the naturally occurring noise, and as such not represented separately. the taxonomy is designed to serve as an initial point of reference. it should be seen as language-independent, but it may require modifications to cover very specific language phenomena the authors are not familiar with. our taxonomy is based on previous works that address different types of noise as well as taxonomies of errors and also on our wide experience improving robustness in nlp. the selection criteria are driven by the intention to include general categories that are applicable to different languages and tasks, and for non-standard content. in what follows, we describe the types and sub-types of noise coming under the proposed taxonomy and provide examples. orthography: this type of noise is concerned with the way words are written. several sub-types come under this type, which we describe below. some of them are considered as errors, e.g. spelling errors, while others are looked at as variations or deviations from the standard way of writing to serve a purpose, e.g. word obfuscation, word lengthening. spelling errors: this is when a word is spelt in a way that deviates from reference dictionaries, standardised or accepted norms, or recognised usage. the misspelling of a word takes different forms. for example, the word "receive" can be misspelt by deleting a character, e.g. "receve"; inserting an extra character, e.g. "recceive"; swapping adjacent characters, e.g. "recieve"; or replacing a character with another, e.g. "reciece" @xcite . additionally, a spelling error can occur when writing a word without a hyphen where needed or with a space where it should be written as one word such as writing "4 mb" instead of "4mb" @xcite . orthographic variants: this covers the spelling of words in different ways due to: 1) regional variations: e.g. british english spelling vs. american english spelling, e.g. "centre", "center"; 2) words with different correct spellings: e.g. "spelled", "spelt", " " or " " for "syria" in arabic, or when words are transliterated, e.g. proper name " ", having one spelling in arabic, could be written as "muhamed", "mohamed" or "mohammed" in english; or 3) diacritical marks: repetition of punctuation code-switching urls, links and markup internet jargon disfluencies in human (transcribed) data pause-filling words repetition of words grammatical errors function words word form word order incorrect punctuation cohesion errors figure 1: types of noise in nlp in some languages, such as arabic, words have accents (or what is called diacritics), which are not always used. ## experimental setup this section describes the settings of the experiments we carried out so as to show the impact of different pre-proccessing strategies on different nlp tasks. we experiment on three tasks based on data (in english) collected from twitter: offensive language identification, informative covid-19 tweets identification, and tweets sentiment analysis tasks. we start by describing each task and the data used (section 4.1), followed by the preprocessing steps of different sources of noise (section 4.2), and the model architectures (section 4.3). due to length restrictions, we limit ourselves to experimenting with a few types of noise, with the aim to show their role as of being "harmful noise" to filter out or "useful noise" to keep, and to what extent this is dependent on the task. ## results we first present our baseline performance without any pre-processing in it is worth noting that if the performance drops, that means removing the noise might deviate the intended meaning of the original sentence. hashtags are more useful on the sentiment analysis task because, when removed, the performance drops on the sa task while improves on the other two tasks. we notice that in the data for sentiment analysis, hashtags are mostly single words such as "#mad" and "#senseless" so that the sentiment could be detected by the model. however, in the olid dataset, hashtags hinting toxicity mostly include multiple words, e.g. "#liberalismisamentaldisorder", which increases the difficulty of utilising these hashtags. in the covid dataset, almost all tweets consist of hashtags related to covid-19, thus the hashtags do not help identifying whether the tweet is informative or not. removing @mentions causes a more obvious decrease in the sa task than the other two tasks. this is because the @mentions in olid and covid datasets have been normalised with a unified "@user" token, but in the sa data @mentions stay in the form of usernames. we found that the @mentions in sentiment analysis data helps indicate the sentiment. for example, 32.0% of sentences with "@virginamerica" are labeled as positive while there are only 10.9% positive tweets with "@united". similarly, removing emojis, codeswitching and punctuation leads to a decrease on all three tasks. however, emojis are less influential for identifying informative tweets. the decrease therefore is not significant for this specific task. furthermore, as the non-english words are mostly named entities, the removal of codeswitching could break the sentence structure. regarding punctuation, it is important for the three tasks where its removal causes larger performance drop especially on the sa task as it leads to the removal of emoticons ":-)", which can be useful for classifying sentiment. based on our findings that show how different strategies lead to different results, we took a step further to show the validity of our reasoning for how noise should be understood and handled on different nlp tasks. to that end, we combined the different pre-processing strategies and trained two other systems: remove all, which does the lowercasing and removes all other types of "noise" we dealt with in our experiment (i.e. urls, hashtags, @mentions, emojis, code-switching and punctuation), and remove+keep, which only removes the harmful noise in the specific task as showed in figure 2 for each type of "noise" (e.g. for olid task only hashtag is removed while for sa task, url is removed and the texts are lowercased). in addition, to make use of the potentially useful information in hashtags and emojis, we followed the state-ofthe-art approaches @xcite and segmented hashtags into separate words and transformed emojis into corresponding english phrases as we stated in section 4.2 (preprocessing of other types of noise is the same as remove+keep). the system trained on this data is noted as remove+keep+transform. the results are presented in figure 3 . the system with data removing all sources of noise in the pre-processing shows a poorer performance than the baseline, which keeps all sources of noise. however, both "remove+keep" and "re-move+keep+transform" systems outperform the baseline, with improvement on sentiment analysis task being the most significant. however, after transforming hashtags and emojis into english phrases, the performance only improves on the olid task compared to the "remove+keep" system, which confirms our claim that the same noise normalisation strategy to noise might have different olid covid sa -3.0% -2.0% -1.0% 0.0% 1.0% 2.0% 3.0% 4.0% percentage change in macro-f1 removeall removekeep removekeeptransform influence on different tasks. these results are in line with our proposed definition of noise in nlp where some types of noise can be useful and others harmful, and how this is greatly dependent on the task. they also confirm our suggestion that one should not simply follow "standard" pre-processing pipelines, but carefully devise appropriate strategies to deal with different types of noise depending on the task. ## conclusions in this paper, we proposed a definition and taxonomy of noise in nlp so as to serve as a point of reference for nlp researchers to consult when they devise strategies to clean, normalise, or embrace non-standard content at either training or inference time to improve the robustness of their systems to this unseen or unexpected naturally occurring harmful and useful noise. we highlighted that noise in nlp should be carefully handled in light of what we call "harmful noise" that needs to be removed when it affects the performance of the system and/or it does not carry the intended meaning of the text, and "useful noise" that needs to be kept because it is an integral part of the data and useful for a task, or even should be added to the training data when it only happens naturally at test time. our experiments support our argument by demonstrating that tailored approaches are better than blanket, all-encompassing solutions generally applied by researchers through "standard" preprocessing pipelines. for instance, we found out that casing and urls are useful noise and should be kept on olid task, but having a negative impact on the other two tasks (sa and covid tasks) and should therefore be removed. we have also shown how special handling of harmful and useful noise could result in better performance where removeall and keep-all approaches resulted in poorer performance. our approach to noise was based on their impact on the tasks -we removed types of noise which had negative influence on the tasks. our goals were to bring awareness to the different types of unexpected content and provide a definition and a taxonomy, and to highlight the fact that they need to be handled carefully rather than being avoided or treated using the same strategies.