TNT : Text Normalization based Pre-training of Transformers for Content Moderation (2020)

Abstract: In this work, we present a new language pre-training model TNT (Text Normalization based pre-training of Transformers) for content moderation. Inspired by the masking strategy and text normalization, TNT is developed to learn language representation by training transformers to reconstruct text from four operation types typically seen in text manipulation: substitution, transposition, deletion, and insertion. Furthermore, the normalization involves the prediction of both operation types and token labels, enabling TNT to learn from more challenging tasks than the standard task of masked word recovery. As a result, the experiments demonstrate that TNT outperforms strong baselines on the hate speech classification task. Additional text normalization experiments and case studies show that TNT is a new potential approach to misspelling correction.

## introduction language model pre-training (self-supervised or unsupervised learning) has been a popular thread in natural language understanding (nlp) studies recently due to its universal representation capacity @xcite @xcite . it has been thus widely used in a multitude of language processing tasks such as named entity recognition, sentiment analysis, question answering and content moderation @xcite . in addition, the masking pre-training paradigm introduced by bert (bidirectional encoder representations from transformers) @xcite has been employed for other tasks such as image processing @xcite , optical flow @xcite , and audio-visual co-segmentation @xcite . recently, many variants have been proposed to further improve the pre-training procedure @xcite @xcite . they have also advanced the state-of-the-art performance on multiple downstream natural language understanding tasks (leaderboard) consistently. almost all these studies train language models by predicting the masked words in different manners. the underlying mechanism is cloze task @xcite . the pre-training model itself, however, has not been fully exploited to address complicated yet feasible tasks. it is reasonable to expect that models can learn a better universal language representation if the pre-training procedure can be aligned with more challenging tasks. in this article, we attempt to improve the language representation by proposing tnt: text normalization based pre-training of transformers. tnt enhances the language learning by utilizing text normalization pre-training objective, inspired by misspelling correction. specifically, tnt randomly manipulates tokens from the input text. the objective is then to reconstruct the original tokens of the manipulated words based on the context by predicting both recovery operation type and original token labels as illustrated in fig. 1 . unlike the masked language model, tnt has to offer two predictions to reconstruct the original text. in addition, tnt does not have to be given the prediction positions in advance. this aligns with the fact that misspelling correction needs to perform the position-agnostic prediction for both aspects. perpetrators often intentionally obfuscate certain words about groups, or abusive words, by misspelling, or leetspeak (e.g.,"f@ggot","ph*ck", "w.e.t.b.a.c.k.") @xcite . this could sidestep the content moderation algorithms easily as exemplified in our contributions are summarized as follows: (1) we introduce text normalization into the language training paradigm, which involves challenging tasks of predicting both operation types and token labels; (2) we show that tnt advances the challenging downstream text classification task, which benefits the content moderation; (3) tnt offers a new perspective on misspelling correction. for online abusive language moderation, bert has also been shown effective and advances the overall performance largely @xcite . in addition, early works formulate the hate speech detection as the generic text classification, alternatively focus on certain ethnic groups or building up blacklists of swear words @xcite . misspelling correction is also a long-standing problem in nlp @xcite @xcite and has been widely used in real-word scenarios like word processing system and email spell checking. she is a nice lady. c(4) ## experiments we perform the hate speech classification and misspelling correction tasks based on the pre-trained model. ## misspelling correction misspelling correction is a long-standing research topic @xcite @xcite and has been widely commercialized as a service such as bing spell check (bing) and grammarly @xcite . tnt can be readily employed in misspelling correction. we here evaluate tnt using its character-level 3 variant without additional fine-tuning. we aggressively misspell the test set of yahoo in ## discussions and future work we conducted experiments on three classification tasks. the data size for yahoo finance and news, while being one of the largest in the context of hatespeech classification, is nevertheless small in the context of language modeling. we plan to perform large-scale pre-training and evaluation on glue datasets for the comprehensive analysis. this work targets sentence level language understanding. as far as we know, no data available for misspelled words in the context of sentences, we thus have to generate the evaluation set by ourselves. the main goal here is not to develop a more powerful misspelling corrector, but rather to propose a new and stronger language modeling approach. we thus don't set up the strict and comprehensive evaluation for apples-to-apples comparison on spelling correction. we will continue to explore this line in the future. ## conclusion in this work, we propose a new language representation training strategy tnt. tnt improves language modeling by training a transformer to reconstruct text from four operation types typically seen in text manipulation. we show that when finetuned for the content moderation task of detecting hatespeech, the new model performed better than the state of the art baselines. we also demonstrate its effectiveness in misspelling correction.