Mind Your Inflections! I mproving NLP for Non-Standard E nglishes with B ase- I nflection E ncoding (2020)

Abstract: Inflectional variation is a common feature of World Englishes such as Colloquial Singapore English and African American Vernacular English. Although comprehension by human readers is usually unimpaired by non-standard inflections, current NLP systems are not yet robust. We propose Base-Inflection Encoding (BITE), a method to tokenize English text by reducing inflected words to their base forms before reinjecting the grammatical information as special symbols. Fine-tuning pretrained NLP models for downstream tasks using our encoding defends against inflectional adversaries while maintaining performance on clean data. Models using BITE generalize better to dialects with non-standard inflections without explicit training and translation models converge faster when trained with BITE. Finally, we show that our encoding improves the vocabulary efficiency of popular data-driven subword tokenizers. Since there has been no prior work on quantitatively evaluating vocabulary efficiency, we propose metrics to do so.

## introduction large-scale neural models have proven successful at a wide range of natural language processing (nlp) tasks but are susceptible to amplifying discrimination against minority linguistic communities @xcite due to selection bias in the training data and model overamplification @xcite . most datasets implicitly assume a distribution of error-free standard english speakers, but this does not accurately reflect the majority of the global english speaking population who are either second language (l2) or non-standard dialect speakers @xcite . these world englishes differ at lexical, morphological, and syntactic levels @xcite ; sensitivity to these variations predisposes english nlp systems to discriminate against speakers of world englishes by either misunderstanding or misinterpreting them @xcite . left unchecked, these biases could inadvertently propagate to future models via metrics built around pretrained models, such as bertscore @xcite . in particular, @xcite show that current question answering and machine translation systems are overly sensitive to non-standard inflections-a common feature of dialects such as colloquial singapore english (cse) and african american vernacular english (aave). 2 since people naturally correct for or ignore non-standard inflection use @xcite , we should expect nlp systems to be equally robust. existing work on adversarial robustness for nlp primarily focuses on adversarial training methods @xcite @xcite or classifying and correcting adversarial examples @xcite . however, this effectively increases the size of the training dataset by including adversarial examples or training a new model to identify and correct perturbations, thereby significantly increasing the overall computational cost of creating robust models. these approaches also only operate on either raw text or the model, ignoring tokenization-an operation that transforms raw text into a form that the neural network can learn from. we introduce a new representation for word tokens that separates base from inflection. this improves both model robustness and vocabulary efficiency by explicitly inducing linguistic structure in the input to the nlp system @xcite . many extant nlp systems use a combination of a whitespace and punctuation tokenizer followed by a data-driven subword tokenizer such as byte pair encoding (bpe; @xcite ). however, a purely data-driven approach may fail to find the optimal encoding, both in terms of vocabulary efficiency and cross-dialectal generalization. this could make the neural model more vulnerable to inflectional perturbations. hence, we: to make subword regularization more tractable, @xcite proposed an alternative method of building a subword vocabulary by reducing an initially oversized vocabulary down to the required size with the aid of a unigram language model, as opposed to incrementally building a vocabulary as in wordpiece and bpe variants. however, machine translation systems operating on subwords still have trouble translating rare words from highlyinflected categories @xcite . @xcite , @xcite , and @xcite propose to improve machine translation and morphological reinflection by encoding morphological features separately while @xcite propose a schema for inflectional features. @xcite explore the effect of learning word embeddings from base forms and morphological tags for hebrew, while @xcite show that representing words as base forms, phonemes, and morphological tags improve crosslingual transfer for low-resource languages. adversarial robustness in nlp. to harden nlp systems against adversarial examples, existing work largely uses adversarial training @xcite @xcite @xcite @xcite . however, this generally involves retraining the model with the adversarial data, which is computationally expensive and time-consuming. @xcite showed that simply fine-tuning a trained model for a single epoch on appropriately generated adversarial training data is sufficient to harden the model against inflectional adversaries. instead of adversarial training, @xcite train word embeddings to be robust to misspellings, while @xcite propose using a bert-based model to detect adversaries and recover clean examples. jia et al. (2019) and huang et al. (2019) use interval bound propagation to train provably robust pre-transformer models, while @xcite propose an efficient algorithm for training certifiably robust transformer architectures. summary. popular subword tokenizers operate on surface forms in a purely data-driven manner. existing adversarial robustness methods for largescale transformers are computationally expensive, while provably robust methods have only been shown to work for pre-transformer architectures and small-scale transformers. our work uses linguistic information (inflectional morphology) in conjunction with data-driven subword encoding schemes to make large-scale nlp models robust to non-standard inflections and generalize better to l2 and world englishes, while preserving performance for standard english. we also show that our method helps existing subword tokenizers use their vocabulary more efficiently. ## linguistically-grounded tokenization data-driven subword tokenizers like bpe improve a model's ability to approximate the semantics of unknown words by splitting them into subwords. although the fully data-driven nature of such methods make them language-agnostic, this forces them to rely only on the statistics of the surface forms when transforming words into subwords since they do not exploit any language-specific morphological regularities. to illustrate, the past tense of go, take, and keep have the inflected forms went, took, and kept, respectively, which have little to no overlap with their base forms 4 and each other even though they share the same tense. these six surface forms would likely have no subwords in common in the vocabulary. consequently, the neural model would have the burden of learning both the relation between base forms and inflected forms and the relation between inflections for the same tense. additionally, since vocabularies are fixed before model training, such an encoding does not optimally use a limited vocabulary. even when inflections do not orthographically alter the base form and there is a significant over- when encoded in conjunction with another inflected form like entered, which should be encoded as @xcite , this encoding scheme also produces two different subwords for the same type of inflection -ed vs -d. as in the first example, the burden of learning that the two suffixes correspond to the same tense is transferred to the learning model. a possible solution is to instead encode danced as [danc, ed] and dancing as [danc, ing], but there is no guarantee that a data-driven encoding scheme will learn this pattern without some languagespecific linguistic supervision. in addition, this unnecessarily splits up the base form into two subwords danc and e; the latter contains no extra semantic or grammatical information yet increases the encoded sequence length. although individually minor, encoding many base words in this manner increases the computational cost for any encoder or decoder network. finally, although it is theoretically possible to force a data-driven tokenizer to segment inflected forms into morphologically logical subwords by limiting the vocabulary size, many inflected forms are represented as individual symbols at common vocabulary sizes (30-40k). we found that the bert base wordpiece tokenizer and bpe foot_4 encoded each of the above examples as single symbols. ## model-based experiments we first demonstrate the effectiveness of bite using the pretrained cased bert base @xcite before training a full transformer @xcite from scratch. we do not replace word-piece and bpe but instead incorporate them into the bite framework as described in §3.2. the advantages and disadvantages to this approach will be discussed in the next section. we do not do any hyperparameter tuning but use the original models' in all experiments (detailed in appendix b). ## model-independent analyses finally, we analyze wordpiece, bpe, and unigram lm subword tokenizers that are trained with and without bite. implementation details can be found in appendix b.4. through our experiments, we explore how bite improves adversarial robustness and helps the data-driven tokenizer use its vocabulary more efficiently. we use 1m examples from wikipedia+bookcorpus for training. ## limitations our bite implementation relies on an external pos tagger to assign inflection tags to each word. this tagger requires language-specific training data, which can be a challenge for low resource languages. however, this could be an advantage since the overall system can be improved by training the tagger on dialect-specific datasets, or readily extended to other languages given a suitable tagger. another drawback of bite is that it increases the length of the encoded sequence which may lead to extremely long sequences if used on morphologically rich languages. however, this is not an issue for english transformer models since the increase in length will always @xmath0, such that the increase in complexity is a constant factor. ## conclusion the tokenization stage of the modern deep learning nlp pipeline has not received as much attention as the modeling stage, with researchers often defaulting to common subword tokenizers like bpe. we can do better. by encoding raw text into operable symbols, we can improve the generalization and adversarial robustness of resulting systems. hence, we guide the data-driven tokenizer by incorporating linguistic information to learn a more efficient vocabulary and generate symbol sequences that increase the network's robustness to inflectional variation. this improves its generalization to l2 and world englishes without requiring explicit training on such data. since dialectal data is often scarce or even nonexistent, an nlp system's ability to generalize across dialects in a zero-shot manner is crucial for it to work well for diverse linguistic communities. a more general, bite-like algorithm should enable further gains on morphologically rich languages. finally, given the effectiveness of the common task framework for spurring progress in nlp @xcite , we hope to do the same for tokenization. as a first step, we propose to evaluate an encoding scheme's efficacy by measuring its vocabulary coverage and symbol complexity (which may have interesting connections to informationtheoretic limits @xcite ). we have already shown that base-inflection encoding helps a data-driven tokenizer use its limited vocabulary more efficiently by reducing its symbol complexity when the combination is trained from scratch.