Morphological inflection is a process of word formation where base words are modified to express different grammatical categories such as tense, case, voice, person, or number. World Englishes , such as Colloquial Singapore English (CSE) and African American Vernacular English (AAVE), differ from Standard English dialects in inflection use . Although comprehension by human readers is usually unimpaired by non-standard inflection use, NLP systems are not so robust. We introduce a new Base-Inflection Encoding of English text that is achieved by combining linguistic and statistical techniques . Fine-tuning pre-trained NLP models for downstream tasks under this novel encoding achieves robustness to non-standard inflection use while maintaining performance on Standard English examples . Models using this encoding also generalize better to non-standard dialects without explicit training . We suggest metrics to evaluate tokenizers and extensive model-independent analyses demonstrate the efficacy of the encoding when used together with data-driven subword tokenizers  .
Inflectional variation is a common feature of World Englishes such as Colloquial Singapore English and African American Vernacular English . Although comprehension by human readers is usually unimpaired by non-standard inflections, current NLP systems are not yet robust. We propose Base-Inflection Encoding (BITE), a method to tokenize English text by reducing inflected words to their base forms before reinjecting the grammatical information as special symbols . Fine-tuning pretrained NLP models for downstream tasks using our encoding defends against inflectional adversaries while maintaining performance on clean data . Models using BITE generalize better to dialects with non-standard inflections without explicit training and translation models converge faster when trained with BITE. Finally, we show that our encoding improves the vocabulary efficiency of popular data-driven subword tokenizers . Since there has been no prior work on quantitatively evaluating vocabulary efficiency, we propose metrics to do so .
R
Morphological inflection is a process of word formation where base words are modified to express different grammatical categories such as tense, case, voice, person, or number. World Englishes ,
Inflectional variation is a common feature of World Englishes
D
(CSE)
D
(AAVE), differ from Standard English dialects in inflection use
R
inflection use,
inflections, current
R
so
yet
R
introduce a new
propose
R
of English text that is achieved by combining linguistic and statistical techniques
(BITE), a method to tokenize English text by reducing inflected words to their base forms before reinjecting the grammatical information as special symbols
R
pre-trained
pretrained
R
under this novel encoding achieves robustness to non-standard inflection use
using our encoding defends against inflectional adversaries
R
Standard English examples
clean data
R
this encoding also
BITE
R
non-standard dialects
dialects with non-standard inflections
R
. We suggest metrics to evaluate tokenizers and extensive model-independent analyses demonstrate the efficacy of the encoding when used together with
and translation models converge faster when trained with BITE. Finally, we show that our encoding improves the vocabulary efficiency of popular
A
. Since there has been no prior work on quantitatively evaluating vocabulary efficiency, we propose metrics to do so