BERT wich: Extending BERT ’s Capabilities to Model Dialectal and Noisy Text (2023)

Abstract: Real-world NLP applications often deal with nonstandard text (e.g., dialectal, informal, or misspelled text). However, language models like BERT deteriorate in the face of dialect variation or noise. How do we push BERT’s modeling capabilities to encompass nonstandard text? Fine-tuning helps, but it is designed for specializing a model to a task and does not seem to bring about the deeper, more pervasive changes needed to adapt a model to nonstandard language. In this paper, we introduce the novel idea of sandwiching BERT’s encoder stack between additional encoder layers trained to perform masked language modeling on noisy text. We find that our approach, paired with recent work on including character-level noise in fine-tuning data, can promote zero-shot transfer to dialectal text, as well as reduce the distance in the embedding space between words and their noisy counterparts.

## introduction pre-trained language models such as bert @xcite contain large amounts of knowledge within their numerous parameters but require relatively low computational power to fine-tune on a downstream task, making them the state of the art for a wide range of popular natural language processing (nlp) tasks. however, evaluation settings and metrics are not always representative of real world use cases. one recurring problem is the severe deficit in performance when bert-based models are used with noisy text. we define noisy text as any text that deviates from the pre-training data of the model by way of orthographic and/or grammatical variation. the possible sources of this variation are diverse and include dialect variation, user errors (typos, misspellings, grammatical errors), and transcription errors (propagated from errors in optical character recognition or automatic speech recognition systems). we aim to understand how bert's modeling effectiveness deteriorates in the face of noisy text and develop novel methods to improve bert's modeling of noisy text. studies have shown that humans can easily read text with high amounts of noise @xcite . it is curious, then, that bert's accuracy on downstream tasks drops dramatically when noise is added to test data @xcite @xcite . this seeming contradiction comes down to the way in which bert's textual input is expressed (subword tokenization) and the resultant units to which bert assigns contextual embeddings (subwords). for instance, take the sentence and corresponding tokens after applying the bert-base uncased tokenizer: i am a student i, am, a, student if there were a single typo in the topical cue student, the corresponding tokens would be as follows: i am a studebt i, am, a, stud, eb, t the subword tokenizer's rigidity manifests in two ways: over-segmentation and subword replacement @xcite . in the presence of noise, words get over-segmented into shorter-length tokens (stud, eb and t above), which may have less meaningful embeddings or even change the meaning of the sentence (stud above). these two roadblocks hinder the model from arriving at the correct meaning of a noisy sentence, particularly when, as above, the noise falls on a word with important content. in fact, we find that the distance between the vector encodings of words and their noisy counterparts is extremely high in bert's embedding space, and we aim to minimize this distance ( in this work, we aim to make bert-based models more robust to the issues of over-segmentation and subword replacement by exposing the models to text with induced character-level perturbations at various stages of training. previous efforts (see section 2) have demonstrated the effectiveness of bert encoder stack l0 lf new bert encoder stack continue mlm pre-training on noisy wiki text bert encoder stack l0 lf bert encoder stack l0 lf new bert encoder stack fine-tune on task with noised dataset classifier figure 1: a sketch of our method, bertwich. we sandwich bert's encoder stack between prepended and appended randomly-initialized encoder layers, l0 and lf. we continue the masked language modeling pre-training of this expanded encoder stack on synthetically-noised wikipedia text. the bertwich model can be fine-tuned on a downstream task with noised fine-tuning data to evaluate the model in settings with dialectal variation or noise in text. in this paper, we propose that the way out of this conundrum is to include additional transformer encoder layers (prepended and/or appended to bert's encoder stack) before continuing pre-training the full model for a few more epochs on syntheticallyperturbed wikipedia text (figure 1 ). we call this method bertwich. we evaluate our method on nonstandard text (from language varieties or dialects unseen during training) as well as synthetically-noised text (i.e., typo simulation). we find that sandwiching bert between added layers optimized for noisy text modeling improves bert's performance on text from unseen dialects and reduces the distance in the embedding space between words and their noisy counterparts, as well as the distance between sentence representations of standard and nonstandard text. our results indicate that without the additional layers, the model would have to re-learn representations of stud, eb, and t that are equivalent to the representation of student, while our method makes it easier for the model to simply learn to map the shorter-length tokens to the existing representation of student. ## related work the detrimental effect of noisy test data on bert's performance is well-documented @xcite @xcite . synthetically-induced errors (typos, misspellings, grammatical errors, and lexical changes) significantly decrease performance across a medley of tasks and test cases. moreover, past findings exhibit a large performance gap when evaluating on standard and nonstandard varieties of a language @xcite @xcite . as discussed above, the source of these issues boils down to the rigid relationship between subword tokenization and resultant subword token embeddings @xcite . three general approaches have been taken in past work to address this issue of noisy text modeling. some have tried to tackle it as a text normalization problem during preprocessing @xcite @xcite . others have attempted to get to the root of the problem by improving the tokenization algorithm or using a character-based model @xcite @xcite . the third approach is to inject synthetic noise in training, which we find to be the most flexible and generalizable while requiring a low computational load, and our method fits in this final category of solutions. by exposing the model to noise during some stage of training, the model is expected to be better-equipped to deal with noisy text during inference. for instance, @xcite include synthetic noise while training machine translation systems and find performance improvements in evaluation settings with natural noise; however, the approach depends on training a model from scratch rather than adapting an existing model like bert. using this approach in fine-tuning, past work has found augmenting the task training data with noisy or adversarial examples and fine-tuning the model with the augmented dataset to provide improvements in performance on noisy text @xcite . applying this approach to dialects, aepli and sennrich (2022) inject synthetic character-level noise in continued pre-training and fine-tuning steps for various tasks and find that part-of-speech tagging on dialectal text improves. followup work by @xcite develops methods of predicting the effectiveness of cross-lingual transfer based on the amount of noise injected. srivastava and chiang (2023) extend the method to strictly zero-shot settings and apply it to various sequence classification tasks, finding that the approach works well in tasks that can be solved using surface-level cues, but it is not as helpful in more challenging tasks, indicating that more can be done to truly improve the modeling of nonstandard text. a related but distinct approach involves adding adapter layers tuned to nonstandard text @xcite . our method is similar to this work when it comes to adding parameters to the model beyond those used in fine-tuning. however, the methods differ in terms of their functionality; while adapter-based approaches are typically specific to a set of language varieties, our approach provides general improvement to the language modeling capabilities of a bert-like model with wider applicability to various dialects and to noisy text. ## method as demonstrated by previous work, fine-tuning is a powerful method of domain adaptation and crosslingual transfer. however, while the model may adapt to performing a task on noisy text via fine-tuning, this does not speak to truly extending the model's capabilities and allowing it to better model noisy text. starting with a standard pre-trained model such as bert, our approach has two stages: 1) the bertwich method: add untrained layers and continue pre-training on noisy data (sections 3.1 to 3.3), and (2) fine-tune on a specific task (section 3.4). ## experiments we conduct experiments on sentiment analysis in english with simulated typographical errors, and intent classification in german across multiple dialects. ## results the results of our experiments are shown in tables 1 to 3 . models following the bertwich approach (added layers + cpt) fall in the final four rows of each table. ## conclusion in this paper, we explore the issues surrounding language modeling of nonstandard text and extend bert's language modeling capabilities to common real-world scenarios involving nonstandard text. such issues could arise for a variety of reasons, including inference on text exhibiting dialectal variation, user errors such as misspellings or typos, and propagated errors from optical character recognition and automatic speech recognition systems. we introduce the novel idea of adding randomlyinitialized layers to the encoder stack of a pre-trained bert model before continuing the pre-training of the new model on text with synthetically-induced noise. this way, the newly added layers improve bert's ability to model nonstandard text. our approach is called bertwich. we find that our approach not only improves downstream task performance on nonstandard text, but also expands bert's modeling capabilities. for instance, we find that the distance in bert's embedding space between words and their noisy counterparts (e.g., student vs. studebt) is extremely high, but is dramatically reduced in the embedding space of our bertwich models. furthermore, we find that the sentential (cls) representations within bert's layers are not equipped to represent dialect text, while those extracted from the upper layers of the best bertwich model are. our findings indicate a transformation in the role the lower layer(s) can play in representing noisy text -while bert would need to re-learn representations of the new token sequence for a noised word, our bertwich method promotes knowledge transfer from standard to nonstandard text modeling, making it easier for the model to perform mappings to standard variants of words.