C har2 S ubword: Extending the Subword Embedding Space Using Robust Character Compositionality (2021)

Abstract: Byte-pair encoding (BPE) is a ubiquitous algorithm in the subword tokenization process of language models as it provides multiple benefits. However, this process is solely based on pre-training data statistics, making it hard for the tokenizer to handle infrequent spellings. On the other hand, though robust to misspellings, pure character-level models often lead to unreasonably long sequences and make it harder for the model to learn meaningful words. To alleviate these challenges, we propose a character-based subword module (char2subword) that learns the subword embedding table in pre-trained models like BERT. Our char2subword module builds representations from characters out of the subword vocabulary, and it can be used as a drop-in replacement of the subword embedding table. The module is robust to character-level alterations such as misspellings, word inflection, casing, and punctuation. We integrate it further with BERT through pre-training while keeping BERT transformer parameters fixed–and thus, providing a practical method. Finally, we show that incorporating our module to mBERT significantly improves the performance on the social media linguistic code-switching evaluation (LinCE) benchmark.

## introduction byte-pair encodings (bpe) is a ubiquitous algorithm in the tokenization process among transformer-based language models such as bert @xcite , gpt-2 @xcite , roberta @xcite , and ctrl @xcite . this method addresses the open vocabulary problem by segmenting unseen or rare words into smaller subword units while keeping a reasonable vocabulary size @xcite figure 1 : examples of subword tokenization from oov words. the word helllo changes its meaning (e.g., hell), business is split almost to characters, and nanotechnologi do not resemble any of its morphemes. @xcite . however, bpe and its variants are sensitive to small perturbations in the text, potentially distorting the sentences' meaning @xcite (see figure 1 ). moreover, this tokenization process is rigid to changes such as adding more subwords to the vocabulary or correcting the segmentation splits. that is because the tokenization relies on the original corpus where the vocabulary was generated (e.g., wikipedia), resulting in a fixed set of subword pieces tied to an embedding lookup table @xcite . although these aspects are not a problem with clean and properly formatted text, that is not the case when the text presents substantial noise (e.g., wikipedia vs. social media). noisy text can result in extensive subword pieces per word (see figure 1 ), preventing the models from capturing the meaning effectively and adapting to such domains. this is particularly prominent on social media text @xcite , where the noise permeates even across languages and in codeswitching scenarios @xcite @xcite . this paper proposes a character-to-subword (char2subword) module trained to handle rare or unseen spellings robustly while being less restrictive to a particular tokenization method. our method works as a drop-in alternative to the embedding table in pre-trained language models like mbert. it improves performance and reduces the number of embedding parameters by 45% without sacrificing inference speed. we train our module to approximate the embedding table using characters from the original vocabulary words and subwords. this procedure leverages transfer learning from the pretrained embedding table rather than starting from scratch-thus, saving precious computational time and resources. besides, the subword vocabulary provides enough character-level patterns to learn from already-segmented tokens. we integrate our module with mbert's transformer layers even further by continuing to train with the pretraining data and the mlm objective. once our char2subword is adapted to the pre-trained language model, we evaluate the overall model performance by fine-tuning it on downstream tasks. we show our method's effectiveness by outperforming mbert on the social media linguistic code-switching (lince) benchmark @xcite , where the fine-tuning domain deviates substantially from the pre-training domain. the results show that the char2subword module can also capture intra-word code-switching. at the sentence level, the model can relate words from the same language to support language prediction. we highlight our main contributions as follows: ## related work word representations most of the initial ground-breaking advances in nlp relied on word embedding representations from methods like word2vec @xcite and glove @xcite . they showed the capability of arranging words in a continuous high-dimensional space encoding semantic relationships and meaning @xcite . however, rare words are weakly represented in such space, and oov words are not representable. to alleviate that, researchers proposed word representations using recursive neural networks guided by morphology @xcite , as well as morpheme embeddings as a prior distribution over probabilistic word embeddings @xcite . regardless, the challenges persist in noisy text, where users do not follow the canonical word forms @xcite . such problems are aggravated in social media due to the inherently multilingual environment. more words per language are required, while the spelling noise is persistent across languages. ## method given a word w, a subword model produces a sequence of subword pieces s " ps 0 , s 1 , . . . , s n q, such that the concatenation of all the segments from s fully reconstructs the word w. regardless of whether a subword piece represents a character in a word or not, all the pieces are treated as semantic units within a sentence. foot_0 such pieces come from a rule-based system that does not take into account semantics or morphology during the tokenization. thus, the subword tokenization has a significant impact on the semantic abstraction from upper layers in pre-trained models like mbert. to alleviate such problems, we build word representations out of characters. the char2subword module allows flexible tokenization patterns, where the model can split by spaces, use the original tokenization method, or employ a different tokenization process as defined by the user. there are two main phases in our proposed method: approximating subword embeddings with the char2subword module (i.e., ideally replicating the embedding space e) (section 3.1), and contextually integrating the char2subword module into the pre-trained model (section 3.2). ## experiments embedding approximation the goal of the approximation experiments is to replicate the original subword embedding table while ensuring robustness at the character level. we experiment with the objective functions described in section 3.1. we use the average precision to determine the best method (we also provide the accuracy for reference). 7 the experiments 1.1-1.4 show the results of each objective separately (see character-level robustness is another essential aspect when optimizing the char2subword. we add single-character edits to the training phase described in section 3.1. for the pos and ner tasks, we see improvements compared to mbert. the hybrid pre-trained experiment for hindi-english is significantly better than the baseline for both pos (89.64% vs. 87.86%) and ner (74.91% vs. 72.94%). one of the reasons for this performance boost is due to the noise that splitting transliterated hindi (i.e., romanized hindi) generates for the baseline. on the contrary, the char2subword compresses the transliterated words into a single vector, reducing the noise in the model. the ner results for spanish-english (es-en) and modern standard arabic-egyptian ara8 the char2subword module never sees a subword from the vocabulary with more than a single character edit (i.e., we defined the robustness procedure this way). that means that the word business never appeared in training for the model. ## analysis attention for language identification figure 5 shows the visualization for the spanish-english lid task with an intra-sentential code-switching example (i.e., code-switching at the clause level of a sentence utterance). the example shows that the strongest connections at the word level (figure 5 (left)) happen for words in the same language. particularly, the word consequencias is slightly ambiguous since its morphology overlaps substantially with both the english and spanish versions. with the context from the surrounding spanish words, the model can determine that the word is spanish. although there are more patterns captured among all the heads in mbert, this pattern suggests that words of the same language can provide contextual support along with the sentence. in addition to the contextual support, characterlevel attention plays an important role when build-ing the word representation. particularly for this word, the ambiguity is introduced due to the letter q. note that the char2subword module creates strong connections with this letter and parts where more ambiguity could happen. for example, the letter i happens where the suffixes -cias (spanish) and -ces english could complete the word). error analysis by inspecting the mistakes of the model in the confusion matrix for the spanish-english lid development set, we noticed 112 english words predicted as spanish, and 101 spanish words predicted as english (see subword sequence lengths sequences from the subword tokenization are the same length or longer than the original sequence of tokens. quantifying that tells us about the opportunity that the char2subword mbert model has in practice. ## conclusion we provide a novel, flexible, and robust method to expand the mbert subword embedding table. the char2subword module provides more control at the tokenization level, and it can generate word embeddings without being restricted to a fixed vocabulary or segmentation method. also, the char2subword module gives the possibility to refine a language or domain of interest (i.e., by pretraining the char2subword module) while preserving its multilingual properties. finally, this method is not limited to code-switching; the char2subword module is a general approach that can be applied to any word or subword-based pre-trained model. macherey, jeff klingner, apurva shah, melvin johnson, xiaobing liu, lukasz kaiser, stephan gouws, yoshikiyo kato, taku kudo, hideto kazawa, keith stevens, george kurian, nishant patil, wei wang, cliff young, jason smith, jason riesa, alex rudnick, oriol vinyals, greg corrado, macduff hughes, and jeffrey dean. 2016. google's neural machine translation system: bridging the gap between human and machine translation. corr, abs/1609.08144.