Learning Mutually Informed Representations for Characters and Subwords (2024)

Abstract: Most pretrained language models rely on subword tokenization, which processes text as a sequence of subword tokens. However, different granularities of text, such as characters, subwords, and words, can contain different kinds of information. Previous studies have shown that incorporating multiple input granularities improves model generalization, yet very few of them outputs useful representations for each granularity. In this paper, we introduce the entanglement model, aiming to combine character and subword language models. Inspired by vision-language models, our model treats characters and subwords as separate modalities, and it generates mutually informed representations for both granularities as output. We evaluate our model on text classification, named entity recognition, POS-tagging, and character-level sequence labeling (intraword code-switching). Notably, the entanglement model outperforms its backbone language models, particularly in the presence of noisy texts and low-resource languages. Furthermore, the entanglement model even outperforms larger pre-trained models on all English sequence labeling tasks and classification tasks. We make our code publically available.

## introduction since the emergence of pretrained language models (lms) like elmo @xcite and bert @xcite , subwords tokenization have become the prevailing approach to tokenization. common techniques include byte-pair-encoding (bpe) @xcite , wordpiece @xcite , and sentencepiece @xcite , which create word-sized character n-grams for the lm to learn reusable representations. however, subword tokenization has limitations: the number and vocabulary of subwords must be predetermined during pretraining. consequently, tasks involving noisy text or low-resource languages often require meticulous engineering to achieve satisfactory performance. a less studied alternative is tokenizing at the character or byte level. pretrained lms like canine @xcite , charformer @xcite , and byt5 @xcite utilize character-level tokenization. though such models usually require careful design to handle longer sequences resulting from fine-grained tokenization, they offer advantages such as better incorporation of morphology and avoidance of tokenization overfitting to the pretraining corpus domain. previous studies have shown that incorporating both character and subword (or full word) representations can enhance model generalization. however, most studies focused on using characters to enhance or refine word representations @xcite @xcite @xcite . however, these models, unlike the character-level pretrained language models mentioned earlier, do not generate usable character-level representations. in this paper, we argue that character and subword representations are distinct yet complementary. we introduce a novel model, named the entanglement model, which combines a pretrained character lm and a pretrained subword lm. inspired by techniques from the vision-language models (specifically vilbert @xcite )), we treat characters and subwords as two modalities and leverage cross-attention to learn new representations by iteratively attending between the character and subword sides of the model. the result is a simple, yet general approach for bringing together the fine-grained representation afforded by characters with the rich memory of subword representations. we evaluate our entanglement model on a variety of tasks (named entity recognition (ner), part-of-speech (pos) tagging, and sentence classification), domains (noisy and formal text), and languages (english and ten african languages). we also evaluate the entanglement model on characterlevel tasks (intraword code-switching), which cannot be processed by subword models. empirically, our model consistently outperforms its backbone models and previous models that incorporate character information. on english sequence labeling and classification tasks, the entanglement model even outperforms larger pre-trained models. further, we found that the usage of subword-aware character representations yields performance gains, compared to using a character-only model. in order to better understand the effectiveness of our model, we also explore two natural extensions: (1) incorporating positional embeddings that explicitly align the characters and subwords and (2) masked language model (mlm) pretraining of the entanglement model. we find that these augmentations of the model are unnecessary, suggesting that our model is capable of learning positional alignment between characters and subwords on its own and leveraging the substantial pretraining of the backbone models without costly pretraining of our entanglement cross-attention layers. ## methods we propose a novel entanglement model that allows information exchange between pretrained character models and subword models, which is facilitated by two separate sets of co-attention modules. our intention is for each layer of co-attention to further entangle the subword and character representations. the model thereby builds subword representations that are character-aware and character representations that are subword-aware which can be used on both character-level and word-level tasks. we apply the model to sequence labeling and text classification assuming a dataset of n samples and k classes, @xmath0, y (i) )} n @xmath1, where x (i) ∈ r n i is a sequence of words of length n i with label y (i) . for sequence labeling, the label y (i) ∈ {1, 2, • • • , k} n i is a vector with the same length as x (i) . for text classification, the label y (i) ∈ {1, 2, • • • , k} is an integer. ## results we conduct an extensive analysis of our model's performance on various sequence labeling and text classification tasks. we evaluate the effectiveness of our model on both formal and noisy english texts, as well as low-resourced languages, in order to assess its capabilities across different scenarios. moreover, for each task, we report the performance of different configurations of our model, such as utilizing the subword or character side and varying the number of co-attention modules. this approach enables us to examine the robustness of our modules under different hyperparameter settings. ## model extensions in this section, we explore two natural extensions that demonstrate how the simplicity of our model eliminates the need for additional complexity. positional embeddings we experimented with several ways to add positional embeddings (pe) in the co-attention module. details on pe training are in appendix a. from ## conclusion in this paper, we introduce a novel entanglement model to effectively combine character and subword language models using co-attention modules. unlike many prior works, our model produces mutually informed representations of subwords and characters, which could be used to process both subword and character-level tasks. its architecture is model-agnostic, and it opens new directions for pretraining and scaling up. empirically, our model has demonstrated improvements over the baseline models on various sequence labeling and text classification tasks. our entanglement model achieves state-of-the-art results on various tasks/settings: pos tagging on tweebank, ner on ibo and wolof from masakhaner, and intraword code-switching on german-turkish. notably, the improvement of our model is most significant for noisy texts and low-resourced, morphologically rich languages. furthermore, the entanglement model outperforms larger pretrained subword models, which have higher parameter counts, on most tasks. while our model features a simple architecture, incorporating extensions like positional embeddings or additional pretraining do not improve its performance, which implies that the model's structure facilitates the learning of relevant information during fine-tuning, rendering additional complexities unnecessary.