From Characters to Words: Hierarchical Pre-trained Language Model for Open-vocabulary Language Understanding (2023)

Abstract: Current state-of-the-art models for natural language understanding require a preprocessing step to convert raw text into discrete tokens. This process known as tokenization relies on a pre-built vocabulary of words or sub-word morphemes. This fixed vocabulary limits the model’s robustness to spelling errors and its capacity to adapt to new domains. In this work, we introduce a novel open-vocabulary language model that adopts a hierarchical two-level approach: one at the word level and another at the sequence level. Concretely, we design an intra-word module that uses a shallow Transformer architecture to learn word representations from their characters, and a deep inter-word Transformer module that contextualizes each word representation by attending to the entire word sequence. Our model thus directly operates on character sequences with explicit awareness of word boundaries, but without biased sub-word or word-level vocabulary. Experiments on various downstream tasks show that our method outperforms strong baselines. We also demonstrate that our hierarchical model is robust to textual corruption and domain shift.

## introduction pre-trained language models with transformers have achieved breakthroughs in many natural language processing (nlp) tasks @xcite . one of the key advantages of transformers over traditional feature-engineered nlp pipelines is that transformers enable endto-end training from vast amount of data to automatically learn the optimal language representation @xcite . however, most recent language models still require a separate preprocessing stage known as tokenization. tokenization is a process that splits raw text parts into a list of discrete tokens from a fixed vocabulary. this based on the granularity of the basic token units, tokenization methods can be divided into three categories: character-based, subword-based and wordbased. a word-based tokenizer segments sentence into smaller chunks of words. due to language complexity and memory limit, a word-based vocabulary can not represent all possible words. wordlevel tokenization thus frequently runs into the issue of out-of-vocabulary words. a character-based tokenizer simply splits the text into a sequence of its characters. it is flexible to encode arbitrary words, but character-level tokenization produces long sequences, which is undesirable as the computational cost of transformers grows quadratically with the sequence length. to strike a good balance between time and space complexity, most stateof-the-art pre-trained language models thus adopt sub-word tokenization. data-driven sub-word tokenizers @xcite @xcite are typically pre-trained on a general text corpus to learn a subword vocabulary based on the frequency of word fragments. despite their popularity, sub-word tokenizers limit the robustness and generalizability of the language models built upon them. first, sub-word tokenizers are sensitive to small textual perturbations @xcite . while humans can still comprehend text with subtle misspellings and capitalization variants @xcite , these perturbations can drastically change the tokenization results, potentially leading to a suboptimal text representation. second, the sub-word vocabulary is pre-built and remains frozen during the language model pre-training and task-specific fine-tuning. therefore, when adapting a pre-trained language model into a new language context (e.g. biomedical texts and business documents), the tokenizer is prone to excessive fragmentation of subword pieces @xcite , as illustrated in we aim to bring the best of both character-based and word-based models to address the challenges discussed above. to this end, we propose a novel pre-trained language model with a hierarchical twolevel architecture. at the word level, we split the text sequence by characters, and introduce an intraword module that uses transformers to learn a representation for each word in the sequence from the embeddings of their respective characters. at the sequence level, we introduce an inter-word module that contextualizes the embedding for every words in the text sequence. our method does not require explicit sub-word or word-level vocabulary, and can thus be considered as an open-vocabulary approach @xcite . by limiting the attention range to characters within the same word rather than the full sequence in the intra-word module, our model remains computationally efficient. in order to validate our model, we comprehensively compare our method with various baseline methods, including the most popular sub-word based model bert @xcite , some state-of-the-art character-based models @xcite , and an hybrid character/sub-word model @xcite . besides standard benchmarking, we also test the robustness of the various models in two ways: by introducing spelling noise into the validation set and by testing on cross-domain tasks. our contributions can be summarized as follows: an ablation study highlights its effectiveness. ## method most character-level transformer encoder models are sub-optimal for two reasons: (1) dense self-attention on long character sequence is computationally expensive; (2) they do not leverage word boundary, which is an important inductive bias in linguistics. to overcome these challenges, we propose to decompose dense character-level transformer encoder into two parts: intra-word transformer encoder and inter-word transformer encoder. our hierarchical language model (hlm) adopts an hourglass structure @xcite and contains three main components: (1) an intraword module that learns word embeddings from their characters; (2) an inter-word module which contextualizes the word representations by attending to all words in the input sequence; (3) an intra-word prediction head for character-level pretraining. the overall architecture of our model is shown in fig. 1 . in the following sections, we discuss each component separately. ## experiments we evaluate the performance of our pre-trained model on a wide range of downstream tasks. we compare the performance of our pre-trained hierarchical language model (hlm) with various baseline methods, including the popular sub-word based bert model, three recent byte/characterlevel models, as well as a hybrid model referred to as characterbert. for bert, we use the cased bert base model (108m parameters) to match our inter-word transformers module setup. for ca-nine, we adopt canine-c (132m) which also uses a character-level pre-training task. for charac-terbert, we use the general version (105m) which is pre-trained on english wikipedia and openweb-text. for those baseline models, we use the pretrained weights hosted on huggingface † or released by the authors. for charformer (203m) and bytelevel t5 (200m), we use results of the base version from the original paper as pre-trained weight is not available. ## conclusion in this work, we propose a novel hierarchical language model for open-vocabulary language understanding. our method does not rely on explicit sub-word or word vocabulary. we demonstrate that our hlm model outperforms baseline methods on standard benchmarks, and highlight its robustness to spelling errors and domain shifts. in future work, we will expand our language support and explore incorporating a decoder for generative tasks.