H ash F ormers: Towards Vocabulary-independent Pre-trained Transformers (2022)

Abstract: Transformer-based pre-trained language models are vocabulary-dependent, mapping by default each token to its corresponding embedding. This one-to-one mapping results into embedding matrices that occupy a lot of memory (i.e. millions of parameters) and grow linearly with the size of the vocabulary. Previous work on on-device transformers dynamically generate token embeddings on-the-fly without embedding matrices using locality-sensitive hashing over morphological information. These embeddings are subsequently fed into transformer layers for text classification. However, these methods are not pre-trained. Inspired by this line of work, we propose HashFormers, a new family of vocabulary-independent pre-trained transformers that support an unlimited vocabulary (i.e. all possible tokens in a corpus) given a substantially smaller fixed-sized embedding matrix. We achieve this by first introducing computationally cheap hashing functions that bucket together individual tokens to embeddings. We also propose three variants that do not require an embedding matrix at all, further reducing the memory requirements. We empirically demonstrate that HashFormers are more memory efficient compared to standard pre-trained transformers while achieving comparable predictive performance when fine-tuned on multiple text classification tasks. For example, our most efficient HashFormer variant has a negligible performance degradation (0.4% on GLUE) using only 99.1K parameters for representing the embeddings compared to 12.3-38M parameters of state-of-the-art models.

## introduction the majority of transformer-based @xcite pre-trained language models (plms; @xcite @xcite ) are vocabulary-dependent, with each single token mapped to its corresponding vector in an embedding matrix. this one-to-one mapping makes it impractical to support out-of-vocabulary tokens such as misspellings or rare words @xcite . moreover, it linearly increases the memory requirements with the vocabulary size for the token embedding matrix @xcite . for example, given a token embedding size of 768, bert-base with a vocabulary of 30.5k tokens needs 23.4m out of 110m total parameters while roberta-base with 50k tokens needs 38m out of 125m total parameters. hence, disentangling the design of plms from the vocabulary size and tokenization approaches would inherently improve memory efficiency and pretraining, especially for researchers with access to limited computing resources @xcite . previous efforts for making transformer-based models vocabulary-independent include dynamically generating token embeddings on-the-fly without embedding matrices using hash embeddings @xcite over morphological information @xcite . however, these embeddings are subsequently fed into transformer layers trained from scratch for ondevice text classification without any pre-training. @xcite proposed canine, a model that operates on unicode characters using a lowcollision multi-hashing strategy to support ~1.1m unicode codepoints as well as infinite character four-grams. this makes canine independent of tokenization while limiting the parameters of its embedding layer to 12.3m. @xcite proposed models that take as input byte sequences representing characters without explicit tokenization or a predefined vocabulary to pre-train transformers in multilingual settings. in this paper, we propose hashformers a new family of vocabulary-independent plms. our models support an unlimited vocabulary (i.e. all possible tokens in a given pre-training corpus) with a considerably smaller fixed-sized embedding matrix. we achieve this by employing simple yet computationally efficient hashing functions that bucket together individual tokens to embeddings inspired by the hash embedding methods of @xcite and @xcite . our contributions are as follows: ## hashformers in this section, we present hashformers, a family of vocabulary-independent hashing-based pretrained transformers. ## conclusions we have proposed hashformers, a family of vocabulary-independent hashing-based pre-trained transformers. we have empirically demonstrated that our models are computationally cheaper and more memory efficient compared to standard pretrained transformers, requiring only a fraction of their parameters to represent token embeddings. hashformer-lsh-proj variant needs 99.1k parameters for representing the embeddings compared to millions of parameters required by state-ofthe-art models with only a negligible performance degradation. for future work, we plan to explore multilingual pre-training with hashformers and explore their ability in encoding linguistic properties @xcite .