Generation with Dynamic Vocabulary (2024)

Abstract: We introduce a new dynamic vocabulary for language models. It can involve arbitrary text spans during generation. These text spans act as basic generation bricks, akin to tokens in the traditional static vocabularies. We show that, the ability to generate multi-tokens atomically improve both generation quality and efficiency (compared to the standard language model, the MAUVE metric is increased by 25%, the latency is decreased by 20 %). The dynamic vocabulary can be deployed in a plug-and-play way, thus is attractive for various downstream applications. For example, we demonstrate that dynamic vocabulary can be applied to different domains in a training-free manner. It also helps to generate reliable citations in question answering tasks (substantially enhancing citation results without compromising answer accuracy).

## introduction vocabulary, which defines basic bricks (tokens) for composing new sentences, bridging different languages, and alleviating harmful generations, is essential for language models @xcite @xcite @xcite . in modern development, vocabularies are often obtained by training tokenizers with a pre-defined vocabulary size on a pre-defined corpus. once built, they are kept unchanged in the following model construction and deployment @xcite . though it is sufficient for basic language modeling, this static setting makes vocabulary be quietly ignored in advanced generation tasks @xcite @xcite . for example, it can not be augmented token 1 token 2 … … … written by mark ravenhill theatre production was performed at language model … … … … token 50257 fixed token vocabulary phrase table input token embedding layer output token embedding layer theatre production next token/phrase generation dynamic vocabulary constructed based on input text written by mark ravenhill with new phrases for better adapting to an unseen domain @xcite @xcite or verbatim reference text spans for better inline evidence generation @xcite . to bring vocabulary back to the stage, it is natural to ask whether prior constraints posted by tokenization corpus and fixed vocabulary size can be relaxed. here, we explore vocabulary in a new dynamic setting. instead of being a fixed token table, dynamic vocabulary is required to be able to include arbitrary text spans on demand. this setup brings new challenges to the language model. on the input side, using a single embedding layer is no longer feasible as the full table can not be enumerated. on the output side, the model needs a stronger next-token predictor as the model allows multiple oracles (tokenized to different granularity) for a single string. in this work, we build a dynamic vocabulary by building a dynamic phrase encoder. akin to the embedding layer, the encoder maps arbitrary text spans (called phrases) to the input space of language models. it can be trained with existing language models in the same self-supervised manner, despite that multiple tokens (in the original static vocabulary) can be input or output at a single step. though the paradigm is almost unchanged, supporting dynamic tokens needs non-trivial modification on data curation. specifically, we find that, to prevent the learned model from either biased towards full static token outputs or towards full new phrase outputs, it is crucial to make the two properly interleaved in training samples. we also show that the phrase encoder is hard to learn without informative negative samples. we thus develop two retrieval-based and generation-based methods for accelerating the learning of the dynamic phrase encoder. the obtained dynamic vocabulary can be deployed in the way of plug-and-play: the underlying architecture (and backbone parameters) of language models are kept, and those new on-demand phrases can be used as ordinary tokens during the generation. to evaluate the dynamic vocabulary, we investigate three exemplar applications, including basic language modeling, domain adaptation, and generating citations for question answering. results show that the new flexibility of vocabulary both improve basic generation performances (e.g., stronger fluency and diversity scores on wikitext-103 @xcite with lower latency) and provide a new tool to handle advanced language modeling tasks (e.g., generating more accurate citations with qa scores also increased). ## related work tokenizer tokenizer is an essential component of language models @xcite , responsible for transforming raw text into a sequence of tokens. byte-pair encoding (bpe) is commonly used to build tokenizer @xcite @xcite and, there exist other tokenization algorithms, such as unigram @xcite and wordpiece tokenization used in bert @xcite . however, these tokenizations are limited to subwords or whole words. @xcite and @xcite generalize the bpe algorithm to multi-words and multi-tokens separately. whereas these approaches necessitate training the tokenizer and remain static. cog @xcite and cog-2 @xcite both employ a "dynamic vocabulary" by expanding vocabulary with phrases extracted from related documents. however, these two methods only employ dynamic vocabulary in the output module and split phrases into tokens in the input. in this paper, we treated phrases as atomic units same as tokens, and dynamically expanded vocabulary both in input and output layers. sequence compression language models are constrained by the limited length of input sequences they can process. increasing this length results in a prohibitive computational overhead. a series of techniques have been proposed to compress sentences into one or a few tokens or latent representations @xcite @xcite . mwt @xcite enhances compression by retraining the tokenizer, incorporating the most frequent n-grams of a support corpus into the vocabulary. in contrast to the static vocabulary of mwt, our method dynamically adapts the model's vocabulary to the input text, resulting in a more flexible and efficient adaptation. ## conclusion in this paper, we propose a novel approach for dynamically adjusting the model's vocabulary based on the input text. it is a plug-and-play approach that can be simultaneously performed with pre-training tasks. we investigated standard language modeling, domain adaptation, and citation generation, and discussed the impact of different training samples and negative phrase construction methods on the quality of generated text. our experimental results show that our proposed model can rapidly generate high-quality, high-compression text compared to baselines.