Gold Panning in Vocabulary: An Adaptive Method for Vocabulary Expansion of Domain-Specific LLM s (2024)

Abstract: While Large Language Models (LLMs) demonstrate impressive generation abilities, they frequently struggle when it comes to specialized domains due to their limited domain-specific knowledge. Studies on domain-specific LLMs resort to expanding the vocabulary before fine-tuning on domain-specific corpus, aiming to decrease the sequence length and enhance efficiency during decoding, without thoroughly investigating the results of vocabulary expansion to LLMs over different domains. Our pilot study reveals that expansion with only a subset of the entire vocabulary may lead to superior performance. Guided by the discovery, this paper explores how to identify a vocabulary subset to achieve the optimal results. We introduce VEGAD, an adaptive method that automatically identifies valuable words from a given domain vocabulary. Our method has been validated through experiments on three Chinese datasets, demonstrating its effectiveness. Additionally, we have undertaken comprehensive analyses of the method. The selection of a optimal subset for expansion has shown to enhance performance on both domain-specific tasks and general tasks, showcasing the potential of VEGAD.

## introduction despite achieving satisfactory performance on a wide range of tasks @xcite @xcite , large language models (llms) continue to encounter challenges, particularly in domain-specific tasks, such as the generation of legal, medical, and financial texts. the expansion of vocabulary @xcite @xcite serves as a strategy to enhance the decoding efficiency for domain-specific llms. by concatenating specific, frequent n-grams into new words, the token sequence is shortened, thereby visibly boosting efficiency. @xcite extended llama's existing vocabulary with an additional 20,000 chinese tokens, thereby improving its encoding efficiency and semantic understanding of chinese. lawgpt foot_0 is fine-tuned based on the general chinese llms (such as chinese-llama, chatglm @xcite , etc.), the legal domain specific vocabulary is expanded to enhance the semantic understanding ability of the llms. current researches primarily focus on some specific domains. nonetheless, they have not thoroughly elucidate the performance enhancements resulting from vocabulary expansion in various domains. we conduct a pilot study illustrating the domain performance and general capabilities after vocabulary expansion with different sizes, and the results are illustrated in figure 1 . it is revealed that augmenting the size of the newly added vocabulary does not invariably result in improved model performance. hence, an essential question arises regarding the generation of an optimal subset for vocabulary expansion given a candidate vocabulary. the process of selecting highvalue vocabulary during the expansion of domainspecific llms is akin to gold panning, as it requires careful selection rather than indiscriminate enlargement of the lexicon to enhance the performance of the llms. we recognize the following challenges for vocabulary subset generation: to effectively identify the crutial words from a candidate vocabulary, we have proposed ve-gad (abbreviation of "vocabulary expansion via gradients"), which is an adaptable vocabulary expansion method via gradients. figure 2 provides an illustration of the framework. intuitively, token groups displaying larger gradients in domain instances are deemed more pivotal to the task and should be integrated into the vocabulary as domainspecific terms. therefore, it is a straightforward approach to trace the gradient of each word, while there are several difficulties, such as the algorithm to efficiently retrieve the candidate words from the token sequences, and the gradient calculation across various tokens rather than the whole sequence. to identify candidate words from the token sequences, we build a trie @xcite based on the candidate vocabulary, and design an algorithm to record the gradient for each word with the trie. to distinguish the effect of each token, the gradient is calculated on the running tensors, instead of the weights of the llms. to scrutinize the efficacy of vegad, we have undertaken comprehensive studies. the findings across three chinese datasets, pertaining to the domains of law and medicine, underscore a superiority in comparison to other lexicon generation techniques, as well as the promising prospects of domain-specific vocabulary expansion. our inquiry reveals that the domain-specific lexicon by vegad enhances performance in tasks requiring specialized knowledge as well as tasks demanding general skills. we hope that our multi-perspective analysis serves as a catalyst for future investigations into enhancing domain-task performance and mitigating the catastrophic forgetting through domain vocabulary adaptation. in summary, our contributions are three folds: ## related work large language models, such as chatgpt foot_1 , @xcite , exhibit amazing abilities on understanding and text generation. they can handle the tasks of qa, reasoning and math calculation even under zero-shot scenarios. llama @xcite is a collection of open foundation language models ranging from 7b to 65b parameters. @xcite developed and released llama 2, a collection of llms ranging in scale from 7b to 70b parameters. the fine-tuned llama 2-chat, are optimized for dialogue use cases. there are other popular llms developed with various skills @xcite @xcite @xcite . ## method in this section, we introduce vegad, a vocabulary expansion method via gradient for domainspecific llms. the process is shown in figure 3 . our approach is inspired by an naive intuition: ngram tokens exhibiting larger gradients in response to domain-specific instances are deemed crucial for the task at hand, and therefore, warrant inclu- specifically, starting from the domain-specific data, sentences are divided into discrete words. the candidate vocabulary is constructed with words absent from the general lexicon. subsequently, the process of selection is executed on domain-specific instances by computing the gradients for each node within the embedding tensor and the language modeling tensor, with reference to a trie constructed based on the candidate vocabulary. the top k words exhibiting the highest overall gradients are retained to establish the specialized domain vocabulary. then we resize the llm and incorporate the tokenizer with new vocabulary, following an optional weight initialization. then we conduct domain sft on the llm, to develop the domainspecific llm. the advantage of vegad can be summarized as following: 1) vegad is a plug-and-play taskadaptive vocabulary selection method, seamlessly integrating with diverse techniques utilized in supervised fine-tuning. 2) in contrast to previous methods such as @xcite , which might alter the intrinsic behaviors of current tokenizers such as wordpiece and bpe by imposing an obligatory scoring mechanism for sampling in accordance with their guidelines, vegad is tokenizeragnostic, and compatible to any tokenization algo-rithms. ## conclusion the influence of adding domain-specific words and the generation of domain vocabulary are far from being explored for llms. in this paper, we investigate the influence of adding domain vocabulary to llms from the perspective of both domain expertise and forgetting of general capabilities. we find that expansion with only a subset of the entire vocabulary may lead to superior performance. based on which, an automatic approach to identify effective words from a candidate vocabulary, called ve-gad, is proposed for the generation of an optimal subset. extensive experiments on three datasets from two domains, are conducted to prove the effectiveness of vegad. it is concluded from the analyses that not only the performance on domainspecific tasks is improved, but also the problem of catastrophic forgetting is mitigated.