From Characters to Tokens: Dynamic Grouping with Hierarchical BPE (2025)

Abstract: Subword tokenization methods like Byte Pair Encoding (BPE) are widely used in large language models due to their balance of vocabulary compactness and representational power. However, they suffer from inefficiencies in representing rare words and require large embedding matrices. Character-level models address these issues but introduce performance bottlenecks, particularly in Transformer-based architectures. Recent hierarchical models attempt to merge the benefits of both paradigms by grouping characters into patches, but existing patching strategies either rely on whitespace—limiting applicability to certain languages—or require auxiliary models that introduce new dependencies. In this paper, we propose a dynamic character grouping method that leverages the structure of existing BPE tokenization without requiring additional models. By appending explicit end-of-patch markers to BPE tokens and introducing a second-level BPE compression stage to control patch granularity, our method offers efficient, flexible, and language-agnostic representations. Empirical results demonstrate that our approach matches or exceeds the performance of dynamic entropy- and whitespace-based patching strategies, while maintaining a compact vocabulary.

## introduction subword tokenization algorithms, particularly byte pair encoding (bpe), have become the de facto standard for text representation in large language models due to their balance between vocabulary compactness and representational flexibility. however, despite their widespread use, subword methods introduce notable limitations. embedding matrices tied to large vocabularies become parameter inefficient, for which rare words often have bad representations. while bpe provides a degree of compression over raw byte sequences-enhancing computational efficiency, the achievable compression is fundamentally constrained by the vocabulary size. for example, modern tokenizers used in models such as gemma 2 and llama 3 employ vocabularies of around 250k tokens, which inherently limits the extent to which sequence length can be reduced. character-level models directly address many of the limitations inherent in subword-based tokenization, particularly with regard to rare word handling and parameter efficiency. however, this comes at the cost of runtime performance, especially in quadratic-complexity architectures such as standard transformers. to bridge this gap, recent work has explored hierarchical representations that aim to combine the flexibility of characterlevel input with the efficiency of subword models. these approaches typically group characters into larger units-referred to as "patches"-and learn representations via neural networks rather than embedding tables. the effectiveness of such models is closely tied to the patching strategy. naively grouping consecutive characters has been shown to underperform relative to traditional bpe tokenization @xcite . in contrast, dynamic grouping strategies, particularly those that treat whitespace as a delimiter, have achieved competitive or state-of-the-art results. however, whitespace-based segmentation does not generalize to logographic writing systems such as chinese, where spacing is not semantically meaningful. one proposed solution involves training a lightweight character-level model to identify patch boundaries based on information-theoretic criteria, such as local entropy. in this setup, new patches are initiated in regions of high entropy (i.e., high surprise). while promising, this approach introduces a dependency on an additional model, which may be sensitive to domain shifts and data variability. in this paper, we address the limitations of character-level and subword-based models by introducing a dynamic character grouping method that avoids the need for training an additional model to determine patch boundaries. our approach defines each patch as the sequence of characters that constitute a bpe token, effectively repurposing the tokenization process itself as a grouping mechanism. to enable incremental processing, we modify the standard bpe encoder by appending an explicit end-of-patch marker after each token. while prior work has briefly mentioned this idea, to the best of our knowledge, we are the first to provide a detailed empirical evaluation. to offset the additional cost introduced by appending an end-of-patch symbol to each patch, we introduce a hierarchical bpe algorithm that compresses each character-level patch to a maximum predefined length s. this compression leverages the observation that short n-grams appear frequently across tokens, allowing for effective merging and shorter patch lengths. consequently, our method can be viewed as performing dynamic grouping of tokens, rather than operating solely at the character level. empirically, our model outperforms entropybased patching and achieves comparable performance with whitespace-based dynamic grouping, while maintaining broader applicability across writing systems. moreover, compared to standard bpe where the vocabulary is a large, our approach yields better efficiency, requiring significantly fewer flops. ## methodology we introduce a hierarchical representation model that enables more effective trade-offs between granularity, sequence length, and run-time efficiency. given an input sequence of characters c 1 , c 2 , . . . , c l , we apply a pre-trained bpe tokeniser to produce a sequence of variable-length subword tokens x 1 , . . . , x t . each token is sequence of characters, which we pass to our hierarchical bpe algorithm. our algorithm takes the sequence of characters and compresses it to a shorter sequence of integers and adds an end of sequence marker. then the sequence representing the initial bpe token is padded to the maximum length s foot_0 and passed to the neural network detailed in section 2.2. ## related work model-based dynamic grouping. numerous prior works have explored hierarchical models, particularly those involving dynamic grouping of characters into patches. the byte-level transformer (blt) @xcite employs a lightweight model trained on next-character prediction to determine patch boundaries. a new patch is initiated either when the entropy is high or when the current patch exceeds a predefined maximum length. this approach eliminates the need for an explicit endof-patch token but introduces an auxiliary model solely for patching. in contrast, our method avoids the need for such a model and handles the overhead introduced by end-of-patch tokens via a secondstage bpe compression, which effectively reduces the average patch size. other works, such as @xcite , explore dynamic pooling through learnable models. while they also incorporate groupings informed by subword tokenizers, a key distinction is that they explicitly train a model to replicate the behavior of a bpe tokenizer-an essential step in the absence of explicit patch boundary markers. earlier research has also attempted character-level compression using convolutional layers prior to applying a global transformer @xcite @xcite . space-based grouping. space patching is a well-studied method, with several recent works exploring its effectiveness at medium to large scales, including @xcite al. (2025), slagle (2024), sun et al. (2023), and thawani et al. (2023) . the latter introduces a minor modification to the hierarchical architecture, where the local encoder and decoder utilize more than one hidden representation per patch. prior to space patching, grouping of consecutive characters was explored by @xcite . token grouping. recent work has also investigated the consecutive grouping of tokens @xcite . this is related to our approach in that our hierarchical bpe tokenization can be interpreted as a form of dynamic grouping, where tokens are composed from a much smaller vocabulary v ′ . ## conclusion in this work, we presented a dynamic character grouping method that leverages bpe token boundaries to define patches, eliminating the need for an auxiliary neural network to determine patch segmentation. by augmenting the bpe process with an explicit end-of-patch token and introducing a second-stage compression step, we control patch granularity while maintaining a compact vocabulary. this design bridges the gap between character-level flexibility and subword-level efficiency, resulting in a hierarchical representation that generalizes beyond whitespace-delimited languages. furthermore, our method can also be interpreted as a dynamic token grouping strategy. the empirical results demonstrate improved efficiency and performance over entropy-based patching and standard bpe, with competitive results relative to space-based grouping. these findings highlight the effectiveness of token-level dynamic grouping as a lightweight yet expressive alternative to conventional tokenization strategies in large language models.