Breaking Token Into Concepts: Exploring Extreme Compression in Token Representation Via Compositional Shared Semantics (2025)

Abstract: Standard language models employ unique, monolithic embeddings for each token, potentially limiting their ability to capture the multifaceted nature of word meanings. We investigate whether tokens can be more effectively represented through a compositional structure that accumulates diverse semantic facets. To explore this, we propose Aggregate Semantic Grouping (ASG), a novel approach leveraging Product Quantization (PQ). We apply ASG to standard transformer architectures (mBERT, XLM-R, mT5) and evaluate this representational scheme across diverse tasks (NLI, NER, QA), as well as a biomedical domain-specific benchmark (BC5CDR) using BioBERT. Our findings demonstrate that representing tokens compositionally via ASG achieves extreme compression in embedding parameters (0.4–0.5%) while maintaining > 95% task performance relative to the base model, even in generative tasks and extends to both cross lingual transfer and domain-specific settings. These results validate the principle that tokens can be effectively modeled as combinations of shared semantic building blocks. ASG offers a simple yet concrete method for achieving this, showcasing how compositional representations can capture linguistic richness while enabling compact yet semantically rich models.

## introduction in modern language models, each token is typically represented by an individual, unique embedding. however, this approach may not be optimal, as semantically similar tokens (e.g., "mother," "mom," and their respective translations in different languages) can be assigned entirely distinct representations, potentially overlooking shared conceptual underpinnings. recent works @xcite @xcite suggests that token representations in llms implicitly encode higher-level semantic regularities, often described as concepts, which may be shared across words or subwords. while these studies analyze such concepts as emergent semantic categories or directions in representation space, our work explores an explicit, compositional formulation where tokens are represented as sequences of shared concept vectors. in parallel, @xcite proposed concept-level representations, grouping semantically similar tokens, using k-means. while this method achieved significant vocabulary compression with retained performance, it struggles with polysemy (e.g., "father" as family vs. religious figure) and is limited to encoder-only models, hindered by not explicitly predicting subword in autoregressive decoding. to address these limitations, we introduce aggregate semantic grouping (asg). asg maintains concept-level sharing but represents tokens as sequences of 'conceptids', thereby accumulating multiple semantic facets. this sequence-based representation is inspired by successful applications in information and generative retrieval @xcite @xcite . we employ product quantization (pq) @xcite to transform tokens into these conceptid sequences, aiming to preserve token's uniqueness and nuances while benefiting from shared semantics. our primary contribution is the introduction of aggregate semantic grouping (asg), a novel method leveraging product quantization to represent tokens as sequences of shared concep-tids, thereby capturing multiple semantic facets while significantly compressing embedding layer parameters. we provide a detailed methodology for applying asg to both encoder and encoder-decoder transformer models. conducting experiments across diverse tasks (nli, ner, qa) and models (mbert, xlm-r, mt5), we demonstrate that even with extreme compression on embeddings (down to 0.4-0.5% of the original embedding parameters), asg maintains high performance (@xmath0, including in zero-shot crosslingual transfer scenarios. furthermore, we extend our evaluation to a domain-specific benchmark (bc5cdr; @xcite and a domainspecialized model (biobert; @xcite , where asg achieves similar robustness, confirming its applicability beyond general-domain tasks. the code will be available at https://github.com/ kavinrv/aggregate-semantic-grouping . ## aggregate semantic grouping (asg) our approach, aggregate semantic grouping, reframes token representation by learning compositional embeddings from pre-trained models. ## conclusion this work investigated equipping language models with shared, compositional token representations as an alternative to traditional monolithic embeddings. we explored this through aggregate semantic grouping (asg), where product quantization transforms embeddings into sequences of concep-tids that map to shared, learned concept vectors, enabling multifaceted semantic capture alongside significant compression. extensive experiments on diverse models (including mbert, xlm-r, and mt5) and nlu tasks (such as nli, ner, and qa) found asg maintains high performance (@xmath1. asg also outperformed prior semantic grouping methods, and proved effective for generative architectures. these findings confirm that asg's decomposition of tokens into shared components offers an efficient, semantically rich, and promising direction for language modeling; future work may explore dynamic or adaptive quantization techniques.