E m B yte: Decomposition and Compression Learning for Small yet Private NLP (2025)

Abstract: Recent breakthroughs in natural language processing (NLP) have come with escalating model sizes and computational costs, posing significant challenges for deployment in real-time and resource-constrained environments. We introduce EMBYTE, a novel byte-level tokenization model that achieves substantial embedding compression while preserving NLP accuracy and enhancing privacy. At the core of EMBYTE is a new Decompose-and-Compress (DeComp) learning strategy that decomposes subwords into fine-grained byte embeddings and then compresses them via neural projection. DeComp enables EMBYTE to be shrunk down to any vocabulary size (e.g., 128 or 256), drastically reducing embedding parameter count by up to 94% compared to subword-based models without increasing sequence length or degrading performance. Moreover, EMBYTE is resilient to privacy threats such as gradient inversion attacks, due to its byte-level many-to-one mapping structure. Empirical results on GLUE, machine translation, sentiment analysis, and language modeling tasks show that EMBYTE matches or surpasses the performance of significantly larger models, while offering improved efficiency. This makes EMBYTE a lightweight and generalizable NLP solution, well-suited for deployment in privacy-sensitive or low-resource environments.

## introduction recent advances in natural language processing (nlp) have yielded impressive performance across a wide range of tasks. however, these gains come at the cost of massive computational and memory requirements, which make such models expensive, slow, and impractical for deployment on resourceconstrained devices. model compression is therefore essential to enable real-time applications, ex-pand accessibility, and reduce the environmental impact of large-scale ai systems. to this end, various compression techniques, such as knowledge distillation @xcite , quantization @xcite , and parameter sharing @xcite , have enabled compact, efficient models with minimal accuracy loss. notable examples like albert @xcite , tinybert @xcite , and distil-bert @xcite demonstrate the success of these approaches. however, each presents tradeoffs: distillation may miss subtle patterns, quantization can hinder complex reasoning, and parameter sharing may reduce model expressiveness. beyond architectural compression, advanced tokenization strategies, such as subword and byte-level representations @xcite , have also improved efficiency and generalization. subword methods @xcite shrink vocabulary size and handle rare words effectively, especially in morphologically rich languages. yet, they often lead to longer sequences, fragmented semantics, and reduced domain robustness. byte-level models, with their fixed 256-symbol vocabulary, offer strong multilingual generalization and language neutrality @xcite , but at the cost of slower training and limited semantic abstraction. moreover, both approaches face challenges in preserving privacy and providing secure, interpretable representations, highlighting the need for continued innovation in scalable and robust nlp. to address above challenges, we introduce a novel method, embyte , that significantly shrinks model size and provides privacy protections simultaneously. it combines the strengths of byte-based nlp with architectural optimizations to deliver superior performance across key dimensions. we refer to this procedure as our decomposeand-compress (decomp) learning strategy. it follows a divide-and-conquer principle: first, it decomposes subword representations into finergrained units to reduce model size and provide privacy guarantees; then, it aggregates and compresses these units to reduce the input length for time efficiency. as illustrated in figure 1 , embyte employs a neural network to project the aggregated byte-level embeddings into a reduced-dimensional space. as a result, embyte maintains or even shortens input lengths compared to subword-based models, thereby preserving comparable training complexity. this enables us to leverage a smaller vocabulary without incurring the typical trade-off of increased input sequence length. specifically, the procedure consists of four steps: in the following, we provide further intuition on how embyte balances efficiency, privacy, and accuracy in a unified framework. efficiency. as shown in privacy. conventional subword models use a one-to-one mapping between tokens and embeddings, leaving them vulnerable to gradient inversion attacks (gias). in contrast, embyte performs byte-level decomposition, introducing a oneto-many mapping where each subword corresponds to multiple byte embeddings. this increases ambiguity and makes it significantly harder to reconstruct original tokens from gradients, thereby improving resistance to gias and offering stronger privacy guarantees than both subword and bytebased baselines. accuracy. despite its compact design, embyte matches or exceeds the accuracy of larger models. the use of smaller vocabulary improves generalization and reduces overfitting. furthermore, bytelevel decomposition captures fine-grained morphological patterns, such as suffixes (e.g., -s, -ed), which subword tokenizers may fragment or overlook. this leads to richer, more flexible representations that align better with linguistic structure. ## background subword model subword models address key limitations of word-level tokenization by splitting words into frequent subword units, enabling better handling of rare or unseen words and reducing vocabulary size. methods like byte pair encoding (bpe)! @xcite and sentence-piece @xcite strike a balance between character and word granularity, improving generalization and efficiency. this makes subword models, such as bert @xcite and gpt @xcite , the standard in modern nlp due to their strong performance and manageable memory footprint. byte model byte-level models process text directly as sequences of bytes, removing the need for tokenizers and enabling true language agnosticism. with a fixed 256-token vocabulary, they handle rare words, noisy input, and multilingual text uniformly. models like canine @xcite , byt5 @xcite , and byte-level mbart @xcite show competitive performance while improving robustness and privacy. however, they incur longer input sequences and higher computational costs due to learning from lower-level representations. threat models we consider a white-box adversary aiming to recover at least one sentence from a victim's private training batch b i using model parameters θ t and gradients ∇ θ t l θ t (b i ). this access enables extraction of vocabulary v and embedding matrix w, allowing identification of updated tokens. repeated attacks can reveal a significant portion of the dataset, violating federated learning privacy. we investigate two gradient leakage attacks: (1) optimization-based attacks that reconstruct inputs by minimizing the gradient difference between dummy and real data @xcite @xcite ; and (2) the film attack @xcite , which extracts likely tokens from embedding gradients, generates candidate sentences via beam search with a pretrained language model, and refines them by reordering subwords. while defenses exist for the first, film remains largely unmitigated. ## embyte we introduce our method, embedding bytes of subwords (embyte ), illustrated in figure 1 . the approach converts input text into byte sequences, retrieves corresponding byte embeddings, aggregates them, and applies a low-dimensional projection to produce final subword byte embeddings. our objective is to develop a subword encoding scheme that leverages a compact byte-level embedding matrix to defend against gradient inversion attacks, preserve subword boundaries for time efficiency, reduce memory usage, and improve prediction accuracy. this gives rise to three key challenges: (1) how to represent subwords as byte sequences, (2) how to compute effective embeddings from these byte-level representations, and (3) how to project high-dimensional byte embed-dings into the same low-dimensional space used for standard subword embeddings. ## experiments we conduct experiments to demonstrate the advantages of embyte in preserving privacy for nlp models, slightly improving on model prediction accuracy, reducing space complexity, and maintaining time efficiency. in all experiments, we set v @xmath0, to minimize the risk of reassigning two subwords to the same byte sequence. implementation and result details are deferred to the appendix b, with section numbers corresponding (e.g., section 4.1.1 → appendix b.1.1). ## conclusion we present embyte , a novel language representation method using decomp learning for efficient, privacy-preserving training. by projecting subword byte embeddings into lower dimensions, embyte reduces model size, improves accuracy, and defends against gradient-based privacy attacks.