webmp3/Sakura-Gemma4-E4B-EmbeddingGemma2-DualMode

Sakura Gemma 4 E4B x EmbeddingGemma 2 - DualMode

One Gemma 4 E4B-it backbone for generation and EmbeddingGemma-2-distilled 768d retrieval. Research prototype. Community model. Not official from Google.

In 30 seconds

• What: one Gemma 4 E4B-it model that can chat and produce 768-dimensional search vectors for RAG, so no second embedding model has to be kept in memory.
• How: the same backbone runs in two modes. Generate is the untouched Gemma 4. Embed uses bidirectional attention, a small LoRA (rank 16) and a 768d head: +4.60 M parameters (8.8 MB).
• Quality: on external benchmarks the embed mode reaches 74 % (SciFact) and 66 % (NFCorpus) of the teacher's MRR. This is lower than the smaller E2B sibling (79 % / 71 %) with the same recipe: a bigger backbone did not give better embeddings here. Generation stays bitwise identical to plain Gemma 4.
• Formats: this repository (Safetensors adapter + Python runtime) and a GGUF version with a small llama.cpp patch.
• Honest limit: research prototype. Not a drop-in for EmbeddingGemma 2 and not index-compatible with its vectors. If you want the best retrieval from this project, use the E2B model.

Positioning

• Runtime backbone: google/gemma-4-E4B-it (loaded from the Hub, not duplicated here).
• Embedding teacher: google/embeddinggemma-2, used only for distillation. Not merged, not needed at runtime.
• Not index-compatible with EmbeddingGemma 2 vectors. No claim of being the first of its kind is made.

What it is

                        GENERATE MODE  (original Gemma 4 path, adapter fully bypassed)
Input -> Gemma 4 E4B-it backbone --+-- causal attention -> LM head
        (one resident model)       |
                                   +-- EMBED MODE
                                       bidirectional attention
                                       embedding-only LoRA (q_proj, o_proj, layers 24-40, rank 16)
                                       hidden state after layer 40  (hidden_states[41], before the final norm)
                                       mean pooling over valid tokens
                                       Linear 2560 -> 768 (no bias)
                                       L2 normalization   (MRL: 768 / 512 / 256 / 128)

Added parameters

Component	Shape / scope	Parameters	BF16
Embedding head	Linear(2560 -> 768, bias=False)	1,966,080	3.75 MB
Embedding-only LoRA	rank 16, alpha 32, q_proj + o_proj, layers 24-40 (34 modules)	2,637,824	5.03 MB
Total overhead		4,603,904	8.78 MB

That is +0.0576 % of the 7.996 B stored parameters of the E4B-it checkpoint. As with E2B, Gemma 4 E4B shares K/V across its last 18 layers (24-41), so k_proj/v_proj LoRA there has no effect and layer 41 lies after the pooled hidden state; only q_proj/o_proj of layers 24-40 are trained. Replacing a standalone EmbeddingGemma 2 text model (271.00 M parameters) saves about 266 M parameters (about 508 MB BF16 weights) on paper; runtime memory was not measured for E4B.

Results

Teacher EmbeddingGemma 2 vs. this model on the same texts, queries with task: search result | query: , documents with title: <title> | text: , at most 128 tokens for both (the training length). Both benchmarks were never used for training or model selection.

Benchmark (500-doc subset)	Teacher MRR	This model MRR	MRR vs teacher	nDCG@10 teacher / this	nDCG vs teacher	Gate
MTEB SciFact test (300 queries)	0.8971	0.6679	74.4 %	0.9137 / 0.6984	76.4 %	STRONG (>= 70 %)
MTEB NFCorpus test (249 queries)	0.4609	0.3031	65.8 %	0.3099 / 0.1836	59.3 %	RESEARCH (>= 50 %)

SciFact detail (student -> student): Recall@1 0.5833, Recall@5 0.7767, Recall@10 0.8500. Cross-space: student query -> teacher docs MRR 0.7093; teacher query -> student docs 0.7325. NFCorpus: Recall@1 0.221, Recall@5 0.402, Recall@10 0.498.

Internal validation (1,280 held-out pairs): Student -> Student MRR 0.8498 (teacher 0.9203), Student -> Teacher 0.8512, Teacher -> Student 0.8710.

E4B vs E2B (same recipe, same data): E2B reaches SciFact 79.1 % / NFCorpus 70.8 % of the teacher, E4B 74.4 % / 65.8 %. One candidate was trained (rank 16, layers 24-40, 3 epochs, epoch 2 selected by validation); a wider-LoRA candidate (layers 12-40) was started and stopped unfinished, so no tuning beyond the first run was done for E4B.

Generation is untouched

Regression on this exact export (fresh unmodified google/gemma-4-E4B-it as reference, 6 chat prompts, greedy 24 tokens): token parity 6/6, max logit difference 0.0, embed -> generate -> embed and generate -> embed -> generate identical, 18 alternating switching patterns with 0 mismatches, adapter save/reload identical. The adapter is never merged into the backbone.

Training

Same data and recipe as the E2B model (about 12k public retrieval pairs from 13 sources, two-stage distillation from google/embeddinggemma-2 revision 914f7f89142e33e77833254d9c9b90c3cef7303b; no SciFact/NFCorpus/ArguAna/FiQA/TREC-COVID data). See the E2B model card and its technical note for the full description. Some source datasets carry research-use terms (e.g. MS MARCO, Yahoo Answers).

Important behavior notes

• No BOS token in embed mode. The tokenizer does not prepend <bos> and the adapter was trained that way.
• 128-token training length. Truncate to 128 tokens to reproduce the reported numbers.
• Prefixes: queries task: search result | query: , documents title: none | text: (or title: <title> | text: ). embed() adds the prefix if missing.
• bf16 numerics matter: a few short texts differ between bf16 and fp16 runs (cos about 0.97); this is rounding noise of the bf16 backbone.

Files

File	SHA256
embedding_head.safetensors	FD8007A1798EC2DE881B14A49AD2C74E27C6D04C968490414D4680ECA3E1BBEC
embedding_lora.safetensors	D1AD640E9F159D846AACD8BFEE103B6F0F362C52CE857693AE6B051D164F7E1A

Plus dual_mode_config.json, runtime/ (Python runtime), tokenizer and config files. The Gemma 4 E4B-it weights are loaded from google/gemma-4-E4B-it.

Quickstart

import torch
from runtime.dual_mode_model import Gemma4DualModeForConditionalGeneration

# download this repository (runtime/ + adapter) and keep `runtime` importable
model = Gemma4DualModeForConditionalGeneration.from_pretrained(
    "webmp3/Sakura-Gemma4-E4B-EmbeddingGemma2-DualMode", torch_dtype=torch.bfloat16, device_map="cuda")

# GENERATE (original Gemma 4 path)
msgs = [{"role": "user", "content": "Explain rotary position embeddings in one sentence."}]
inp = model.tokenizer.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt", return_dict=True).to("cuda")
out = model.generate(**inp, max_new_tokens=60, do_sample=False)
print(model.tokenizer.decode(out[0], skip_special_tokens=True))

# EMBED (bidirectional, LoRA on, 768d, L2 normalized)
q = model.embed("How do dense vector representations work?", task="SearchQuery")
d = model.embed("Dense embeddings map text to points in a continuous vector space.", task="Document")
print(float(q @ d))

Limitations

1. About 12k training pairs, one training run; the OOD gap to the teacher (26-34 % MRR) is larger than for E2B.
2. Not index-compatible with EmbeddingGemma 2 vectors.
3. Text only; the vision and audio towers are not part of the embedding path.
4. Runtime memory savings versus a standalone EmbeddingGemma 2 were not measured for E4B.
5. The GGUF version needs a small llama.cpp patch; stock llama.cpp cannot run embed mode.

Citation

@misc{sakura_gemma4_e4b_dualmode_2026,
  author = {webmp},
  title = {Sakura Gemma 4 E4B x EmbeddingGemma 2: DualMode for Generation and Dense Retrieval},
  year = {2026},
  publisher = {Hugging Face},
  howpublished = {\url{https://huggingface.co/webmp3/Sakura-Gemma4-E4B-EmbeddingGemma2-DualMode}}
}

---

中文说明 · 樱花 (Simplified Chinese)

English above. 本节为上文的中文翻译(Sakura = 樱花 yīnghuā);完整的独立中文版见 README_zh.md。

Sakura Gemma 4 E4B x EmbeddingGemma 2 - DualMode

同一个 Gemma 4 E4B-it 主干既用于生成,也用于 EmbeddingGemma-2 蒸馏的 768 维检索。 研究原型。社区模型。并非 Google 官方发布。

30 秒了解

• 是什么: 一个 Gemma 4 E4B-it 模型,既能 聊天,又能为 RAG 生成 768 维的搜索向量,因此无需再在内存中常驻第二个嵌入模型。
• 如何实现: 同一个主干以两种模式运行。Generate(生成)模式就是原封不动的 Gemma 4。Embed(嵌入)模式使用双向注意力、一个小型 LoRA(秩 16)和一个 768 维的头:**增加 4.60 M 参数(8.8 MB)**。
• 质量: 在外部基准测试上,嵌入模式达到教师模型 MRR 的 **74 %(SciFact)和 66 %(NFCorpus)**。在相同配方下,这 低于较小的 E2B 同级模型(79 % / 71 %):在这里,更大的主干并没有带来更好的嵌入。生成结果与原版 Gemma 4 逐位一致。
• 格式: 本仓库(Safetensors 适配器 + Python 运行时)以及一个带有小型 llama.cpp 补丁的 GGUF 版本。
• 坦率的局限: 研究原型。不能直接替代 EmbeddingGemma 2,其向量也与 EmbeddingGemma 2 的索引不兼容。如果你想从这个项目中获得最好的检索效果,请使用 E2B 模型。

定位

• 运行时主干: google/gemma-4-E4B-it(从 Hub 加载,此处不重复提供)。
• 嵌入教师模型: google/embeddinggemma-2,仅用于蒸馏。没有合并,运行时也不需要。
• 与 EmbeddingGemma 2 的向量在索引上不兼容。我们不声称这是同类中的第一个。

它是什么

                        GENERATE MODE  (original Gemma 4 path, adapter fully bypassed)
Input -> Gemma 4 E4B-it backbone --+-- causal attention -> LM head
        (one resident model)       |
                                   +-- EMBED MODE
                                       bidirectional attention
                                       embedding-only LoRA (q_proj, o_proj, layers 24-40, rank 16)
                                       hidden state after layer 40  (hidden_states[41], before the final norm)
                                       mean pooling over valid tokens
                                       Linear 2560 -> 768 (no bias)
                                       L2 normalization   (MRL: 768 / 512 / 256 / 128)

新增参数

组件	形状 / 范围	参数量	BF16
嵌入头	Linear(2560 -> 768, bias=False)	1,966,080	3.75 MB
仅用于嵌入的 LoRA	rank 16, alpha 32, q_proj + o_proj, layers 24-40 (34 modules)	2,637,824	5.03 MB
总开销		4,603,904	8.78 MB

这相当于 E4B-it 检查点所存储的 7.996 B 参数的 +0.0576 %。与 E2B 相同,Gemma 4 E4B 在最后 18 层(24-41)共享 K/V,因此那里的 k_proj/v_proj LoRA 不起作用,而第 41 层位于池化隐藏状态之后;只训练 24-40 层的 q_proj/o_proj。替换独立的 EmbeddingGemma 2 文本模型(271.00 M 参数)在理论上可节省约 266 M 参数(约 508 MB 的 BF16 权重);E4B 的运行时内存 没有测量。

结果

教师模型 EmbeddingGemma 2 与本模型在相同文本上的对比,查询使用 task: search result | query: ,文档使用 title: <title> | text: ,两者最多 128 个 token(训练长度)。这两个基准都从未用于训练或模型选择。

基准(500 篇文档子集)	教师 MRR	本模型 MRR	相对教师的 MRR	nDCG@10 教师 / 本模型	相对教师的 nDCG	门槛
MTEB SciFact test(300 个查询)	0.8971	0.6679	74.4 %	0.9137 / 0.6984	76.4 %	STRONG (>= 70 %)
MTEB NFCorpus test(249 个查询)	0.4609	0.3031	65.8 %	0.3099 / 0.1836	59.3 %	RESEARCH (>= 50 %)

SciFact 详情(学生 -> 学生):Recall@1 0.5833,Recall@5 0.7767,Recall@10 0.8500。跨空间:学生查询 -> 教师文档 MRR 0.7093;教师查询 -> 学生文档 0.7325。NFCorpus:Recall@1 0.221,Recall@5 0.402,Recall@10 0.498。

内部验证(1,280 个留出对):学生 -> 学生 MRR 0.8498(教师 0.9203),学生 -> 教师 0.8512,教师 -> 学生 0.8710。

E4B 对比 E2B(相同配方、相同数据): E2B 达到教师的 SciFact 79.1 % / NFCorpus 70.8 %,E4B 为 74.4 % / 65.8 %。只训练了一个候选(rank 16、24-40 层、3 个 epoch,由验证集选出第 2 个 epoch);更宽的 LoRA 候选(12-40 层)已启动但未完成就停止了,因此对 E4B 没有进行首次运行之外的调优。

生成部分未被改动

在此导出版本上的回归测试(以全新的、未经修改的 google/gemma-4-E4B-it 作为参照,6 个聊天提示,贪心解码 24 个 token):token 一致性 6/6,最大 logit 差异 0.0,embed -> generate -> embed 与 generate -> embed -> generate 均一致,18 种交替切换模式 0 处不一致,适配器保存/重新加载一致。适配器从不合并进主干。

训练

与 E2B 模型使用相同的数据和配方(来自 13 个来源的约 12k 个公开检索对,从 google/embeddinggemma-2 修订版本 914f7f89142e33e77833254d9c9b90c3cef7303b 进行两阶段蒸馏;没有 SciFact/NFCorpus/ArguAna/FiQA/TREC-COVID 的数据)。完整描述请参阅 E2B 模型卡及其技术说明。部分来源数据集带有仅限研究使用的条款(例如 MS MARCO、Yahoo Answers)。

重要的行为说明

• 嵌入模式下没有 BOS token。 分词器不会在前面加 <bos>,适配器也是按这种方式训练的。
• 128 个 token 的训练长度。 截断到 128 个 token 可复现所报告的数值。
• 前缀: 查询 task: search result | query: ,文档 title: none | text: (或 title: <title> | text: )。如果缺少前缀,embed() 会添加。
• bf16 数值很重要:少数短文本在 bf16 与 fp16 运行之间有差异(余弦约 0.97);这是 bf16 主干的舍入噪声。

文件

文件	SHA256
embedding_head.safetensors	FD8007A1798EC2DE881B14A49AD2C74E27C6D04C968490414D4680ECA3E1BBEC
embedding_lora.safetensors	D1AD640E9F159D846AACD8BFEE103B6F0F362C52CE857693AE6B051D164F7E1A

另有 dual_mode_config.json、runtime/(Python 运行时)、分词器和配置文件。Gemma 4 E4B-it 权重从 google/gemma-4-E4B-it 加载。

快速开始

import torch
from runtime.dual_mode_model import Gemma4DualModeForConditionalGeneration

# download this repository (runtime/ + adapter) and keep `runtime` importable
model = Gemma4DualModeForConditionalGeneration.from_pretrained(
    "webmp3/Sakura-Gemma4-E4B-EmbeddingGemma2-DualMode", torch_dtype=torch.bfloat16, device_map="cuda")

# GENERATE (original Gemma 4 path)
msgs = [{"role": "user", "content": "Explain rotary position embeddings in one sentence."}]
inp = model.tokenizer.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt", return_dict=True).to("cuda")
out = model.generate(**inp, max_new_tokens=60, do_sample=False)
print(model.tokenizer.decode(out[0], skip_special_tokens=True))

# EMBED (bidirectional, LoRA on, 768d, L2 normalized)
q = model.embed("How do dense vector representations work?", task="SearchQuery")
d = model.embed("Dense embeddings map text to points in a continuous vector space.", task="Document")
print(float(q @ d))

局限

1. 约 12k 个训练对,只进行了一次训练;与教师模型之间的域外差距(26-34 % 的 MRR)比 E2B 更大。
2. 与 EmbeddingGemma 2 的向量在索引上不兼容。
3. 仅限文本;视觉和音频塔不属于嵌入路径。
4. 相对于独立的 EmbeddingGemma 2,E4B 的运行时内存节省没有测量。
5. GGUF 版本需要一个小型 llama.cpp 补丁;原版 llama.cpp 无法运行嵌入模式。

引用

@misc{sakura_gemma4_e4b_dualmode_2026,
  author = {webmp},
  title = {Sakura Gemma 4 E4B x EmbeddingGemma 2: DualMode for Generation and Dense Retrieval},
  year = {2026},
  publisher = {Hugging Face},
  howpublished = {\url{https://huggingface.co/webmp3/Sakura-Gemma4-E4B-EmbeddingGemma2-DualMode}}
}