Sakura Gemma 4 E2B x EmbeddingGemma 2 - DualMode

One Gemma 4 E2B-it backbone for generation and EmbeddingGemma-2-distilled 768d retrieval. Research prototype. Community model. Not official from Google.

Version: Phase 3 / v0.2 (Phase 2 / v0.1 is kept in the git history of this repository).

In 30 seconds

What: one Gemma 4 E2B-it model that can chat and produce 768-dimensional search vectors for RAG, so no second embedding model has to be kept in memory.
How: the same backbone runs in two modes. Generate is the untouched Gemma 4. Embed uses bidirectional attention, a small LoRA (rank 16) and a 768d head: +3.56 M parameters (6.8 MB).
Quality: on external benchmarks the embed mode reaches 79 % (SciFact) and 71 % (NFCorpus) of the teacher's MRR; generation stays bitwise identical to plain Gemma 4.
Formats: this repository (Safetensors adapter + Python runtime) and a GGUF version (link becomes active when that repository is public) with a small llama.cpp patch.
Honest limit: research prototype. Not a drop-in for EmbeddingGemma 2 and not index-compatible with its vectors.

Positioning

Runtime backbone: google/gemma-4-E2B-it (the single shared model, loaded from the Hub).

Embedding teacher: google/embeddinggemma-2, used only for distillation. It is not merged into any weights and not required at runtime.

This is not a drop-in replacement for google/embeddinggemma-2 and is not index-compatible with vectors produced by it (see cross-space numbers below). Use it as a self-contained retriever: embed queries and documents with this model.

No claim of being the first of its kind is made. In the public implementations reviewed, no other Gemma 4 system was found that exposes both autoregressive generation and EmbeddingGemma-2-distilled dense retrieval from the same resident backbone.

What it is

Local RAG stacks usually keep two text models resident: a generator (Gemma 4) and a separate embedding model. DualMode reuses one Gemma 4 E2B-it backbone for both:

                        GENERATE MODE  (original Gemma 4 path, adapter fully bypassed)
Input -> Gemma 4 E2B-it backbone --+-- causal attention -> LM head
        (one resident model)       |
                                   +-- EMBED MODE
                                       bidirectional attention
                                       embedding-only LoRA (q_proj, o_proj, layers 15-33, rank 16)
                                       hidden state after layer 33  (hidden_states[34], before the final norm)
                                       mean pooling over valid tokens
                                       Linear 1536 -> 768 (no bias)
                                       L2 normalization   (MRL: 768 / 512 / 256 / 128)

Added parameters (Phase 3)

Component	Shape / scope	Parameters	BF16
Embedding head	Linear(1536 -> 768, bias=False)	1,179,648	2.25 MB
Embedding-only LoRA	rank 16, alpha 32, `q_proj` + `o_proj`, layers 15-33 (38 modules)	2,375,680	4.53 MB
Total overhead		3,555,328	6.78 MB

Relative to the stored Gemma 4 E2B-it checkpoint: +0.0694 % of 5.123 B total stored parameters, +0.1575 % of the 2.257 B dense text parameters.

Why only q_proj / o_proj and layers 15-33: Gemma 4 E2B shares K/V across its last 20 layers (layers 15-34 read the K/V of earlier layers), so k_proj/v_proj LoRA in those layers has no effect, and layer 34 lies after the pooled hidden state. The Phase 2 adapter (rank 8, q/k/v/o, layers 27-34) contained such dead modules; Phase 3 trains only modules that change the output.

Replacing a standalone EmbeddingGemma 2 text model (271.00 M parameters, 516.89 MB BF16) by this adapter saves about 267.4 M parameters / 510 MB of BF16 weights (744.37 M / 1,413 MB versus the full standalone checkpoint).

Results (Phase 3)

Everything below is teacher EmbeddingGemma 2 vs. this model on the same texts, queries with task: search result | query: , documents with title: <title> | text: . Both models see at most 128 tokens (the training length), so the teacher numbers are slightly below its published long-context numbers (SciFact teacher at 512 tokens: MRR 0.935).

External out-of-domain retrieval (never used for training or model selection)

Benchmark (500-doc subset)	Teacher MRR	This model MRR	MRR vs teacher	nDCG@10 teacher / this	nDCG vs teacher	Gate
MTEB SciFact test (300 queries)	0.8971	0.7097	79.1 %	0.9137 / 0.7413	81.1 %	STRONG (>= 70 %)
MTEB NFCorpus test (249 queries)	0.4609	0.3262	70.8 %	0.3099 / 0.1906	61.5 %	STRONG on MRR (nDCG 61.5 %)

SciFact detail (this model, student -> student): Recall@1 0.6233, Recall@5 0.8067, Recall@10 0.8733. Cross-space: student query -> teacher docs MRR 0.7190; teacher query -> student docs MRR 0.7517. NFCorpus: Recall@1 0.241, Recall@5 0.418, Recall@10 0.482.

For comparison, Phase 2 (v0.1): SciFact MRR 0.0380 (4.2 % of teacher). The earlier model was over-fitted to 400 training pairs; Phase 3 fixes this with a 12k-pair multi-domain distillation set.

The remaining gap is real: the model keeps roughly 71-79 % of the teacher's MRR, and nDCG@10 on NFCorpus is only about 61 % of the teacher. It is a research-grade retriever, not a teacher replacement.

Internal validation (1,280 held-out pairs from the 13 training sources, ranking against all 1,280 positives)

Direction	MRR	Recall@1	Recall@5
Teacher -> Teacher (upper bound)	0.9203	0.8891	0.9570
Student -> Student	0.8585	0.8109	0.9195
Student -> Teacher	0.8528	0.7953	0.9187
Teacher -> Student	0.8810	0.8391	0.9313

Generation is untouched

Regression run on this exact export (fresh unmodified google/gemma-4-E2B-it as reference, 6 chat prompts, greedy 24 tokens):

Check	Result
Token parity of `generate()` vs. unmodified model	6/6 identical
Max logit difference (last position, full vocabulary)	0.0
`embed -> generate -> embed` embedding difference	0.0
`generate -> embed -> generate` outputs identical	yes
18 alternating switching patterns (embed/generate)	0 embedding mismatches, 0 generation mismatches
Adapter save / reload	embeddings identical (max diff 0.0)
LoRA state after calls	disabled

The adapter is never merged into the backbone; generate() bypasses it completely.

Training (Phase 3)

Teacher: google/embeddinggemma-2 (revision 914f7f89142e33e77833254d9c9b90c3cef7303b), 768d embeddings cached.
Data: 11,954 train pairs + 1,280 validation pairs (query, positive) from public datasets, de-duplicated by query and passage; passages truncated to 900 characters, training length 128 tokens. Sources: Natural Questions, GooAQ, SQuAD, TriviaQA, MS MARCO (BM25 triplets), SPECTER, S2ORC (title-abstract), CodeSearchNet, StackExchange duplicates, Yahoo Answers, ELI5, GermanQuAD, WikiMatrix (German query -> English document). Domains: general knowledge, science, technical Q&A, code, German, cross-lingual, short queries, long documents. No SciFact, NFCorpus, ArguAna, FiQA or TREC-COVID data was used. Licenses follow the source datasets (some, e.g. MS MARCO and Yahoo Answers, carry research-use terms); check them before commercial use.
Stage 1 (epoch 1): representation distillation: cosine + smooth similarity-matrix (MSE) + MRL (512/256/128) losses.
Stage 2 (epochs 2-3): adds in-batch contrastive loss, student-query -> teacher-doc and teacher-query -> student-doc alignment. Source-homogeneous batches make in-batch negatives domain-near.
Model selection: by the internal validation set only. SciFact was logged as a monitor, NFCorpus was only evaluated at the end.
Candidates: rank 8 / layers 27-34 / q,k,v,o (SciFact MRR 0.684), rank 8 / layers 15-34 / q,o (0.699), rank 16 / layers 15-33 / q,o (0.710, chosen).
Diagnosis of v0.1: the 0.038 SciFact result was reproduced; no pipeline bug (masking, pooling, prefixes, padding, ids all verified). The cause was collapse of the embedding space (effective rank 32 vs. 140 for the teacher, mean pairwise cosine 0.91, positive-negative margin 0.027 vs. 0.196).

Important behavior notes

No BOS token in embed mode. The tokenizer in this repository does not prepend <bos> and the adapter was trained that way. (Runtimes that always prepend BOS, such as llama.cpp, must mask it in embed mode.)
128-token training length. Longer inputs work but are untested beyond 512 tokens; truncating to 128 tokens reproduces the reported numbers.
Prefixes: queries task: search result | query: , documents title: none | text: (or title: <title> | text: ). embed() adds the prefix automatically if missing.
bf16 numerics matter: a few short texts differ by cos ~0.97 between bf16 and fp16 reference runs. This is rounding noise of the bf16 backbone, not a bug.

Files

File	SHA256
`embedding_head.safetensors`	`A008DA414E71FBCD41F8D9244D9F633DAD0DF89861B727EE594F832286335B00`
`embedding_lora.safetensors`	`ED890947BB440AC4F0E86D6D099270BF8A52C6E258F9BBBBD53810E98E671FF0`

dual_mode_config.json, runtime/ (Python runtime: dual_mode_model.py, embedding_lora.py), tokenizer and config files. The Gemma 4 E2B-it base weights are loaded from google/gemma-4-E2B-it (pinned base commit 3e22461f65e89153144f8adb70e3b8c2cc9845a7); they are not duplicated here.

Quickstart

import torch
from runtime.dual_mode_model import Gemma4DualModeForConditionalGeneration

# download this repository (runtime/ + adapter) and keep `runtime` importable
model = Gemma4DualModeForConditionalGeneration.from_pretrained(
    "webmp3/Sakura-Gemma4-E2B-EmbeddingGemma2-DualMode", torch_dtype=torch.bfloat16, device_map="cuda")

# GENERATE (original Gemma 4 path)
msgs = [{"role": "user", "content": "Explain rotary position embeddings in one sentence."}]
inp = model.tokenizer.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt", return_dict=True).to("cuda")
out = model.generate(**inp, max_new_tokens=60, do_sample=False)
print(model.tokenizer.decode(out[0], skip_special_tokens=True))

# EMBED (bidirectional, LoRA on, 768d, L2 normalized)
q = model.embed("How do dense vector representations work?", task="SearchQuery")
d = model.embed("Dense embeddings map text to points in a continuous vector space.", task="Document")
print(float(q @ d))
# Matryoshka: model.embed(text, truncate_dim=256)

Limitations

Trained on about 12k pairs for three epochs; broader or longer training should help. The OOD gap to the teacher (21-29 % MRR) is real.
Not index-compatible with EmbeddingGemma 2 vectors; cross-space retrieval is clearly worse than native student -> student retrieval for NFCorpus-like data.
Text only. The vision and audio towers of Gemma 4 are not part of the embedding path.
Memory numbers from v0.1 (AMD Radeon 8060S UMA, PyTorch ROCm: 516 MB VRAM/UMA and 251 MB RSS saved versus a standalone EmbeddingGemma 2 text model) were measured with the smaller Phase 2 adapter; the Phase 3 adapter is 3.4 MB larger in BF16, which is below that measurement's noise.
The GGUF version needs a small llama.cpp patch; stock llama.cpp cannot run embed mode (see the GGUF repository).

Citation

@misc{sakura_gemma4_dualmode_2026,
  author = {webmp},
  title = {Sakura Gemma 4 x EmbeddingGemma 2: DualMode for Generation and Dense Retrieval},
  year = {2026},
  publisher = {Hugging Face},
  howpublished = {\url{https://huggingface.co/webmp3/Sakura-Gemma4-E2B-EmbeddingGemma2-DualMode}}
}

中文说明 · 樱花 (Simplified Chinese)

English above. 本节为上文的中文翻译(Sakura = 樱花 yīnghuā);完整的独立中文版见 README_zh.md。

Sakura Gemma 4 E2B x EmbeddingGemma 2 - DualMode

同一个 Gemma 4 E2B-it 主干既用于生成,也用于 EmbeddingGemma-2 蒸馏的 768 维检索。 研究原型。社区模型。并非 Google 官方发布。

版本:Phase 3 / v0.2(Phase 2 / v0.1 保留在本仓库的 git 历史中)。

30 秒了解

是什么: 一个 Gemma 4 E2B-it 模型,既能聊天,又能为 RAG 生成 768 维的搜索向量,因此无需再在内存中常驻第二个嵌入模型。
如何实现: 同一个主干以两种模式运行。Generate(生成)模式就是原封不动的 Gemma 4。Embed(嵌入)模式使用双向注意力、一个小型 LoRA(秩 16)和一个 768 维的头:**增加 3.56 M 参数(6.8 MB)**。
质量: 在外部基准测试上,嵌入模式达到教师模型 MRR 的 79 %(SciFact)和 71 %(NFCorpus);生成结果与原版 Gemma 4 逐位一致。
格式: 本仓库(Safetensors 适配器 + Python 运行时)以及一个带有小型 llama.cpp 补丁的 GGUF 版本 *(该仓库公开后链接才会生效)*。
坦率的局限: 研究原型。不能直接替代 EmbeddingGemma 2,其向量也与 EmbeddingGemma 2 的索引不兼容。

定位

运行时主干: google/gemma-4-E2B-it(唯一的共享模型,从 Hub 加载)。

嵌入教师模型: google/embeddinggemma-2,仅用于蒸馏。它 没有被合并 进任何权重,运行时也不需要。

这不是 google/embeddinggemma-2 的直接替代品,也不与它所产生的向量在索引上兼容(见下文跨空间数据)。请将其作为独立的检索器使用:用本模型嵌入查询和文档。

我们不声称这是同类中的第一个。在我们查阅的公开实现中,没有发现其他 Gemma 4 系统能够在同一个常驻主干上同时提供自回归生成和 EmbeddingGemma-2 蒸馏的稠密检索。

它是什么

本地 RAG 方案通常会同时常驻两个文本模型:一个生成器(Gemma 4)和一个独立的嵌入模型。DualMode 让一个 Gemma 4 E2B-it 主干同时承担两种用途:

                        GENERATE MODE  (original Gemma 4 path, adapter fully bypassed)
Input -> Gemma 4 E2B-it backbone --+-- causal attention -> LM head
        (one resident model)       |
                                   +-- EMBED MODE
                                       bidirectional attention
                                       embedding-only LoRA (q_proj, o_proj, layers 15-33, rank 16)
                                       hidden state after layer 33  (hidden_states[34], before the final norm)
                                       mean pooling over valid tokens
                                       Linear 1536 -> 768 (no bias)
                                       L2 normalization   (MRL: 768 / 512 / 256 / 128)

新增参数(Phase 3)

组件	形状 / 范围	参数量	BF16
嵌入头	Linear(1536 -> 768, bias=False)	1,179,648	2.25 MB
仅用于嵌入的 LoRA	rank 16, alpha 32, `q_proj` + `o_proj`, layers 15-33 (38 modules)	2,375,680	4.53 MB
总开销		3,555,328	6.78 MB

相对于已存储的 Gemma 4 E2B-it 检查点:占已存储总参数 5.123 B 的 +0.0694 %,占稠密文本参数 2.257 B 的 +0.1575 %。

为什么只用 q_proj / o_proj 和 15-33 层:Gemma 4 E2B 在最后 20 层共享 K/V(15-34 层读取更早层的 K/V),因此这些层中的 k_proj/v_proj LoRA 不起作用,而第 34 层位于池化隐藏状态之后。Phase 2 适配器(rank 8、q/k/v/o、27-34 层)包含这类无效模块;Phase 3 只训练会改变输出的模块。

用这个适配器替换独立的 EmbeddingGemma 2 文本模型(271.00 M 参数,516.89 MB BF16),大约可节省 267.4 M 参数 / 510 MB 的 BF16 权重(相对于完整的独立检查点为 744.37 M / 1,413 MB)。

结果(Phase 3)

下面所有结果都是 教师模型 EmbeddingGemma 2 与本模型在相同文本上的对比,查询使用 task: search result | query: ,文档使用 title: <title> | text: 。两个模型最多看到 128 个 token(训练长度),因此教师模型的数值略低于它已发布的长上下文数值(教师模型在 512 个 token 下的 SciFact:MRR 0.935)。

外部域外检索(从未用于训练或模型选择)

基准(500 篇文档子集)	教师 MRR	本模型 MRR	相对教师的 MRR	nDCG@10 教师 / 本模型	相对教师的 nDCG	门槛
MTEB SciFact test(300 个查询)	0.8971	0.7097	79.1 %	0.9137 / 0.7413	81.1 %	STRONG (>= 70 %)
MTEB NFCorpus test(249 个查询)	0.4609	0.3262	70.8 %	0.3099 / 0.1906	61.5 %	STRONG on MRR (nDCG 61.5 %)

SciFact 详情(本模型,学生 -> 学生):Recall@1 0.6233,Recall@5 0.8067,Recall@10 0.8733。跨空间:学生查询 -> 教师文档 MRR 0.7190;教师查询 -> 学生文档 MRR 0.7517。NFCorpus:Recall@1 0.241,Recall@5 0.418,Recall@10 0.482。

作为对比,Phase 2(v0.1):SciFact MRR 0.0380(教师的 4.2 %)。早期模型在 400 个训练对上过拟合;Phase 3 用一个 12k 对的多领域蒸馏数据集解决了这个问题。

剩余的差距是真实存在的:该模型保留了教师模型约 71-79 % 的 MRR,而 NFCorpus 上的 nDCG@10 只有教师的约 61 %。它是研究级的检索器,不能取代教师模型。

内部验证(来自 13 个训练来源的 1,280 个留出对,对全部 1,280 个正例进行排序)

方向	MRR	Recall@1	Recall@5
教师 -> 教师(上界)	0.9203	0.8891	0.9570
学生 -> 学生	0.8585	0.8109	0.9195
学生 -> 教师	0.8528	0.7953	0.9187
教师 -> 学生	0.8810	0.8391	0.9313

生成部分未被改动

在此导出版本上进行的回归测试(以全新的、未经修改的 google/gemma-4-E2B-it 作为参照,6 个聊天提示,贪心解码 24 个 token):

检查项	结果
`generate()` 与未修改模型的 token 一致性	6/6 完全一致
最大 logit 差异(最后一个位置,整个词表)	0.0
`embed -> generate -> embed` 嵌入差异	0.0
`generate -> embed -> generate` 输出是否一致	是
18 种交替切换模式(embed/generate)	0 个嵌入不一致,0 个生成不一致
适配器保存 / 重新加载	嵌入完全一致(最大差异 0.0)
调用之后的 LoRA 状态	已禁用

适配器从不合并进主干;generate() 完全绕过它。

训练(Phase 3)

教师模型: google/embeddinggemma-2(修订版本 914f7f89142e33e77833254d9c9b90c3cef7303b),缓存 768 维嵌入。
数据: 来自公开数据集的 11,954 个训练对 + 1,280 个验证对(查询、正例),按查询和段落去重;段落截断为 900 个字符,训练长度 128 个 token。来源:Natural Questions、GooAQ、SQuAD、TriviaQA、MS MARCO(BM25 三元组)、SPECTER、S2ORC(标题-摘要)、CodeSearchNet、StackExchange duplicates、Yahoo Answers、ELI5、GermanQuAD、WikiMatrix(德语查询 -> 英语文档)。领域:通用知识、科学、技术问答、代码、德语、跨语言、短查询、长文档。 没有使用 SciFact、NFCorpus、ArguAna、FiQA 或 TREC-COVID 的数据。 许可证遵循各来源数据集(其中一些,例如 MS MARCO 和 Yahoo Answers,带有仅限研究使用的条款);商业使用之前请先核对。
阶段 1(第 1 个 epoch): 表示蒸馏:余弦 + 平滑相似度矩阵(MSE)+ MRL(512/256/128)损失。
阶段 2(第 2-3 个 epoch): 加入批内对比损失,以及学生查询 -> 教师文档和教师查询 -> 学生文档的对齐。来源同质的批次使批内负例在领域上相近。
模型选择: 仅依据内部验证集。SciFact 仅作为监控指标记录,NFCorpus 只在最后评估。
候选: rank 8 / 27-34 层 / q,k,v,o(SciFact MRR 0.684),rank 8 / 15-34 层 / q,o(0.699),**rank 16 / 15-33 层 / q,o(0.710,被选中)**。
对 v0.1 的诊断: 0.038 的 SciFact 结果得到复现;没有流水线错误(掩码、池化、前缀、填充、id 均已核对)。原因是嵌入空间的坍缩(有效秩 32,而教师为 140;平均成对余弦 0.91;正负例间隔 0.027,而教师为 0.196)。

重要的行为说明

嵌入模式下没有 BOS token。 本仓库中的分词器不会在前面加 <bos>,适配器也是按这种方式训练的。(总是在前面加 BOS 的运行时,例如 llama.cpp,必须在嵌入模式下将其屏蔽。)
128 个 token 的训练长度。 更长的输入可以工作,但超过 512 个 token 的情况未经测试;截断到 128 个 token 可复现所报告的数值。
前缀: 查询 task: search result | query: ,文档 title: none | text: (或 title: <title> | text: )。如果缺少前缀,embed() 会自动添加。
bf16 数值很重要:少数短文本在 bf16 与 fp16 参照运行之间的余弦约为 0.97。这是 bf16 主干的舍入噪声,不是错误。

文件

文件	SHA256
`embedding_head.safetensors`	`A008DA414E71FBCD41F8D9244D9F633DAD0DF89861B727EE594F832286335B00`
`embedding_lora.safetensors`	`ED890947BB440AC4F0E86D6D099270BF8A52C6E258F9BBBBD53810E98E671FF0`

dual_mode_config.json、runtime/(Python 运行时:dual_mode_model.py、embedding_lora.py)、分词器和配置文件。Gemma 4 E2B-it 基础权重从 google/gemma-4-E2B-it 加载(固定的基础提交 3e22461f65e89153144f8adb70e3b8c2cc9845a7);此处不重复提供。

快速开始

import torch
from runtime.dual_mode_model import Gemma4DualModeForConditionalGeneration

# download this repository (runtime/ + adapter) and keep `runtime` importable
model = Gemma4DualModeForConditionalGeneration.from_pretrained(
    "webmp3/Sakura-Gemma4-E2B-EmbeddingGemma2-DualMode", torch_dtype=torch.bfloat16, device_map="cuda")

# GENERATE (original Gemma 4 path)
msgs = [{"role": "user", "content": "Explain rotary position embeddings in one sentence."}]
inp = model.tokenizer.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt", return_dict=True).to("cuda")
out = model.generate(**inp, max_new_tokens=60, do_sample=False)
print(model.tokenizer.decode(out[0], skip_special_tokens=True))

# EMBED (bidirectional, LoRA on, 768d, L2 normalized)
q = model.embed("How do dense vector representations work?", task="SearchQuery")
d = model.embed("Dense embeddings map text to points in a continuous vector space.", task="Document")
print(float(q @ d))
# Matryoshka: model.embed(text, truncate_dim=256)

局限

在约 12k 个对上训练了三个 epoch;更广泛或更长时间的训练应该会有帮助。与教师模型之间的域外差距(21-29 % 的 MRR)是真实存在的。
与 EmbeddingGemma 2 的向量在索引上不兼容;对于类似 NFCorpus 的数据,跨空间检索明显不如原生的学生 -> 学生检索。
仅限文本。Gemma 4 的视觉和音频塔不属于嵌入路径。
v0.1 的内存数据(AMD Radeon 8060S UMA,PyTorch ROCm:相比独立的 EmbeddingGemma 2 文本模型节省 516 MB VRAM/UMA 和 251 MB RSS)是用较小的 Phase 2 适配器测得的;Phase 3 适配器在 BF16 下大 3.4 MB,低于该测量的噪声。
GGUF 版本需要一个小型 llama.cpp 补丁;原版 llama.cpp 无法运行嵌入模式(见 GGUF 仓库)。

引用

@misc{sakura_gemma4_dualmode_2026,
  author = {webmp},
  title = {Sakura Gemma 4 x EmbeddingGemma 2: DualMode for Generation and Dense Retrieval},
  year = {2026},
  publisher = {Hugging Face},
  howpublished = {\url{https://huggingface.co/webmp3/Sakura-Gemma4-E2B-EmbeddingGemma2-DualMode}}
}

Downloads last month: 109

Model tree for webmp3/Sakura-Gemma4-E2B-EmbeddingGemma2-DualMode

Base model

google/embeddinggemma-2

Finetuned

(116)

this model