hdae/karume-gemma4-e2b

Gemma 4 E2B Instruct — Karume

What is this

A chat distribution: the text decoder of Gemma 4 E2B Instruct, converted into the WebGPU inference runtime Karume's container format (safetensors files carrying the graph in __metadata__). Runs as-is in the browser and in Deno — string in, string out.

• Tokenizer and chat format ship with the weights. chat() renders the turns, encodes them, samples, and decodes incrementally, so callers hand over messages and read back text fragments as they are decided.
• Per-layer embeddings are a sidecar, not graph weights. They are gathered on the host and handed to the graph as an input, which keeps them out of the GPU's resident set; the loader reads only the vocabulary ranges a conversation actually touches.
• Context: the default capacity is 4096 tokens per conversation, and the loader accepts any capacity between the chunk length in use and 131072 (the model's declared position limit); rotary cos/sin are generated on the host per chunk from the declared rope parameters, so no position table ships. Prefill runs in chunks of 768 rows by default, and the chunk length itself can be raised up to 768 — the traced upper bound of the chunk symbol, which the graph does not carry.
• Not multimodal here. The published checkpoint also carries vision and audio towers; this distribution contains neither, and cannot take images or audio.
• Not readable by transformers (it's a different container with an embedded graph); the reader is a pipeline that implements gemma4/1.
• Exporter used for the conversion: karume/0.8.0. The distribution manifest is karume.json (karume/4).

Base weights and attribution

Converted and quantized from the upstream checkpoint — the original weights are not distributed here.

• e2b: google/gemma-4-E2B-it, licensed Apache 2.0 (license / full text; a verbatim copy is in LICENSE.md).
• Changes made here (also listed in NOTICE.md, per Apache 2.0 §4(b)): the text decoder was extracted and re-expressed in the Karume container format; linear weights were quantized to packed int4 (group 32) and the embedding tables to int8; the per-layer embeddings were moved out of the graph into a host-gathered sidecar; the exit was narrowed to the last row's logits; rotary cos/sin arrive as host-generated inputs. No retraining and no fine-tuning.
• What the model can and cannot do — capabilities, limitations, intended use, evaluations and safety guidance — is documented on the upstream model card, and this repository does not restate it: read google/gemma-4-E2B-it before using it.

Models

Model	Pipeline	Quants	Default quant
e2b (default)	gemma4/1	i4	i4

model selects one of these; omitted, it is e2b. quant defaults to that model's own default quant.

Usage

import { Gemma4Pipeline } from "jsr:@karume/models/gemma";

await using pipeline = await Gemma4Pipeline.fromPretrained({
  repo: "hdae/karume-gemma4-e2b",
  // Pin a commit for reproducible builds — without it you track `main`, and a future
  // repo update (renamed files, new manifest format) may break your app.
  // Copy the full hash from this repo's "Files and versions" tab:
  // revision: "<full commit sha>",
}, {
  // model: "e2b", // default — available: e2b
  // quant: "i4", // default — available: i4
});

const stream = pipeline.chat([
  { role: "user", content: "What is the capital of France?" },
], {
  maxNewTokens: 128,
  // sampler: { temperature: 0 }, // greedy — overrides the recommended default below
});

// Fragments arrive as the decoder settles them (multi-byte characters are held back
// until they are complete).
for await (const text of stream) Deno.stdout.write(new TextEncoder().encode(text));
console.log(await stream.done); // { reason: "eos" | "max-tokens" | "aborted", … }

Messages are plain system / user / assistant turns; tool calls, thinking channels and image or audio parts are rejected rather than silently dropped. With no sampler in the request, generation uses this repository's recommended default: temperature 1.0, top-k 64, top-p 0.95. Weights are fetched once and cached (verified against karume.json's size / sha256).

Model: e2b

Quants

Quant	What it is	Download	Weights	Compute
i4 (default)	Packed int4 linear, int8 embeddings — The only storage series: linear weights in packed int4 (group 32) and the embedding tables in int8, which are not int4-eligible.	3.70 GiB (2.23 GiB of assets, read on the host)	model = i4	—

If no quant is given, it runs as i4 (this model's recommended default). Per-file size and sha256 live in karume.json — verify against that at the fetch layer. Dtype labels use the runtime's storage dtype vocabulary (f16 / i8 / i4), not the fp16 spelling common elsewhere in the ecosystem. A component stored as i4 uses a packed int4 dtype (I4) that is not part of the official safetensors specification — the official safetensors library rejects a file that contains it (checked with 0.8.0). Karume's runtime and exporter read it; files without i4 stay fully compatible.

Generation

Derived from the exported graph and the checkpoint's own generation_config.json, and checked against each other when this repository was assembled.

• context: 4096 tokens per conversation (prompt + generated) by default; any capacity from the chunk length in use up to 131072 can be chosen at load time
• prefill chunk: 768 rows per step by default, up to 768 (the traced upper bound of the chunk symbol)
• recommended sampler: temperature 1.0, top-k 64, top-p 0.95 — used when a request omits sampler; pass { temperature: 0 } for greedy decoding
• stop tokens come from the tokenizer asset in this repository, not from the caller
• device limits: maxBufferSize ≥ 402,653,184 B / maxStorageBufferBindingSize ≥ 402,653,184 B (the largest single tensor must bind)