Instructions to use h2oai/h2o-lightning-e4b with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use h2oai/h2o-lightning-e4b with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="h2oai/h2o-lightning-e4b")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("h2oai/h2o-lightning-e4b")
model = AutoModelForMultimodalLM.from_pretrained("h2oai/h2o-lightning-e4b", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use h2oai/h2o-lightning-e4b with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "h2oai/h2o-lightning-e4b"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "h2oai/h2o-lightning-e4b",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/h2oai/h2o-lightning-e4b

SGLang

How to use h2oai/h2o-lightning-e4b with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "h2oai/h2o-lightning-e4b" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "h2oai/h2o-lightning-e4b",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "h2oai/h2o-lightning-e4b" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "h2oai/h2o-lightning-e4b",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use h2oai/h2o-lightning-e4b with Docker Model Runner:
```
docker model run hf.co/h2oai/h2o-lightning-e4b
```

Accept the H2O Model Non-Commercial Licence to access this model

This repository is publicly accessible, but you have to accept the conditions to access its files and content.

This model is released under the H2O Model Non-Commercial Licence v1.1 (see the LICENSE file): non-commercial, non-production use only.

H2O Lightning E4B

H2O Lightning E4B is a decision model from H2O.ai, built on google/gemma-4-E4B-it. It answers typed decision questions about a record (a document, a ticket, a policy, a conversation), and about images that come with the record, with one output token per decision and a probability for every option:

choice: picks one of a set of named options (up to 255).
yes/no (noul): gives the probability that a statement is true.
score: gives an ordinal level on a stated scale, with its probability distribution.
box: finds a named target in an image and returns its bounding box, with a confidence and a presence probability. See Box.

This card describes v1.0.1 (main; pin --revision v1.0.1).

How to run it:

Default: our vLLM build, two-way reading. READING=two-way bash h2o-lightning-e4b/serve.sh starts vLLM and the standard-library shim in front of it, which serves the decision API on POST /v1/systemone. See Run it and Reading: one-way and two-way.
Stock vLLM 0.30.0, one-way reading: bash h2o-lightning-e4b/serve.sh. See Reading.
Modes: the server answers in AA mode by default; JevBench and the Decision Index each have their own mode. See Modes.
Cost: one output token per question, so the cost is input tokens only.

This card describes v1.0.1 (main; pin --revision v1.0.1): release v1.0 plus one shim fix. When the image re-read or the reverse-order second read is refused or fails upstream, the request now returns its first read instead of an error. The weights and every other file are unchanged (lightning_31b_shim.py, test_shim.py and engine/package.SHA256SUMS changed).

Run it

1. Download this release. The repository is gated: accept the licence on the model page, then log in.

pip install "huggingface_hub>=0.34"
hf auth login
hf download h2oai/h2o-lightning-e4b --revision v1.0.1 --local-dir h2o-lightning-e4b

The weights are bf16 in two files: model.safetensors and model-base-kvshared.safetensors (the base model's 54 tensors for the layers that share their key/value cache, unchanged), listed in model.safetensors.index.json. Their combined sha256 (the sha256 of the lines <file> <sha256> of the two files, sorted by name, each line ending in a newline) is 92a1882573da69457b650cddca85228231af1f78a2126c75d3ff7e06f039deba.

2. Install vLLM 0.30.0. For two-way reading, install our per-request attention build as run_decision_index.sh does (it pins the build and checks every file against engine/vllm_per_request_attention.SHA256SUMS). Run serve.sh from that environment: the shim uses its Python and its tokenizers package.

3. Start the server on one GPU with at least 44,000 MiB, and wait for its ready line:

READING=two-way bash h2o-lightning-e4b/serve.sh

The context is 131,072 tokens (MAX_LEN), the two-way window 16,384 tokens (TWO_WAY_WINDOW), GPU memory use 0.9 (GPU_UTIL), up to 4 images per request at 560 soft tokens each. For benchmark runs, use the run settings in For benchmark maintainers: each benchmark has its one command, and there is nothing to tune.

4. Ask a question:

curl http://127.0.0.1:8741/v1/systemone -H 'Content-Type: application/json' -d '{
  "state": {"customer": "I was charged twice for order 1182 and nobody has replied.", "plan": "pro"},
  "questions": {
    "route":   {"type": "choice", "instructions": "Which team should handle this?", "criteria": {"billing": "payments and refunds", "shipping": "delivery problems", "technical": "app or website errors"}},
    "angry":   {"type": "noul",   "instructions": "Is the customer upset?"},
    "urgency": {"type": "score",  "instructions": "How urgent is this?", "criteria": ["can wait", "this week", "today", "right now"]}}}'

Modes

Each request is answered in one of three modes. A request names its mode in "mode" (or kwargs.mode); a request without one is answered in the server's default mode, AA unless serve.sh was started with another PROFILE.

Mode	`"mode"`	Server default with	What it does
AA (the default)	`"aa"` or `"auto"`	no `PROFILE` (or `PROFILE=aa`)	the Decision Index mode; a choice question with an explicit "I don't know" option answers it when its top real answer, renormalised without that option, is under 0.75
Decision Index	`"decision_index"`	`PROFILE=di`	every record read whole when it fits the context (a longer one is cut down to the parts most relevant to the question, BM25); one read per text question; an image answer under 0.8 read again at 1,120 soft tokens
JevBench	`"jevbench"`	`PROFILE=jevbench`	one read per question; a record over 3,000 tokens cut down to the parts most relevant to the question (BM25)

Each response says its mode in effort_used.mode; /health lists the modes. Any other "mode" value is an HTTP 400.

Reading: one-way and two-way

The model was trained to answer under two readings: one-way (causal, as a text generator reads) and two-way (its full-attention layers read the prompt in both directions). READING=two-way serves both from one engine, chosen per request: a request is read two-way unless it asks for one-way ("kwargs": {"attention_mode": "one-way"}), its record, as the request's mode reads it, is over the two-way window, or the two-way read is refused for the context; those are read one-way. There is no latency cap: every request that fits the two-way window is read two-way, however many questions it asks. A request with a Box question is always read one-way. Without READING=two-way, serve.sh serves one-way only, on stock vLLM, and a request that asks for two-way is refused (HTTP 400). Each reading has its own fitted values (serve_config.json one-way, serve_config_two_way.json two-way); every response says which reading answered it in effort_used.attention_mode.

Two-way is the default because the model was trained for it and it matches the rest of the H2O Lightning family. On JevBench's 231 public items, two-way answers 203 correctly and one-way 204. The one-way reading runs on stock vLLM.

Box

curl http://127.0.0.1:8741/v1/systemone -H 'Content-Type: application/json' -d '{
  "state": {"photo": "data:image/png;base64,..."},
  "questions": {"mug": {"type": "box", "instructions": "the coffee mug in `photo`"}}}'

The answer's box holds top, bottom, left and right as fractions of the image from its top-left corner; confidence is a yes/no reading of the image with the box drawn on it (does the box tightly contain the target?), and present a yes/no reading of whether the target is in the image at all. On RefCOCOg val, 600 referring expressions (board COCO images excluded): mean IoU 0.6907, Acc@0.5 73.2 %, confidence calibration error 0.0611 against IoU >= 0.5.

Hardware

Measured on one NVIDIA L40S (48 GB) in bf16 on 2026-11-01, on JevBench-sized requests (the 231 public JevBench items' requests, cycled), server defaults: 47.62 requests per second at 32 in flight and a median of 24.5 ms one request at a time, two-way. Sizes for 80 GB and 96 GB GPUs are not measured: set MAX_LEN, TWO_WAY_WINDOW and GPU_UTIL yourself there.

For benchmark maintainers

Each benchmark below has one way to run it, the way our numbers were measured. Use it as written.

Benchmark	Server command	Mode
JevBench and ImageJevBench (including its Box track)	`READING=two-way PROFILE=jevbench bash h2o-lightning-e4b/serve.sh`	JevBench
Decision Index	`PROFILE=di bash h2o-lightning-e4b/serve.sh`	Decision Index
Artificial Analysis Decision Index	`READING=two-way bash h2o-lightning-e4b/serve.sh`	AA (as shipped)

For benchmark maintainers: JevBench and ImageJevBench

Run it this way for our measured score: two-way reading in the JevBench mode (see Modes). On a server that was started without PROFILE=jevbench (for example a hosted endpoint), send "mode": "jevbench" with each request instead; the answers are the same.

READING=two-way PROFILE=jevbench bash h2o-lightning-e4b/serve.sh

cd <JEVBENCH> && python3 -m jevbench.cli run \
  --tasks datasets/public/easy.jsonl,datasets/public/original.jsonl,datasets/public/hard.jsonl \
  --adapter typesafe --endpoint http://127.0.0.1:8741 --key-env '' --model h2oai/h2o-lightning-e4b \
  --cost-basis self_hosted_gpu --reserve-usd 0 --results <OUT>/results.jsonl --raw-dir <OUT>/raw \
  --ledger <OUT>/ledger.jsonl --manifest <OUT>/manifest.json --run-label h2oai/h2o-lightning-e4b --delay-s 0

Expected result: 203 / 231 correct.

Do not change: the temperatures, yes/no floor and other values in serve_config.json and serve_config_two_way.json (as shipped); the engine settings serve.sh sets.

For benchmark maintainers: ImageJevBench Box track

Ask a Box question through the package's decision API (the request in Box), never as a chat prompt: the box comes from a guided generation that serve.sh sets up, and its confidence is not a number the model writes but a yes/no reading the shim makes. Serve in the JevBench mode, as for JevBench (a Box question is answered the same way in every mode). Read answers.<id>.box, answers.<id>.confidence (score boxes with this) and answers.<id>.present (the probability that the target is in the image; not the box's score).

For benchmark maintainers: Decision Index

Run it this way for our measured score: the Decision Index profile, PROFILE=di (the shim's measured profile: its engine and its settings, checked at start). On a hosted server, send "mode": "decision_index" with each request.

PROFILE=di bash h2o-lightning-e4b/serve.sh

Then the kit (edition 0.3, the official scorer), against http://127.0.0.1:8741 with --option model=h2oai/h2o-lightning-e4b, text and vision, as in its own instructions.

Expected result: text index 59.37 and public vision index 59.83.

For benchmark maintainers: Artificial Analysis Decision Index

Run it this way: serve.sh as shipped, two-way: its default AA mode is the Decision Index mode plus the "I don't know" rule (0.75). Send the benchmark's requests unchanged.

READING=two-way bash h2o-lightning-e4b/serve.sh

Optional settings (not for benchmark runs)

FAST_PREFILL=0: turns off vLLM's fast prefill for the layers that share their key/value cache (on by default; answers move by at most a few hundredths of a probability).
GRAMMAR_BACKEND=guidance: llguidance instead of xgrammar for the box generation (not measured on this model).
MAX_LEN, TWO_WAY_WINDOW, GPU_UTIL: the context, the two-way window and the share of GPU memory.

Measured

	H2O Lightning E4B
JevBench's 231 public items, JevBench's own CLI, two-way, JevBench mode: correct	203 / 231
Decision Index 0.3 public text index (the kit's own scorer), Decision Index mode	59.37
Decision Index 0.3 public vision index, Decision Index mode	59.83
calibration error of the top option, text rows, 10 equal-width bins (lower is better)	0.0448

Served values: temperatures choice 1, score 2.84, yes/no 1 one-way and choice 1, score 1.24, yes/no 1 two-way; image temperature 1.11 one-way and 1.11 two-way; yes/no floor 0 one-way and 0 two-way. The Decision Index rows are our own runs of the public 0.3 kit, not a leaderboard result.

Limitations

Tested on NVIDIA L40S (48 GB) GPUs only (see Hardware).
Two-way reading needs our per-request attention build of vLLM; on stock vLLM, serve one-way.
Decisions only: the model is trained to answer typed decision questions, not to write free text.

License

Non-commercial. See LICENSE and NOTICE.

Based on google/gemma-4-E4B-it (Apache-2.0, Copyright Google); the Apache-2.0 text is in LICENSE-APACHE-2.0.

The portions of the shim taken from Cygnet are under the MIT License in LICENSE-cygnet; the header layout's sources are in LICENSE-flexdm and LICENSE-wcole3.