Instructions to use h2oai/h2o-lightning-bolt-74b with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use h2oai/h2o-lightning-bolt-74b with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="h2oai/h2o-lightning-bolt-74b", trust_remote_code=True)
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoModelForMultimodalLM
model = AutoModelForMultimodalLM.from_pretrained("h2oai/h2o-lightning-bolt-74b", trust_remote_code=True, device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use h2oai/h2o-lightning-bolt-74b with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "h2oai/h2o-lightning-bolt-74b"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "h2oai/h2o-lightning-bolt-74b",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/h2oai/h2o-lightning-bolt-74b

SGLang

How to use h2oai/h2o-lightning-bolt-74b with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "h2oai/h2o-lightning-bolt-74b" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "h2oai/h2o-lightning-bolt-74b",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "h2oai/h2o-lightning-bolt-74b" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "h2oai/h2o-lightning-bolt-74b",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use h2oai/h2o-lightning-bolt-74b with Docker Model Runner:
```
docker model run hf.co/h2oai/h2o-lightning-bolt-74b
```

Accept the H2O Model Non-Commercial Licence to access this model

This repository is publicly accessible, but you have to accept the conditions to access its files and content.

This model is released under the H2O Model Non-Commercial Licence v1.1 (see the LICENSE file): non-commercial, non-production use only.

H2O-Lightning-Bolt-74B

H2O-Lightning-Bolt-74B is a 74B-parameter (9B active) decision model from H2O.ai, built on nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB. It answers typed decision questions about a record (a document, a ticket, a policy, a conversation), and about images that come with the record. It returns a probability for every option:

choice: picks one of a set of named options.
yes/no (noul): gives the probability that a statement is true.
score: gives an ordinal level on a stated scale, with its probability distribution.
box: finds a named target in an image and returns its bounding box, with a calibrated confidence and a presence probability. See Box.

A choice question may have up to 255 options.

This card describes v1.1 (main; pin --revision v1.1).

How to run it:

vLLM 0.32.0rc1.dev100+g77154ca61 with b12x. bash h2o-lightning-bolt-74b/serve.sh starts vLLM on the merged weights and the standard-library shim (lightning_bolt_shim.py) in front of it, which serves the decision API on POST /v1/systemone (it ignores a request's "model"), and the OpenAI Decisions form on POST /v1/decisions, which requires "model": "h2oai/h2o-lightning-bolt-74b" (or "h2o-lightning-bolt-74b"). It reads two-way by default; see Run it and Reading: one-way and two-way.
Hardware: one NVIDIA RTX PRO 6000 (96 GB, Blackwell); the 2-bit (IQ2_XXS) routed experts run through b12x's kernels with bf16 activations. See Hardware.
Modes: the server answers in AA mode by default (every record read whole, every text question read twice, and the "I don't know" rule); a request may choose another mode. See Modes.
Cost: one output token per question, so the cost is input tokens only (in AA mode each text question's input is read twice).

This card describes v1.1 (main; pin --revision v1.1), checkpoint step 30000: the start-up check confirms the two-way reading on the model's own answers (see Start-up check); a start that cannot confirm it serves every request one-way and reports it; an optional extra read that fails returns the first read. The weights are those of v1.0.1.

Run it

The weights are 32.2 GiB: the base model's routed experts in NVIDIA's IQ2_XXS format, byte for byte, and its dense modules in bf16, with our fine-tune folded into the ones it trained. vLLM reads them with b12x's kernels for the 2-bit experts.

1. Download this release. The repository is gated: accept the licence on the model page, then log in.

pip install "huggingface_hub>=0.34"
hf auth login
hf download h2oai/h2o-lightning-bolt-74b --revision v1.1 --local-dir h2o-lightning-bolt-74b

2. Install vLLM 0.32.0rc1.dev100+g77154ca61 and b12x 1.5.1 in a fresh virtual environment, and run serve.sh from it (the shim uses that environment's Python and its tokenizers package; box answers also need Pillow).

3. Start the server:

bash h2o-lightning-bolt-74b/serve.sh

serve.sh starts vLLM with the model's head in float32, at both levels of the config, since this model builds its head from llm_config (--hf-overrides '{"head_dtype": "float32", "llm_config": {"head_dtype": "float32"}}'): vLLM runs the output head in the model's dtype by default, and bf16 logits round the option probabilities before the softmax. It waits until the shim answers, then prints H2O-Lightning-Bolt-74B ready on http://127.0.0.1:8741/v1/systemone. Its context is 131,072 tokens (MAX_LEN), and it answers in AA mode (below) unless a request asks for another mode. For benchmark runs, use the run settings in For benchmark maintainers: each benchmark has its one command, and there is nothing to tune.

4. Ask a question:

curl http://127.0.0.1:8741/v1/systemone -H 'Content-Type: application/json' -d '{
  "state": {"customer": "I was charged twice for order 1182 and nobody has replied.", "plan": "pro"},
  "questions": {
    "route":   {"type": "choice", "instructions": "Which team should handle this?", "criteria": {"billing": "payments and refunds", "shipping": "delivery problems", "technical": "app or website errors"}},
    "angry":   {"type": "noul",   "instructions": "Is the customer upset?"},
    "urgency": {"type": "score",  "instructions": "How urgent is this?", "criteria": ["can wait", "this week", "today", "right now"]}}}'

Images go in a top-level images array of data URLs ("images": ["data:image/png;base64,..."]).

Modes

Each request is answered in one of three modes. A request names its mode in "mode" (or kwargs.mode); a request without one is answered in the server's default mode, AA unless serve.sh was started with another PROFILE.

Mode	`"mode"`	Server default with	What it does
AA (the default)	`"aa"` or `"auto"`	no `PROFILE` (or `PROFILE=aa`)	the Decision Index mode, plus the "I don't know" rule: a choice question with an explicit "I don't know" option answers it when its top real answer, renormalised without that option, is under 0.75
Decision Index	`"decision_index"`	`PROFILE=di`	every record read whole up to the context (131,072 tokens less 4,096 for the question; only a longer one is cut down to the parts most relevant to the question, BM25), and every text question read a second time with the input written twice, the answer the mean of the two readings; images read once
JevBench	`"jevbench"`	`PROFILE=jevbench`	one read per question, and a record over 3,000 tokens cut down to the parts most relevant to the question (BM25)

Each response says its mode in effort_used.mode (used, and whether it came from the request or the server default); /health lists the modes. Any other "mode" value is an HTTP 400.

Box

curl http://127.0.0.1:8741/v1/systemone -H 'Content-Type: application/json' -d '{
  "state": {"photo": "data:image/png;base64,..."},
  "questions": {"mug": {"type": "box", "instructions": "the coffee mug in `photo`"}}}'

The answer's box holds top, bottom, left and right as fractions of the image from its top-left corner. The model finds the target in one guided generation; confidence is a separate yes/no reading of the image with the box drawn on it (does the box tightly contain the target?), and present a yes/no reading of whether the target is in the image at all.

Reading: one-way and two-way

The model was trained to answer under two readings of its 8 attention layers (its Mamba layers always read left to right): one-way (causal, as a text generator reads) and two-way (those attention layers read the prompt in both directions, so the answer position sees the record, the question and the options at once). serve.sh serves two-way by default (READING=two-way).

bash h2o-lightning-bolt-74b/serve.sh (the default, the same as READING=two-way) serves both readings from one engine, chosen per request: a request is read two-way unless it asks for one-way ("kwargs": {"attention_mode": "one-way"}), its prompt is over the two-way window (TWO_WAY_WINDOW, default 90,112 tokens), or the two-way read is refused for the context; those are read one-way, as a one-way server reads them (the record as the request's mode handles it). There is no latency cap: every request that fits the two-way window is read two-way, however many questions it asks. A request with a Box question is always read one-way. It needs our per-request attention build of vLLM: serve.sh lays the pinned port files (vllm_port/, checked by sha256) over the installed vLLM, which must be exactly the version in vllm_port/VLLM_VERSION, in a cache folder of its own; nothing is installed or changed in your environment.
READING=one-way bash h2o-lightning-bolt-74b/serve.sh serves the one-way reading only, on stock vLLM; a request that asks for two-way is refused (HTTP 400).

Each reading has its own fitted temperatures (serve_config.json one-way, serve_config_two_way.json two-way). Every response says which reading answered it in effort_used.attention_mode; with READING=two-way it also says why in effort_used.attention_reason (requested, record over the two-way window, two-way refused the context, box question).

Start-up check

Before its first answer a READING=two-way server checks that the engine honours the two-way flag: it reads the reference probe questions (serve_config_two_way.json mode_probe) flagged and unflagged, twice each, in three rounds, and serves two-way when, in at least two rounds, the flagged answers move away from the one-way answers by at least 0.1 in probability and by at least four times the engine's own repeat spread, each reading staying nearer its own reference. Otherwise it serves every request one-way and says so in /health (two_way_check.verified false, with the reason) and in its log; it never refuses a request for this. A prompt-log-probability check runs beside it and is reported in /health only. Whether the one-way reading looks only at earlier text is checked once per release, against stock vLLM on the same weights. In our tests 14 of 14 fresh starts on one RTX PRO 6000 verified two-way.

Hardware

Measured on one NVIDIA RTX PRO 6000 (96 GB, Blackwell) on 2026-11-01: 5.62 prompts per second at 64 sequences in flight, and a median of 314.2 ms one request at a time. Read once per question (the JevBench mode) and compared with the trainer's forward pass on 200 prompts (the 90th percentile of each prompt's largest probability difference): the two-way reading, the default, gives 0.040 to 0.050 across three fresh engine starts, at the 0.05 parity gate (within it on two starts, 0.0502 on the third); its top answer agrees on 196 to 198 of 200, and on every prompt where the trainer's top answer is decisive. The one-way reading gives 0.053 to 0.069 (0.064 to 0.069 on the default server, 0.053 to 0.063 with READING=one-way on stock vLLM), outside the 0.05 parity gate; its top answer agrees on 193 to 197 of 200, and on 98.9 to 100 % of the decisive prompts. The AA and Decision Index modes read each text question twice by design, so their probabilities differ from a single forward pass. The two-way reading's top answers agreed with an independent two-way read on 99.5 % of text questions and 98.2 % of image questions.

For benchmark maintainers

Each benchmark below has one way to run it, the way our numbers were measured. Use it as written.

Benchmark	Server command	Mode
JevBench and ImageJevBench (including its Box track)	`PROFILE=jevbench bash h2o-lightning-bolt-74b/serve.sh`	JevBench
Decision Index	`PROFILE=di bash h2o-lightning-bolt-74b/serve.sh`	Decision Index
Artificial Analysis Decision Index	`bash h2o-lightning-bolt-74b/serve.sh`	AA (as shipped)

For benchmark maintainers: JevBench and ImageJevBench

Run it this way for our measured score: the JevBench mode, PROFILE=jevbench (see Modes): one read per question, the shipped calibration, images and Box questions on (there is no setting to turn them off). On a server that was started without it (for example a hosted endpoint), send "mode": "jevbench" with each request instead; the answers are the same.

1. Install and download as in Run it.

2. Serve and wait for the ready line (H2O-Lightning-Bolt-74B ready on http://127.0.0.1:8741/v1/systemone); until then /health answers HTTP 503:

PROFILE=jevbench bash h2o-lightning-bolt-74b/serve.sh

3. Run JevBench's harness (commit bb05a33) against it, from its checkout:

cd <JEVBENCH> && python3 -m jevbench.cli run   --tasks datasets/public/easy.jsonl,datasets/public/original.jsonl,datasets/public/hard.jsonl   --adapter typesafe --endpoint http://127.0.0.1:8741 --key-env '' --model h2oai/h2o-lightning-bolt-74b   --cost-basis self_hosted_gpu --reserve-usd 0 --results <OUT>/results.jsonl --raw-dir <OUT>/raw   --ledger <OUT>/ledger.jsonl --manifest <OUT>/manifest.json --run-label h2oai/h2o-lightning-bolt-74b --delay-s 0

Expected result: about 218 / 231 correct (215-218 across seven fresh engine starts); a setup that gives clearly less is not the configuration above.

Images (ImageJevBench). The same server answers image questions with no other setting: send each image as a data URL, unchanged, in the record ("state": {"photo": "data:image/jpeg;base64,..."}) or in the top-level images array, and name it in the question. Box questions go to the same server as below.

Do not change: the temperatures, yes/no floor and box calibration in serve_config.json (as shipped); the float32 head and the other engine settings serve.sh sets.

For benchmark maintainers: ImageJevBench Box track

Ask a Box question through the package's decision API, never as a chat prompt. The box is found by a guided generation that serve.sh sets up (the coordinates are constrained to the trained format), and its confidence is not a number the model writes: the shim computes it from a yes/no reading of the image with the box drawn on it, mapped to a calibrated probability by the shipped box.confidence_calibration in serve_config.json. A harness that sends a chat prompt to vLLM directly, or reads a self-stated "confidence" from generated text, gets neither the guided box nor the calibrated confidence.

1. Serve in the JevBench mode, as for JevBench (a Box question is answered the same way in every mode):

PROFILE=jevbench bash h2o-lightning-bolt-74b/serve.sh          # serves http://127.0.0.1:8741

2. One request per item: the image in the record, unchanged, and one box question that names it:

curl -s http://127.0.0.1:8741/v1/systemone -H 'Content-Type: application/json' -d '{
  "state": {"photo": "data:image/jpeg;base64,<the image, unchanged>"},
  "questions": {"where": {"type": "box", "instructions": "the leftmost dog in `photo`"}}}'

3. Read the answer:

answers.where.box: left, top, right, bottom, fractions of the image from its top-left corner;
answers.where.confidence: the calibrated probability that the box overlaps the target with IoU >= 0.5 (score boxes with this);
answers.where.present: the probability that the named target is in the image at all; it is not the box's score;
if answers.where.found is false, the box is the whole image and the confidence is 0.

Do not change: VLLM_USE_FLASHINFER_SAMPLER (serve.sh sets it; the guided generation goes through the sampler); box.confidence_calibration in serve_config.json.

For benchmark maintainers: Decision Index

Run it this way for our measured score: the Decision Index profile, PROFILE=di. Every record is read whole up to the context (131,072 tokens less 4,096 for the question; only a longer record is cut down by the BM25 selection), and every text question is read a second time with the input written twice, its answer the mean of the two readings. Images are read once. Each response says so in effort_used.reread (questions: how many of the request's questions were read again) and effort_used.passes_per_decision (2 when a question was read again). On a server that was started without it (for example a hosted endpoint), send "mode": "decision_index" with each request instead.

1. Serve with the profile and wait for the ready line:

PROFILE=di bash h2o-lightning-bolt-74b/serve.sh

For a benchmark run, check /health: two_way_check.verified must be true. If a start serves one-way, restart the server (each start runs the check again); in our tests every fresh start on one RTX PRO 6000 verified two-way (14 of 14).

2. Run the kit (edition 0.3, the official scorer; Python 3.11 or later) in its own environment:

python3 -m venv kit && kit/bin/pip install "decision-index[vision] @ git+https://github.com/apolinario/decision-index@f8552a13b643d39f61185f5951bb5fed16bedf51" pillow==13.0.0 numpy
kit/bin/decision-index suite download --edition 0.3 --dir suite-0.3
kit/bin/decision-index run --edition 0.3 --suite-dir suite-0.3 --engine http --option base_url=http://127.0.0.1:8741 --option model=h2oai/h2o-lightning-bolt-74b --out runs/text
kit/bin/decision-index score --edition 0.3 --suite-dir suite-0.3 --results runs/text/results.jsonl --engine http --out runs/text/score
kit/bin/decision-index vision run --dir WORK/vision --engine decide --option base_url=http://127.0.0.1:8741 --option path=/v1/systemone --option protocol=decisions --option model=h2oai/h2o-lightning-bolt-74b --option max_pixels=1638400 --workers 16 --out runs/vision
kit/bin/decision-index vision score --dir WORK/vision --results runs/vision/results.jsonl --out runs/vision/score

Both halves are served with PROFILE=di and serve.sh's defaults (MAX_LEN 131072, GPU_UTIL 0.90) on one NVIDIA RTX PRO 6000 (96 GB) per server; images are read once under the profile.
The text run in one process is slow. Ours was split as a speed-up only: the kit's rows (unchanged) into shards, each read by its own decision-index run --rows <shard>.jsonl against the servers, the untouched results concatenated and scored by decision-index score as above.

Expected result: text index 65.51 and public vision index 68.96.

Do not change: the profile's settings (SHIM_REREAD_*, SHIM_MAX_STATE_TOKENS); serve_config.json.

For benchmark maintainers: Artificial Analysis Decision Index

Run it this way: serve.sh as shipped: its default AA mode is the Decision Index mode plus one rule for questions that offer "I don't know": when a choice question has an explicit "I don't know" option (an option whose text is "I don't know", "Not sure" or one of the other phrasings in serve_config.json idk.labels), and the model's top real answer, with the probabilities renormalised over the real options, is under 0.75, the answer is "I don't know". That option then carries probability 0.75 and is the most likely option; the real options share the rest. Yes/no and score questions, and choice questions without such an option, are answered exactly as in the Decision Index mode. Each such answer carries idk (the real options' renormalised probabilities, top_real, and abstained), and each response effort_used.idk (offered: questions with the option; questions: how many it answered "I don't know").

bash h2o-lightning-bolt-74b/serve.sh

Send the benchmark's requests unchanged; the rule needs no request option.

Optional settings (not for benchmark runs)

PREFIX_CACHE=1: vLLM's prefix cache, so several questions about one record share its computation.
MAX_LEN, GPU_UTIL, MAX_NUM_SEQS: the context, the share of GPU memory and the sequences in flight.

Measured

	H2O-Lightning-Bolt-74B
JevBench's 231 public items, JevBench's own CLI: correct	218 / 231
Decision Index 0.3 public text index (the kit's own scorer)	65.51
Decision Index 0.3 public vision index	68.96
calibration error of the top option, text rows, 10 equal-width bins (lower is better)	0.0416

JevBench is read in the JevBench mode (PROFILE=jevbench); the Decision Index rows with PROFILE=di, as in For benchmark maintainers. The Decision Index rows are our own runs of the public 0.3 kit, not a leaderboard result.

Limitations

Two-way reading needs the per-request attention build that serve.sh lays over the pinned vLLM version (see Reading: one-way and two-way); on any other vLLM, set READING=one-way.
Two-way reading (the default) against one-way, on our own held-out development set: the set as a whole +0.60 [−0.51, +1.75] (95 % interval); its tool-use questions −0.85 [−1.70, 0.00], at the edge of noise. For tool-choice decisions, "kwargs": {"attention_mode": "one-way"} reads a request one-way.
Tested on NVIDIA Blackwell GPUs only (see Hardware).
Decisions only: the model is trained to answer typed decision questions, not to write free text.

License

Non-commercial. See LICENSE.

Based on nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB (revision c6cc2e8ffa353403213d6b9b18aaf8eb684d705a) by NVIDIA, which is under the OpenMDW License Agreement 1.1 (BASE_LICENSE); NVIDIA's notices are kept in NOTICE.

The portions of the shim taken from Cygnet are under the MIT License in LICENSE-cygnet.