Instructions to use google/diffusiongemma-26B-A4B-it with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use google/diffusiongemma-26B-A4B-it with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="google/diffusiongemma-26B-A4B-it")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("google/diffusiongemma-26B-A4B-it")
model = AutoModelForMultimodalLM.from_pretrained("google/diffusiongemma-26B-A4B-it")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use google/diffusiongemma-26B-A4B-it with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "google/diffusiongemma-26B-A4B-it"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "google/diffusiongemma-26B-A4B-it",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/google/diffusiongemma-26B-A4B-it

SGLang

How to use google/diffusiongemma-26B-A4B-it with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "google/diffusiongemma-26B-A4B-it" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "google/diffusiongemma-26B-A4B-it",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "google/diffusiongemma-26B-A4B-it" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "google/diffusiongemma-26B-A4B-it",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use google/diffusiongemma-26B-A4B-it with Docker Model Runner:
```
docker model run hf.co/google/diffusiongemma-26B-A4B-it
```

ModuleNotFoundError: Could not import module 'DiffusionGemmaForBlockDiffusion'. Are this object's requirements defined correctly?

by Phase-Technologies - opened about 13 hours ago

Discussion

Phase-Technologies

about 13 hours ago

I've updated to new version of transformers & accelerate and restarted runtime but this error came pls help

RuntimeError Traceback (most recent call last)
/usr/local/lib/python3.12/dist-packages/transformers/utils/import_utils.py in getattr(self, name)
2261 try:
-> 2262 module = self._get_module(self._class_to_module[name])
2263 value = getattr(module, name)

19 frames/usr/local/lib/python3.12/dist-packages/transformers/utils/import_utils.py in _get_module(self, module_name)
2495 except Exception as e:
-> 2496 raise e
2497

/usr/local/lib/python3.12/dist-packages/transformers/utils/import_utils.py in _get_module(self, module_name)
2493 try:
-> 2494 return importlib.import_module("." + module_name, self.name)
2495 except Exception as e:

/usr/lib/python3.12/importlib/init.py in import_module(name, package)
89 level += 1
---> 90 return _bootstrap._gcd_import(name[level:], package, level)
91

/usr/lib/python3.12/importlib/_bootstrap.py in _gcd_import(name, package, level)

/usr/lib/python3.12/importlib/_bootstrap.py in find_and_load(name, import)

/usr/lib/python3.12/importlib/_bootstrap.py in find_and_load_unlocked(name, import)

/usr/lib/python3.12/importlib/_bootstrap.py in _load_unlocked(spec)

/usr/lib/python3.12/importlib/_bootstrap_external.py in exec_module(self, module)

/usr/lib/python3.12/importlib/_bootstrap.py in _call_with_frames_removed(f, *args, **kwds)

/usr/local/lib/python3.12/dist-packages/transformers/models/diffusion_gemma/modeling_diffusion_gemma.py in
33 from ...modeling_flash_attention_utils import FlashAttentionKwargs
---> 34 from ...modeling_layers import GradientCheckpointingLayer
35 from ...modeling_outputs import (

/usr/local/lib/python3.12/dist-packages/transformers/modeling_layers.py in
26 from .models.auto import AutoModel
---> 27 from .processing_utils import Unpack
28 from .utils import TransformersKwargs, auto_docstring, can_return_tuple, logging

/usr/local/lib/python3.12/dist-packages/transformers/processing_utils.py in
39 from .feature_extraction_utils import BatchFeature
---> 40 from .image_utils import ChannelDimension, ImageInput, is_vision_available, make_flat_list_of_images
41 from .tokenization_utils_base import (

/usr/local/lib/python3.12/dist-packages/transformers/image_utils.py in
53 if is_torchvision_available():
---> 54 from torchvision.io import ImageReadMode, decode_image
55 from torchvision.transforms import InterpolationMode

/usr/local/lib/python3.12/dist-packages/torchvision/init.py in
7 from . import extension # usort:skip # noqa: F401
----> 8 from torchvision import _meta_registrations, datasets, io, models, ops, transforms, utils # usort:skip
9

/usr/local/lib/python3.12/dist-packages/torchvision/_meta_registrations.py in
162
--> 163 @torch .library.register_fake("torchvision::nms")
164 def meta_nms(dets, scores, iou_threshold):

/usr/local/lib/python3.12/dist-packages/torch/library.py in register(func)
1084 use_lib = lib
-> 1085 use_lib._register_fake(
1086 op_name, func, _stacklevel=stacklevel + 1, allow_override=allow_override

/usr/local/lib/python3.12/dist-packages/torch/library.py in _register_fake(self, op_name, fn, _stacklevel, allow_override)
201
--> 202 handle = entry.fake_impl.register(
203 func_to_register, source, lib=self, allow_override=allow_override

/usr/local/lib/python3.12/dist-packages/torch/_library/fake_impl.py in register(self, func, source, lib, allow_override)
49 )
---> 50 if torch._C._dispatch_has_kernel_for_dispatch_key(self.qualname, "Meta"):
51 raise RuntimeError(

RuntimeError: operator torchvision::nms does not exist

The above exception was the direct cause of the following exception:

ModuleNotFoundError Traceback (most recent call last)
/tmp/ipykernel_12943/2344719202.py in <cell line: 0>()
----> 1 from transformers import DiffusionGemmaForBlockDiffusion, AutoProcessor
2
3 MODEL_ID = "google/diffusiongemma-26B-A4B-it"
4
5 # Load model

/usr/local/lib/python3.12/dist-packages/transformers/utils/import_utils.py in getattr(self, name)
2348 ) from e
2349 else:
-> 2350 raise ModuleNotFoundError(
2351 f"Could not import module '{name}'. Are this object's requirements defined correctly?"
2352 ) from e

ModuleNotFoundError: Could not import module 'DiffusionGemmaForBlockDiffusion'. Are this object's requirements defined correctly?

NOTE: If your import is failing due to a missing package, you can
manually install dependencies using either !pip or !apt.

To view examples of installing some common dependencies, click the
"Open Examples" button below.

Open Examples

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment

ModuleNotFoundError: Could not import module 'DiffusionGemmaForBlockDiffusion'. Are this object's requirements defined correctly?

To view examples of installing some common dependencies, click the"Open Examples" button below.

To view examples of installing some common dependencies, click the
"Open Examples" button below.