Instructions to use tbtlr/gemma-shield-e2b-trimodal with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use tbtlr/gemma-shield-e2b-trimodal with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="tbtlr/gemma-shield-e2b-trimodal")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("tbtlr/gemma-shield-e2b-trimodal")
model = AutoModelForMultimodalLM.from_pretrained("tbtlr/gemma-shield-e2b-trimodal", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use tbtlr/gemma-shield-e2b-trimodal with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "tbtlr/gemma-shield-e2b-trimodal"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "tbtlr/gemma-shield-e2b-trimodal",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/tbtlr/gemma-shield-e2b-trimodal

SGLang

How to use tbtlr/gemma-shield-e2b-trimodal with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "tbtlr/gemma-shield-e2b-trimodal" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "tbtlr/gemma-shield-e2b-trimodal",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "tbtlr/gemma-shield-e2b-trimodal" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "tbtlr/gemma-shield-e2b-trimodal",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use tbtlr/gemma-shield-e2b-trimodal with Docker Model Runner:
```
docker model run hf.co/tbtlr/gemma-shield-e2b-trimodal
```

gemma-shield-e2b-trimodal

A ShieldGemma-style safety classifier for text, images and spoken requests, fine-tuned from google/gemma-4-E2B-it with LoRA (merged into the base weights here; the bare adapter is tbtlr/gemma-shield-e2b-trimodal-lora). One model answers, for a user prompt, a prompt/response pair, an image, or a speech clip of a user request, whether it violates a given safety policy. Output is the probability of the single token Yes versus No at the first generated position.

Independent fine-tune of Google's Gemma 4 E2B-it. Not a Google release and not affiliated with the ShieldGemma team; "Gemma" is used only to describe the base model.

Policies

Text (ShieldGemma v1 wording): dangerous content, harassment, hate speech, sexually explicit, violence, obscenity, plus a composite "any harm" policy. Image (ShieldGemma 2 wording): sexually explicit, violence/gore, dangerous content, plus a composite. Audio: the text prompt policies applied to a spoken user request (the audio clip replaces the text of the request). The classifier is policy-conditioned: it also follows policy text it was not trained on (tested with OpenAI Moderation's own eight definitions). Exact prompt templates and policy strings are in shield_policies.json.

Usage

from transformers import AutoProcessor, Gemma4ForConditionalGeneration
import torch
m = Gemma4ForConditionalGeneration.from_pretrained("tbtlr/gemma-shield-e2b-trimodal", dtype=torch.bfloat16).cuda().eval()
p = AutoProcessor.from_pretrained("tbtlr/gemma-shield-e2b-trimodal")
yes, no = p.tokenizer.encode("Yes", add_special_tokens=False)[0], p.tokenizer.encode("No", add_special_tokens=False)[0]
# Build the prompt from shield_policies.json (template + policy text), run one forward pass and read
# the first generated position:  logits = m(**inputs, logits_to_keep=1).logits[:, -1]
# p_violation = softmax([logits[yes], logits[no]])[0]
# Audio: 16 kHz mono waveform via processor(text=..., audio=[wave], sampling_rate=16000).

Reference scorers, data mappings and the full evaluation suite live in the project repository (shield/prompt.py, shield/score*.py).

Training

Data: Aegis 2.0, BeaverTails, ToxicChat (benign), WildGuardMix for text; VLGuard, LlavaGuard (SMID), UnsafeBench for images, COCO val2017 as benign image negatives; for audio, 8.3k labelled text prompts rendered with Piper TTS (en_US-libritts_r-medium, 904 speakers; 12.1k prompts, two speakers each, the second render with randomised speed and prosody) plus LibriSpeech as benign speech. Source categories are mapped onto the ShieldGemma policies as positive / ambiguous (dropped) / negative.
Recipe: LoRA r=16, trained continually on a DGX Spark: text (language-model attention and MLP projections, lr 1e-4), then images with text replay (lr 5e-5), then audio with text and image replay (lr 3e-5). All adapters sit on the language model's attention and MLP projections; the vision and audio towers stay frozen (an ablation with extra adapters on the audio tower changed nothing). The audio stage distils the text shield: the target for the Yes token is 0.5 x hard label + 0.5 x the text shield's probability on the clip's transcript, which is where the spoken-request gain comes from (+0.045 F1 over hard labels). Cross-entropy on the answer token, bf16, one epoch per stage.

Evaluation (Optimal F1 / AU-PRC, composite policy unless noted)

Text: ToxicChat 0.760 / 0.824 (ShieldGemma 2B paper: 0.704 / 0.778), Aegis 2.0 test 0.816 / 0.891, BeaverTails 0.840 / 0.916, WildGuardTest 0.790 / 0.871, XSTest 0.883 / 0.956, OpenAI Moderation 0.764 / 0.842. Image, UnsafeBench test: sexual 0.842 / 0.910, dangerous 0.689 / 0.718, violence 0.652 / 0.698, composite 0.704 / 0.778; LlavaGuard test composite 0.683 / 0.676; VLGuard test composite 0.751 / 0.801. Audio: spoken ToxicChat (595 TTS clips, 904 voices) 0.852 / 0.927; JALMBench harmful speech vs LibriSpeech 0.930 / 0.974. Full tables and error analyses: project README.

License and training data

Weights and code: Apache 2.0, the same license as the base model (LICENSE in this repo). The model was trained on the datasets below; two of the text sets are CC BY-NC 4.0. Whether that restriction carries over to model weights is legally unsettled, so commercial users should review the list and make their own assessment.

dataset	role	declared license
Aegis 2.0 (NVIDIA)	text train / eval	CC BY 4.0
BeaverTails	text train / eval	CC BY-NC 4.0
ToxicChat	text train (benign) / eval	CC BY-NC 4.0
WildGuardMix	text train / eval	ODC-BY
XSTest	text eval	CC BY 4.0
OpenAI Moderation eval	text eval	MIT
VLGuard	image train / eval	MIT
LlavaGuard (images from SMID)	image train / eval	Apache 2.0 (SMID: CC BY 4.0 original release)
UnsafeBench	image train / eval	gated data-use agreement (research, education, responsible commercial use)
COCO val2017	image train (benign)	CC BY 4.0 annotations; images under their Flickr licenses
LibriSpeech	audio train (benign) / eval	CC BY 4.0
LibriTTS-R Piper voice (en_US-libritts_r-medium)	TTS for audio train / eval	CC BY 4.0
JALMBench	audio eval	not declared

Limitations

Binary "any unsafe" scores depend on how far a benchmark's taxonomy is from these policies (hate, political, privacy, fraud and misinformation images/text are out of scope). The dangerous-content image policy was trained with a broader reading (weapons, substance abuse, criminal planning, self-harm imagery) than its text counterpart. Audio training used synthetic (TTS) speech only; real recorded speech, accents, noise and non-English audio are untested. On the same spoken prompts, transcribing with Gemma 4 and scoring the transcript with the text policy still scores higher (0.886 F1) than the single-pass audio shield (0.852); the audio pathway is the faster option with a small accuracy cost. English only. Not a substitute for human review in high-stakes settings.

Downloads last month: 39

Safetensors

Model size

5B params

Tensor type

BF16

Model tree for tbtlr/gemma-shield-e2b-trimodal

Base model

google/gemma-4-E2B

Finetuned

google/gemma-4-E2B-it

Finetuned

(422)

this model

tbtlr
/

gemma-shield-e2b-trimodal