Instructions to use newmes/medgemma-ae-detection with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use newmes/medgemma-ae-detection with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="newmes/medgemma-ae-detection")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("newmes/medgemma-ae-detection")
model = AutoModelForMultimodalLM.from_pretrained("newmes/medgemma-ae-detection", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use newmes/medgemma-ae-detection with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "newmes/medgemma-ae-detection"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "newmes/medgemma-ae-detection",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/newmes/medgemma-ae-detection

SGLang

How to use newmes/medgemma-ae-detection with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "newmes/medgemma-ae-detection" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "newmes/medgemma-ae-detection",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "newmes/medgemma-ae-detection" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "newmes/medgemma-ae-detection",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use newmes/medgemma-ae-detection with Docker Model Runner:
```
docker model run hf.co/newmes/medgemma-ae-detection
```

MedGemma AE Detection

MedGemma 4B fine-tuned to grade visible adverse events (AEs) from patient photographs. Part of CLARA, NEWMES AI team's submission to the MedGemma Impact Challenge (Kaggle, Feb 2026).


Task	Classify a skin/face photo into 21 classes = normal + 7 AE types × CTCAE grade 1–3
AE types	maculopapular rash, acneiform rash, periorbital edema, SJS prodrome, stomatitis, pruritus, alopecia
Initialised from	`newmes/medgemma-4b-antihallu` (MedGemma 4B + RLFR anti-hallucination)
Method	LoRA fine-tuning, 50 epochs, adapters merged into the base weights (bf16, 8.6 GB)
Training data	210 Gemini-generated synthetic patient images, 147 / 21 / 42 train / val / test (dataset → `ae_images/`)
Role in CLARA	Data Collection Agent: compares today's CLARA Call video frame with the patient's baseline photo and reports new AEs with a CTCAE grade

Results

Held-out test set: 42 synthetic images (2 per class). Exact-match accuracy on the 21-class label (AE type and grade).

Model	Accuracy	Correct
MedGemma 4B zero-shot	28.6 %	12 / 42
MedGemma 4B fine-tuned (this model)	85.7 %	36 / 42
MedSigLIP-448 zero-shot	28.6 %	12 / 42
MedSigLIP-448 + linear probe	61.9 %	26 / 42

Remaining errors are mostly off-by-one grade (e.g. pruritus G3 → G2) or normal ↔ grade 1 confusion. The full comparison, per-AE detection rates and a before/after gallery are in notebooks/2_medgemma+medsiglip+HeAR_SAE-detection.ipynb.

Usage

import torch
from PIL import Image
from transformers import AutoModelForImageTextToText, AutoProcessor

repo = "newmes/medgemma-ae-detection"
model = AutoModelForImageTextToText.from_pretrained(repo, torch_dtype=torch.bfloat16, device_map="auto")
processor = AutoProcessor.from_pretrained(repo)

baseline = Image.open("baseline.jpg")   # pre-treatment photo
current  = Image.open("today.jpg")      # today's check-in frame

messages = [{
    "role": "user",
    "content": [
        {"type": "image", "image": baseline},
        {"type": "image", "image": current},
        {"type": "text", "text":
         "Image 1 is the patient's baseline photo, Image 2 was taken today. "
         "Identify any new adverse event visible in Image 2 that is NOT present in Image 1 "
         "and grade it with CTCAE v5. Answer with one of: normal, or <ae_type> grade <1-3>. "
         "AE types: maculopapular rash, acneiform rash, periorbital edema, SJS prodrome, "
         "stomatitis, pruritus, alopecia."},
    ],
}]

inputs = processor.apply_chat_template(messages, add_generation_prompt=True,
                                       tokenize=True, return_dict=True, return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=64, do_sample=False)
print(processor.decode(out[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True))

The model also works with a single image; the two-image prompt is how it is used inside CLARA. Recommended: bf16, one 24 GB GPU; served in CLARA via vLLM.

Intended use and limitations

Research / demonstration only. Not a medical device, not clinically validated, not for diagnostic use.
Trained and tested exclusively on synthetic (Gemini-generated) images — performance on real patient photos is unknown.
Covers 7 dermatologic/visible AE types only; anything else is out of distribution.
Tiny test set (42 images); accuracy figures have wide confidence intervals.
Inherits the Health AI Developer Foundations terms of use from MedGemma.

About CLARA

CLARA (Clinical Longitudinal AI Research Assistant) tackles the 12–21-day blind spot between oncology-trial clinic visits: a daily ~60-second video + voice check-in (Data Collection Agent, on-device) feeds a longitudinal timeline with AE/SAE flagging and automatic MedWatch 3500A / E2B(R3) reports (Data Analysis Agent). A rule-based clinical-trial simulator (100 patients × 126 days, 7 real drug profiles) is used to quantify the benefit: mean AE detection delay 4.6 → 1.2 days, deaths 21 → 16 per 100 patients.

HAI-DEF models used: MedGemma 1.5 4B (nurse dialogue, visual AE grading), MedSigLIP-448 (frame classification), HeAR (dry/wet cough), MedASR (medical speech recognition).

Code: newmes/clara-web · iOS app: newmes/clara-app
Sibling model: newmes/medgemma-4b-antihallu · data: newmes/clinical-trial-engine-data
Competition: The MedGemma Impact Challenge — Google Research / HAI-DEF, $100 000 prize pool, final submission 2026-02-24, 850+ teams. Judged on effective use of HAI-DEF models, problem importance, real-world impact, technical feasibility, and execution.

Originally published as AlphaRaven/medgemma-ae-detection (2026-02-23); moved to the newmes organization in Sep 2026. Weights are identical.

Citation

@misc{clara2026,
  title  = {CLARA: Clinical Longitudinal AI Research Assistant},
  author = {NEWMES AI Team},
  year   = {2026},
  url    = {https://github.com/newmes/clara-web},
  note   = {MedGemma Impact Challenge submission}
}

Downloads last month: 11

Safetensors

Model size

4B params

Tensor type

BF16

Model tree for newmes/medgemma-ae-detection

Base model

google/gemma-3-4b-pt

Finetuned

google/medgemma-4b-pt

Finetuned

google/medgemma-4b-it

Finetuned

newmes/medgemma-4b-antihallu

Finetuned

(1)

this model

newmes
/

medgemma-ae-detection

MedGemma AE Detection

Results

Usage

Intended use and limitations

About CLARA

Citation

Model tree for newmes/medgemma-ae-detection

Dataset used to train newmes/medgemma-ae-detection