hotdog-vlm 🌭

Hot dog or not hot dog, from a language model that has never seen an image.

Qwen3-0.6B is text-only. This repo teaches it to see, LLaVA-style: a frozen SigLIP encoder turns a photo into 196 patch vectors, a small projector folds them into 49 "image tokens" in the LLM's embedding space, and a LoRA on the LLM learns to read them. The whole thing was trained on one GeForce RTX 2080 Ti, just for fun.

The rules

Strict. A hot dog is a frankfurter or wiener in a bun, with any toppings. These are not hot dogs:

• bratwurst, even in a bun
• any other sausage
• corn dogs (no bun)
• sub sandwiches, hoagies, heroes

What's in this repo

File	What it is
adapter.safetensors	The projector (3072 → 1024 → 1024 MLP) and the LoRA weights (r=16 on every attention and MLP projection). 14.3M parameters.
vlm.py	The model: SigLIP features → projector → Qwen3, plus hotdog_score and generate.
load.py	Downloads everything and classifies the photos you pass it.

uv run --with torch --with transformers --with peft --with huggingface_hub python load.py photo.jpg

The verdict is the first answer token: log P("Hot") − log P("Not"), where above 0 means hot dog. The same model also describes photos in one sentence (Describe this image in one sentence.).

Training

1. Stage 1, projector only (LLM frozen): 2,000 steps on captions from Flickr30k (31K photos) and the food photos in COCO (16K photos). The projector learns to speak the LLM's language; the LLM learns nothing new.
2. Stage 2, projector + LoRA: 800 steps. Three of every four batches are the hot dog question; the rest are captions, so the model keeps describing photos.
   • Positives: Food-101 hot_dog and Flickr30k photos captioned as hot dogs. Food-101's hot dog class uses the everyday meaning, so SigLIP zero-shot flagged the least frankfurter-like 300 and a human reviewed them: 72 brats, sausage plates, corn dogs and subs were relabeled "not".
   • Plus 389 positives from Wikimedia Commons and COCO, kept only when SigLIP scores them as frankfurter-like.
   • Negatives: other Food-101 classes (including lookalikes like hamburgers, lobster rolls and burritos), non-food Flickr30k photos, and 559 hard negatives: bratwursts and subs from Commons and COCO.

Image features are cached once, so training never runs the image encoder.

Results

Held-out Food-101 / Flickr30k (1,005 photos): 98.7%.

Hard test set: 179 Wikimedia Commons photos from the hot dog, bratwurst and sub sandwich categories, hand-labeled under the strict rules and never trained on.

	before hard negatives	first retrain	this model
Overall	60.3%	80.4%	85.5%
Bratwursts called "not"	13/60	58/60	54/60
Subs called "not"	46/60	60/60	59/60
Real hot dogs called "hot"	44/48	17/48	33/48
Other photos from the hot dog categories, called "not"	5/11	9/11	7/11

Most of the hot dog "misses" aren't hot dogs. Looking at the 15 photos labeled hot dog that this model rejects, 9 shouldn't have been labeled hot dog: a seagull carrying a sausage, a bakery case of sandwiches, a Rochester garbage plate, a vintage barbecue illustration, a plate of Kasseler with a sausage, a Cincinnati smoked sausage, and three photos where the hot dog is a speck in the frame. With those relabeled or dropped, the hard set score is 90.3% (real hot dogs 33/39). The six genuine misses are franks that are hard to see: wrapped in foil, buried in sauerkraut, or half out of frame.

What went wrong along the way: the first retrain took every hard negative from Commons and every positive from Food-101, so it learned "a Commons-style photo isn't a hot dog" and rejected two thirds of the real Commons hot dogs. Adding Commons and COCO positives fixed most of that.

Speed

On the 2080 Ti, with CUDA graphs (the VM's host CPU is old enough that kernel launches dominate): about 45 ms for a verdict and about 100 ms for a description. Fine on CPU too, just slower.

Limitations

• Trained for fun, on about 1,100 positives. It will be wrong in ways that are funny and some that aren't.
• 224 px input: small or hidden franks are hard for it, and so is telling a frank from a brat by texture alone.
• The hard test set is small (179 photos), and its labels are still being cleaned up.

Data and licenses

The code and weights are Apache 2.0, like both base models. The training data has its own terms: Food-101 and Flickr30k are for research and non-commercial use, COCO annotations are CC BY 4.0 (images under their Flickr licenses), and the Commons photos are under their individual Creative Commons licenses. No training images are redistributed here.