MakeModel-VLM

MakeModel-VLM is a compact vision-language model fine-tuned to recognize vehicles from images and return a structured description of their make/model, class, and color. It is built on top of unsloth/Qwen3.5-2B and specialized for vehicles commonly seen on Indian roads.

The model is trained to respond with a single JSON object, making it easy to plug into downstream pipelines (ANPR/ITMS systems, traffic analytics, fleet monitoring, dashcam processing, etc.).

What it does

Given a vehicle image, the model outputs:

{"make_model": "Maruti Suzuki Swift", "class": "car", "color": "White"}

• make_model — the manufacturer and model (e.g. Hero Honda Splendor, Tata Ace, Mahindra Bolero, Auto Rickshaw).
• class — the vehicle category: car, bike, auto, truck, bus, van, or train.
• color — the dominant color of the vehicle.

Highlights

• Works well on cropped vehicle images. It was trained primarily on tight crops of individual vehicles, so it performs best when the vehicle fills most of the frame — the typical output of an upstream object detector.
• Tuned for the Indian vehicle landscape — two-wheelers, three-wheeler auto-rickshaws, compact cars, and a wide range of commercial trucks/buses.
• Structured JSON output for zero-parsing integration.
• Small and fast — a 2B-parameter backbone that serves comfortably on a single modern GPU.

Intended use

MakeModel-VLM is designed to sit after a vehicle detector in a pipeline: the detector localizes vehicles, and this model classifies each crop. It is well suited to:

• Automatic vehicle attribute tagging in traffic/surveillance feeds
• Fleet and parking analytics
• Enriching detection outputs with make/model/color metadata

Usage

Serving with vLLM (OpenAI-compatible API)

vllm serve sanskar003/MakeModel-VLM \
    --served-model-name sanskar003/MakeModel-VLM \
    --max-model-len 4096 \
    --dtype bfloat16 \
    --trust-remote-code

Then query it like any OpenAI vision chat endpoint:

from openai import OpenAI
import base64

client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")

with open("vehicle_crop.jpg", "rb") as f:
    img = base64.b64encode(f.read()).decode()

resp = client.chat.completions.create(
    model="sanskar003/MakeModel-VLM",
    messages=[{
        "role": "user",
        "content": [
            {"type": "text", "text": (
                "You are a vehicle recognition expert. Look at the vehicle in the "
                "image and identify it. Respond ONLY with a single JSON object with "
                'exactly these keys: {"make_model": string, "class": string, '
                '"color": string}. No extra text.'
            )},
            {"type": "image_url",
             "image_url": {"url": f"data:image/jpeg;base64,{img}"}},
        ],
    }],
    temperature=0.0,
    max_tokens=128,
)
print(resp.choices[0].message.content)

Inference with Transformers

from transformers import AutoModelForImageTextToText, AutoProcessor
from PIL import Image
import torch

model = AutoModelForImageTextToText.from_pretrained(
    "sanskar003/MakeModel-VLM", torch_dtype=torch.bfloat16, device_map="auto"
)
processor = AutoProcessor.from_pretrained("sanskar003/MakeModel-VLM")

image = Image.open("vehicle_crop.jpg").convert("RGB")
instruction = (
    "You are a vehicle recognition expert. Look at the vehicle in the image and "
    "identify it. Respond ONLY with a single JSON object with exactly these keys: "
    '{"make_model": string, "class": string, "color": string}. No extra text.'
)
messages = [{"role": "user", "content": [
    {"type": "image"}, {"type": "text", "text": instruction}]}]
prompt = processor.apply_chat_template(messages, add_generation_prompt=True)
inputs = processor(images=image, text=prompt, return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=128, do_sample=False)
print(processor.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

Prompt format

For best results, use the exact instruction the model was trained with:

You are a vehicle recognition expert. Look at the vehicle in the image and identify it. Respond ONLY with a single JSON object with exactly these keys: {"make_model": string, "class": string, "color": string}. No extra text.

Training

• Base model: unsloth/Qwen3.5-2B
• Method: LoRA fine-tuning (vision + language layers) via Unsloth, merged to 16-bit for serving.
• Data: a curated dataset of Indian road vehicle images with make/model, class, and color labels.
• Precision: bfloat16.

Limitations

• Best on cropped, reasonably clear vehicle images; performance drops on wide scenes, heavy occlusion, extreme angles, or very low resolution.
• make/model is the hardest attribute — visually near-identical models or trim variants can be confused. class and color are more reliable.
• Optimized for Indian-market vehicles; models rarely seen in that market may be misidentified.
• Occasionally the identified make/model is a close but not exact match; treat low-confidence cases accordingly.

License

Released under the Apache-2.0 license, consistent with the base model.