Instructions to use yl4300/TAU-Agent-Question-Answering-VLM with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use yl4300/TAU-Agent-Question-Answering-VLM with PEFT:

from peft import PeftModel
from transformers import AutoModelForCausalLM

base_model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-VL-8B-Instruct")
model = PeftModel.from_pretrained(base_model, "yl4300/TAU-Agent-Question-Answering-VLM")

Transformers

How to use yl4300/TAU-Agent-Question-Answering-VLM with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="yl4300/TAU-Agent-Question-Answering-VLM")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("yl4300/TAU-Agent-Question-Answering-VLM")
model = AutoModelForMultimodalLM.from_pretrained("yl4300/TAU-Agent-Question-Answering-VLM", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use yl4300/TAU-Agent-Question-Answering-VLM with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "yl4300/TAU-Agent-Question-Answering-VLM"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "yl4300/TAU-Agent-Question-Answering-VLM",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/yl4300/TAU-Agent-Question-Answering-VLM

SGLang

How to use yl4300/TAU-Agent-Question-Answering-VLM with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "yl4300/TAU-Agent-Question-Answering-VLM" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "yl4300/TAU-Agent-Question-Answering-VLM",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "yl4300/TAU-Agent-Question-Answering-VLM" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "yl4300/TAU-Agent-Question-Answering-VLM",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use yl4300/TAU-Agent-Question-Answering-VLM with Docker Model Runner:
```
docker model run hf.co/yl4300/TAU-Agent-Question-Answering-VLM
```

TAU-Agent Question-Answering VLM (LoRA Adapter)

This repository contains the fine-tuned LoRA adapter for the question-answering vision-language model used in TAU-Agent, an agentic retrieval-augmented framework for traffic anomaly understanding. The adapter is applied on top of Qwen/Qwen3-VL-8B-Instruct and was fine-tuned for reasoning about anomalous events in traffic videos.

Paper: TAU-Agent: An Agentic Retrieval-Augmented Framework for Traffic Anomaly Understanding
Code: github.com/siri-rouser/TAU-Agent

The framework achieved 2nd place on Track 3 (TAR), 12th place on Track 7 (FETV), and 5th place on Track 8 (PSI-VQA) of the AI City Challenge 2026, with scores of 0.6779, 0.3998, and 67.9275 respectively.

Model Details

Base model: Qwen/Qwen3-VL-8B-Instruct
Adapter type: LoRA (rank 128, alpha 256)
Task: Traffic anomaly understanding (question answering over video evidence)
Language: English (inference on traffic scenes)

Usage

This is a PEFT LoRA adapter. To use it, load the base model and adapter together (e.g., via peft or Hugging Face transformers with PEFT integration). For the full TAU-Agent pipeline, including retrieval, captioning, and tracking, please refer to the GitHub repository, which provides detailed instructions and examples.

Citation

Citation details will be added when the paper is officially published. Please refer to the paper link above.

Downloads last month: 23

Model tree for yl4300/TAU-Agent-Question-Answering-VLM

Base model

Qwen/Qwen3-VL-8B-Instruct

Adapter

(168)

this model

Space using yl4300/TAU-Agent-Question-Answering-VLM 1

Paper for yl4300/TAU-Agent-Question-Answering-VLM

TAU-Agent: An Agentic Retrieval-Augmented Framework for Traffic Anomaly Understanding

Paper • 2608.25935 • Published 3 days ago