HOI-R1 Qwen3.5-4B SFT

This repository contains the supervised fine-tuned (SFT) HOI-R1 Qwen3.5-4B checkpoint. It was trained from Qwen/Qwen3.5-4B using full-parameter fine-tuning with ms-swift.

Training summary

Item	Value
Fine-tuning method	Full-parameter SFT
Base model	Qwen/Qwen3.5-4B
Training checkpoint	1164 (final step)
Training dataset	HICO-DET SFT data for Qwen3
Final training loss	0.3283
Validation accuracy / mAP	Not reported; no validation dataset was configured for this run

The uploaded files are the inference-ready model shards and configuration/tokenizer/processor files. DeepSpeed optimizer state, RNG state, and other training-resume files are not included.

Evaluation results

The Qwen3.5-4B-SFT row (1 epoch) is the SFT result for this model. The separate HOI-R1-Qwen3.5-4B row is the SFT + GRPO result.

HOI-R1 main results

Comparison table supplied for the HOI-R1 models. Scores are reproduced as provided.

Category	Method	Training	Default Full	Default Rare	Default Non-Rare	Known Object Full	Known Object Rare	Known Object Non-Rare
Traditional HOID Methods	HO-RCNN	150k steps	7.81	5.37	8.54	10.41	8.94	10.85
Traditional HOID Methods	HOTR	100 epochs	25.10	17.34	27.42	---	---	---
Traditional HOID Methods	HOITrans	150 epochs	26.61	19.15	28.84	29.13	20.98	31.57
Traditional HOID Methods	QPIC	150 epochs	29.07	21.85	31.23	31.68	24.14	33.93
MLLM Zero-Shot Baselines	Qwen2.5-VL-3B (baseline)	---	8.39	9.60	8.03	8.96	9.83	8.70
MLLM Zero-Shot Baselines	Qwen2.5-VL-7B	---	10.46	14.30	9.31	11.01	14.63	9.93
MLLM Zero-Shot Baselines	Qwen2.5-VL-32B-AWQ	---	18.12	24.56	16.20	19.90	25.77	18.15
MLLM Zero-Shot Baselines	Qwen2.5-VL-72B-AWQ	---	20.71	29.62	18.05	22.93	31.87	20.26
MLLM Zero-Shot Baselines	Qwen3-VL-4B (baseline)	---	16.16	22.31	14.32	17.55	22.87	15.96
MLLM Zero-Shot Baselines	Qwen3.5-4B (baseline)	---	17.17	20.82	16.08	19.60	23.16	18.53
SFT or RL Fine-Tuning	Qwen2.5-VL-3B-SFT	400 steps	11.71	10.52	12.07	12.60	10.70	13.17
SFT or RL Fine-Tuning	Qwen2.5-VL-3B-SFT	1000 steps	15.23	12.26	16.11	16.41	12.60	17.54
SFT or RL Fine-Tuning	Qwen2.5-VL-3B-SFT	1 epoch	16.77	14.20	17.53	18.05	14.45	19.13
SFT or RL Fine-Tuning	Qwen2.5-VL-3B-GRPO	400 steps	12.22	14.56	11.52	13.12	14.84	12.60
SFT or RL Fine-Tuning	Qwen2.5-VL-3B-GRPO	1000 steps	13.25	15.22	12.66	14.18	15.60	13.75
SFT or RL Fine-Tuning	Qwen2.5-VL-3B-GRPO	1 epoch	14.65	15.90	14.28	15.48	16.14	15.28
SFT or RL Fine-Tuning	Qwen3-VL-4B-SFT	1 epoch	22.46	21.53	22.73	24.04	21.97	24.66
SFT or RL Fine-Tuning	Rex-Omni-3B (HOI-SFT)	1 epoch	25.30	21.24	26.51	26.97	21.79	28.51
SFT or RL Fine-Tuning	Qwen3.5-4B-SFT	1 epoch	29.36	30.47	29.03	31.22	30.92	31.31
HOI-R1 (SFT + GRPO)	HOI-R1-Qwen2.5-VL-3B	1 epoch + 40 steps	18.33	16.03	19.02	19.83	16.25	20.90
HOI-R1 (SFT + GRPO)	HOI-R1-Qwen3-VL-4B	1 epoch + 40 steps	23.35	23.58	23.28	25.12	24.11	25.42
HOI-R1 (SFT + GRPO)	HOI-R1-Rex-Omni-3B	1 epoch + 40 steps	25.24	22.57	26.04	27.00	23.21	28.14
HOI-R1 (SFT + GRPO)	HOI-R1-Qwen3.5-4B	1 epoch + 40 steps	29.79	30.39	29.60	31.61	30.67	31.89

The Qwen3.5-4B-SFT row is the SFT result for this model (1 epoch): 29.36 / 30.47 / 29.03 (Default) and 31.22 / 30.92 / 31.31 (Known Object). The separate HOI-R1-Qwen3.5-4B row is the SFT + GRPO result.

Original caption: Comparison of HOI-R1 with traditional HOID methods and MLLM baselines.

Citation

@article{chen2025hoi,
  title={HOI-R1: Exploring the Potential of Multimodal Large Language Models for Human-Object Interaction Detection},
  author={Chen, Junwen and Xiong, Peilin and Yanai, Keiji},
  journal={arXiv preprint arXiv:2510.05609},
  year={2025}
}