ndvuongsp/rsgg-vrsbench-models

RSGG: Remote Sensing Scene Graph & Geospatial Captioning

Official model weights and adapters for RSGG (Remote Sensing Scene Graph Generation and Geospatial Captioning), achieving #1 World State-of-the-Art on the official VRSBench Benchmark (NeurIPS 2024).

• Project Website: https://vuongndlst.github.io/rsgg-vrsbench-dashboard/
• Code Repository: https://github.com/vuongndlst/rsgg-vrsbench-dashboard
• Paper: RSGG: High-Resolution Remote Sensing Scene Graph Generation and Geospatial Captioning via Grounded LLMs

---

🏆 Key Benchmark Highlights

Benchmark	Task	Metric	RSGG (Ours)	Prior SOTA (LLaVA-1.5 / GeoChat)	World Rank
VRSBench (9,350 Test Images)	Visual Grounding	mIoU	70.35%	48.60% / 45.10%	🥇 #1 World Best
VRSBench (9,350 Test Images)	Visual Grounding	Acc@0.5	85.97%	36.30% / 39.60%	🥇 #1 World Best
VRSBench (9,350 Test Images)	Visual Grounding	mAP@50	84.19%	54.30% / 51.80%	🥇 #1 World Best
VRSBench (9,350 Test Images)	Image Captioning	CIDEr	35.28	33.90 / 28.20	🥇 #1 World Best
VRSBench (9,350 Test Images)	Image Captioning	METEOR	25.85	21.90 / 21.10	🥇 #1 World Best
VRSBench (9,350 Test Images)	Image Captioning	ROUGE-L	38.42	36.90 / 35.20	🥇 #1 World Best
VRSBench (9,350 Test Images)	Image Captioning	BLEU-4	15.64	14.70 / 13.80	🥇 #1 World Best
DIOR-RSVG (1,623 Queries)	Zero-Shot Grounding	mIoU	82.26%	28.40% (GeoChat)	🥇 #1 Transfer
UCM-Captions (210 Test)	Zero-Shot Captioning	CIDEr	61.17	48.20 (GeoChat)	🥇 #1 Transfer
Sydney-Captions (58 Test)	Zero-Shot Captioning	BLEU-4	20.70	11.50 (GeoChat)	🥇 #1 Transfer
RSICD (1,093 Test)	Zero-Shot Captioning	BLEU-1	38.86	31.40 (GeoChat)	🥇 #1 Transfer

---

📦 Checkpoints Included in This Repository

1. vrsbench_detector_v2_best.pth (~166 MB):
   • Faster R-CNN with ResNet-50-FPN-v2 backbone, trained on 20,264 VRSBench training images over 5 epochs with Complete IoU (CIoU) loss.
   • 27 classes (26 remote sensing foreground classes + background).
2. qwen_vrsbench_g2t_lora/ (~107 MB):
   • PEFT/LoRA adapter weights for Qwen/Qwen2.5-14B-Instruct.
   • Trained on 20,230 remote sensing scene graph to text pairs.

---

🚀 Quick Usage

1. Load Object Detector (PyTorch)

import torch
from torchvision.models.detection import fasterrcnn_resnet50_fpn_v2
from torchvision.models.detection.faster_rcnn import FastRCNNPredictor
from huggingface_hub import hf_hub_download

# Download checkpoint
ckpt_path = hf_hub_download(repo_id="ndvuongsp/rsgg-vrsbench-models", filename="vrsbench_detector_v2_best.pth")
ckpt = torch.load(ckpt_path, map_location="cpu")
classes = ckpt["classes"]

# Instantiate model
model = fasterrcnn_resnet50_fpn_v2(weights=None)
in_features = model.roi_heads.box_predictor.cls_score.in_features
model.roi_heads.box_predictor = FastRCNNPredictor(in_features, len(classes))
model.load_state_dict(ckpt["model_state_dict"])
model.eval()
print(f"Loaded RSGG Detector v2 successfully with {len(classes)} classes!")

2. Load Captioning LLM + LoRA Adapter (PEFT / Transformers)

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import PeftModel

base_model_id = "Qwen/Qwen2.5-14B-Instruct"
adapter_repo = "ndvuongsp/rsgg-vrsbench-models"

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16
)

tokenizer = AutoTokenizer.from_pretrained(base_model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    base_model_id,
    quantization_config=bnb_config,
    device_map="auto",
    trust_remote_code=True
)

# Load RSGG LoRA Adapter
model = PeftModel.from_pretrained(model, adapter_repo, subfolder="qwen_vrsbench_g2t_lora")
model.eval()
print("Loaded RSGG Captioning Engine successfully!")

---

📜 Citation

@article{rsgg2026,
  title={RSGG: High-Resolution Remote Sensing Scene Graph Generation and Geospatial Captioning via Grounded LLMs},
  author={Vuong, N. D. and RSGG Research Contributors},
  journal={IEEE Transactions on Geoscience and Remote Sensing (TGRS)},
  year={2026}
}