README.md

metadata

title: GraphAttributeLearning
emoji: 🪑
colorFrom: gray
colorTo: red
sdk: gradio
sdk_version: 6.9.0
python_version: '3.10'
app_file: app.py
pinned: false

Adjective-Aware Object Embedding

This project builds adjective-aware object embeddings using:

• CLIP image features
• DINOv2 image features
• Graph Neural Network (GNN) reasoning over object-attribute graphs

Current scope starts with chair and multi-label attributes like broken, blue, wooden, and plastic.

Project Structure

• plan/plan.md - detailed execution strategy and technical plan
• configs/dataset.yaml - dataset, attributes, splits, normalization
• configs/model.yaml - encoder, fusion, graph, and head settings
• configs/train.yaml - optimization, staging, losses, checkpointing
• configs/eval.yaml - metrics, visualization, error analysis
• configs/experiment.yaml - baseline/main/ablation run definitions
• scripts/data/ - data download, extraction, processing, split pipeline
• src/data/ - reusable normalization and IO utilities for data pipeline

Config Overview

dataset.yaml

Defines:

• dataset root/cache/processed paths
• attribute taxonomy and synonym normalization
• split strategy and filtering rules
• basic augmentation

model.yaml

Defines:

• CLIP + DINO encoder settings
• fusion type (concat, optional gated/projected ablations)
• graph variant (bipartite / enriched)
• attribute node initialization (clip_text / trainable)
• GNN layers and baseline MLP head

train.yaml

Defines:

• two-stage training (frozen then partial fine-tuning)
• optimizer, scheduler, AMP, grad clipping
• weighted BCE + optional contrastive loss
• early stopping and checkpoint monitoring

eval.yaml

Defines:

• classification metrics (mAP, macro/micro F1, per-attribute metrics)
• embedding metrics (retrieval@K, silhouette)
• UMAP/t-SNE options
• error analysis outputs

experiment.yaml

Defines:

• baseline runs (A/B/C)
• main proposed run (CLIP+DINO+GNN)
• ablation runs (graph type, node init, loss, fine-tuning)

Execution Strategy (No Timeline)

Follow this order:

1. Data prep + label normalization
2. Baselines (A/B/C)
3. Graph model training
4. Ablations and tuning
5. Robustness checks + packaging
6. Build a minimal Gradio demo interface (final step, after model is fully trained and selected)

Use plan/plan.md as the source of truth for quality gates and deliverables.

How To Run

1) Run Chair-Only Data Pipeline (Visual Genome)

Default full pipeline:

python scripts/data/run_data_pipeline.py --config configs/dataset.yaml

Run individual stages:

• python scripts/data/run_data_pipeline.py --download-only --config configs/dataset.yaml
• python scripts/data/run_data_pipeline.py --extract-only --config configs/dataset.yaml
• python scripts/data/run_data_pipeline.py --process-only --config configs/dataset.yaml
• python scripts/data/run_data_pipeline.py --split-only --config configs/dataset.yaml

Direct scripts (if needed):

• python scripts/data/download_visual_genome.py --config configs/dataset.yaml
• python scripts/data/extract_visual_genome.py --config configs/dataset.yaml
• python scripts/data/process_visual_genome.py --config configs/dataset.yaml
• python scripts/data/build_splits.py --config configs/dataset.yaml

Important: the processing pipeline is strict chair-only (primary_object_class: chair).

2) Train/Evaluate Models

Baseline (CLIP/DINO + MLP)

• Smoke run (quick sanity check):

  python scripts/train_baseline.py --run-name smoke_baseline --mode smoke --device auto

• Full run:

  python scripts/train_baseline.py --run-name baseline_full --mode full --device auto

GNN (CLIP/DINO + bipartite native GNN)

• GNN smoke run:

  python scripts/train_gnn.py --run-name smoke_gnn --mode smoke --device auto

• Compare baseline vs GNN (after both smoke runs complete):

  python scripts/compare_baseline_vs_gnn.py --baseline-run smoke_baseline --gnn-run smoke_gnn

3) Inference and Gradio UI

CLI inference

• Baseline checkpoint:

  python scripts/infer.py --checkpoint outputs/smoke_baseline/best.pt --image path/to/chair.jpg --top-k 5 --threshold 0.5

• GNN checkpoint:

  python scripts/infer.py --checkpoint outputs/smoke_gnn/best.pt --image path/to/chair.jpg --top-k 5 --threshold 0.5

Gradio app (local)

• Launch:

  python app.py

• Then:

  • Open the printed URL.
  • Upload a chair image.
  • Choose baseline or gnn.
  • Adjust threshold/top-k as needed.

Expected Outputs

Data artifacts:

• data/processed/visual_genome/samples.parquet (or samples.csv fallback)
• data/processed/visual_genome/label_vocab.json
• data/processed/visual_genome/label_frequencies.json
• data/processed/visual_genome/processing_report.json
• data/processed/visual_genome/splits/train.json
• data/processed/visual_genome/splits/val.json
• data/processed/visual_genome/splits/test.json
• data/processed/visual_genome/splits/split_report.json

Training/eval artifacts:

• model checkpoints
• evaluation reports (mAP/F1/per-attribute)
• embedding analysis artifacts (UMAP/retrieval results)
• error analysis summaries
• minimal Gradio demo for interactive predictions

Notes

• Keep experiments reproducible with fixed seeds.
• Track all runs with consistent config snapshots.
• Measure gains against Baseline C to isolate graph contribution.