risk-a4d

A CLIP ViT-B-32 (OpenAI pretrained) checkpoint, fully fine-tuned (both image and text encoders unfrozen) for affordance classification, including a set of child-safety "risk" affordances in addition to the original object-manipulation affordances.

This is the "A4D" methodology (affordance-vs-antonym contrastive fine-tuning of CLIP) applied to risk_dataset: 686 images across 35 everyday object categories (furniture, tableware, tools, and hazard items like batteries, pill bottles, and glass vases), labeled across 15 binary affordances.

This checkpoint was trained on 100% of risk_dataset with no held-out test split — it's meant to be used, not benchmarked. For held-out accuracy numbers, see the companion 80/20-split run (frozen-text 0.824 / frozen-vision 0.933 / all-unfrozen 0.965 overall accuracy on that split).

Affordances

Original object-manipulation affordances (10): containable, graspable, liftable, movable, openable, pushable, rollable, stackable, supportable, traversable

New risk affordances (5) — designed for a child-safety-hazard demo (e.g. "a baby is near a knife → knife is sharp → dangerous"): sharp, chokeable, hot, toxic, fragile

Usage

Predictions are made contrastively: for each affordance, compare the image against the affordance word and its antonym, and take whichever scores higher.

import open_clip
import torch
from PIL import Image

AFFORDANCE_ANTONYMS = {
    "graspable": "slippery", "movable": "anchored", "stackable": "wobbly",
    "supportable": "fragile", "liftable": "heavy", "rollable": "fixed",
    "openable": "sealed", "containable": "solid", "traversable": "blocked",
    "pushable": "immovable",
    "sharp": "blunt", "chokeable": "oversized", "hot": "cool",
    "toxic": "nontoxic", "fragile": "durable",
}

model, _, preprocess = open_clip.create_model_and_transforms("ViT-B-32", pretrained="openai")
tokenizer = open_clip.get_tokenizer("ViT-B-32")

ckpt = torch.load("clip_risk_all_unfrozen_full.pt", map_location="cpu")
model.load_state_dict(ckpt["model_state_dict"])
model.eval()

img = preprocess(Image.open("knife.jpg").convert("RGB")).unsqueeze(0)
affordance = "sharp"
antonym = AFFORDANCE_ANTONYMS[affordance]
tokens = tokenizer([affordance, antonym])

with torch.no_grad():
    img_feat = torch.nn.functional.normalize(model.encode_image(img), dim=-1)
    txt_feat = torch.nn.functional.normalize(model.encode_text(tokens), dim=-1)
    sims = (img_feat @ txt_feat.T).squeeze(0)

pred = affordance if sims[0] > sims[1] else antonym
print(pred)  # "sharp"

Training details

• Base model: open_clip ViT-B-32, openai pretrained weights
• Fine-tuning: both vision and text encoders unfrozen (151.3M trainable params)
• Objective: symmetric CLIP contrastive loss, image ↔ (affordance or antonym) text
• Optimizer: AdamW, lr 1e-5, weight decay 0.01
• Schedule: OneCycleLR, 50 warmup steps
• Batch size: 32, Epochs: 20, Seed: 42
• Data: risk_dataset — 686 images, 35 object classes, 15 affordances, 9,866 labeled (image, affordance, label) triples, trained on all of them (no holdout)

Repo / code

Training code: affordance_classification/training/risk_dataset_training.py (80/20 held-out variant) and risk_dataset_training_full.py (this checkpoint, trained on 100% of the data) in the Rule-Discovery repo.