hoilai-ltd/StarLife-V1-WSV

---

StarLife-1-WSV-Lite

Two frames + detections in, structured world events out — zero free-text generation.

This repository is self-contained: it ships the frozen visual backbone weights, the five discriminative heads and all inference code. Download it and run it; no external base model is required.

• Model identity: model_type = starlife_wsv, architectures = ["StarLifeWSVForConditionalGeneration"]
• Version: 1.0.0
• Size: ~55.7 GB (27B visual backbone + 150 MB heads)
• License: Apache-2.0 (visual backbone from Qwen3.8-27B, also Apache-2.0)

What it does

StarLife-1-WSV-Lite is the perception front-end of a world model. Instead of describing a scene in natural language, it classifies how the scene changed into structured state and relation transitions.

two frames + detections
   │  frozen visual backbone (Qwen3.8-27B, 5120-d last hidden state)
   ▼
slot_before / slot_after / |Δ|  + 13-d slot geometry + detector-box pair geometry
   │  5 MLP heads (LayerNorm → 512 → 256 → n_out)
   ▼
change(4) / type(22) / relation(12) / rel_pos(10) / attrs(2)
   │  event translation
   ▼
EntityCreated / StateChanged / RelationAdded / RelationRemoved

Head	Classes	Meaning
change	4	none / appeared / displaced / disappeared
type	22	closed object set (book, mug, monitor, …)
relation	12	above / below / on / in / near / next_to / part_of …
rel_pos	10	front / back / left / right / inside / top …
attrs	2	binary contact / support

Download

Hugging Face:

pip install -U huggingface_hub
huggingface-cli download hoilai-ltd/StarLife-V1-WSV --local-dir ./StarLife-1-WSV-Lite

ModelScope mirror: Hoilai/StarLife-V1-WSV

pip install modelscope
modelscope download --model Hoilai/StarLife-V1-WSV --local_dir ./StarLife-1-WSV-Lite

Quick start

from starlife_wsv import StarLifeWSVModel

model = StarLifeWSVModel.from_pretrained("./StarLife-1-WSV-Lite", device="cuda")

detections = {
    "width": 1024, "height": 768,
    "before": [{"id": "mug", "bbox": [190, 415, 289, 508]}],
    "after":  [{"id": "mug", "bbox": [200, 410, 300, 505]}],
}
result = model.predict_events("before.jpg", "after.jpg", detections)
for e in result["events"]:
    print(e)

Example output:

{"type": "StateChanged", "entity_id": "slot:mug", "valid_from": 1761000000000,
 "properties": {"present": true, "moved": true}}

Equivalent CLI:

python e2e.py --model ./StarLife-1-WSV-Lite \
    --before before.jpg --after after.jpg --detections detections.json --out events.json

Heads only (when you already have features):

from pipeline import StarLifeWSV

heads = StarLifeWSV("./StarLife-1-WSV-Lite", device="cuda",
                    config_name="starlife_wsv_config.json")
out = heads.predict(slot_before, slot_after, slot_mask, slot_geom,
                    frm, to, pair_dist, pair_bbox_geom)
labels = heads.decode(out)

Files

File	Size	Purpose
model-00001..00018-of-00018.safetensors + model.safetensors.index.json	~52 GB	Visual backbone weights
config.json / preprocessor_config.json / tokenizer* / vocab.json / merges.txt / chat_template.jinja / configuration.json	small	Backbone config and tokenizer
heads.safetensors	150 MB	Five discriminative heads
starlife_wsv_config.json	3.5 KB	Model config (head specs, label maps, geometry normalisation)
starlife_wsv.py	4 KB	Entry point: StarLifeWSVModel.from_pretrained(...)
e2e.py	12 KB	End-to-end CLI
pipeline.py	6 KB	Head inference (numpy/torch/safetensors only)
extract_features.py / features.py	small	Backbone feature extraction and feature assembly
validate.py / validate_e2e.py	small	Head-level / end-to-end self-checks
validation_report.json / e2e_report.json / requirements.txt / LICENSE	small	Reports, dependencies, licence

Metrics

Head level (1,000 held-out test pairs, reproducible with validate.py):

Head	top-1	macro-F1
change	0.9973	0.9948
type (22 classes)	0.9606	0.9568
relation (12 classes)	0.9070	0.8735
rel_pos (10 classes)	0.8632	0.8122
attrs	contact F1 0.9548 / support F1 0.9665

End to end (30 image pairs through the detection-box path, validate_e2e.py):

Metric	Value
change accuracy	0.921
type accuracy	0.988
latency	~0.91 s per pair (A800 80 GB, incl. feature extraction and 5 heads)

Limitations

1. type is a closed 22-class head; objects outside the set are misclassified. Extending the set requires retraining that head.
2. relation.next_to (F1 0.429) and rel_pos.back_left (0.678) / left (0.685) are known weak classes.
3. Training used ground-truth 3D pose in the geometry descriptor. This repository approximates it from detector boxes (presence, centre shift, area-ratio depth proxy, box coverage; orientation deltas set to 0). This is a degraded mode: displaced and rel_pos are less accurate than the head-level numbers. Providing upstream pose/depth estimates restores full accuracy.
4. Loading the 52 GB backbone needs ≥ 80 GB of GPU memory (verified on A800).
5. This model only performs classification; it does not generate images.

Credits

Visual backbone: Qwen3.8-27B (Apache-2.0), redistributed here with its original LICENSE. Heads and all inference/wrapper code: StarLife project, Apache-2.0.