harvinderbhullar/second-eye

Second Eye: CLM image state head for fracture detection (FracAtlas, C-RADIOv2-H)

Research prototype, not a medical device. Not validated for clinical use. Do not use it to diagnose or treat anyone.

A small trainable image state head for CLM (Contrastive Language Models). Frozen nvidia/C-RADIOv2-H image features (the vision tower of NVIDIA Nemotron Nano 2 VL) go through this head into CLM's 512-d action space. There they are scored by scale · cos against CLM's frozen action tower (Qwen3-8B last-token pooling + the released CLM-v0.1-8B action head) embedding the answer options:

• Question: "Is there a bone fracture in this X-ray?"
• Options: "Yes, there is a fracture." / "No, there is no fracture."

Code, training and evaluation: https://github.com/HarvinderBhullar/second-eye/tree/hackathon (see docs/VISION.md).

Files

file
model.safetensors	head weights (incl. train-set input standardisation) and options: the 2 frozen option vectors it was trained against
config.json	architecture (3840 → 1024 → 512, GELU), encoder, tile size, pooling, question and options
calibration.json	Platt scaling (a, b) and two thresholds on calibrated p(fracture), all fit on the validation split

Usage

git clone -b hackathon https://github.com/HarvinderBhullar/second-eye && cd second-eye
scripts/vision/step1_env.sh
python -m clm.vision.serve --model harvinderbhullar/second-eye --port 8701
curl -s -X POST localhost:8701/predict -F image=@xray.jpg

Input: one 512×512 RGB tile per image (greyscale X-rays converted to RGB, bicubic resize), RADIO summary vector, L2-normalised. Calibrated probabilities and the decision (p ≥ 0.0675, the val operating point with sensitivity ≥ 0.90) are valid only for the two trained options. Reworded options keep the ranking but not the calibration.

Evaluation

FracAtlas after integrity cleaning (truncated JPEGs, uniform bottom fills and byte duplicates removed): 3,938 images, 70/15/15 split stratified on fracture, seed 42. Every choice was made on validation. The test split (591 images, 104 fractured) was evaluated once. AUROC CI: 1,000-sample stratified bootstrap.

method	test AUROC [95% CI]	sens / spec @ val ≥0.90-sens threshold	ECE raw → Platt
CLM head · radio_summary	0.925 [0.891, 0.953]	0.913 / 0.723	0.062 → 0.020
Linear probe · radio_summary	0.893 [0.850, 0.931]	0.885 / 0.612	0.085 → 0.041
Resolution only (leakage check)	0.616 [0.577, 0.656]	0.933 / 0.269	0.323 → 0.025
(a) random option vectors · radio_summary	0.913 [0.878, 0.945]	0.875 / 0.733	0.075 → 0.027

Limitations.

• With two fixed options, random option vectors do about as well as the CLM option embeddings (see the last row). The head is effectively a classifier, and its value is the CLM interface, not accuracy.
• The head is ahead of a linear probe in point estimate, but the CIs overlap.
• It is trained on one public dataset from a limited set of sources, with no patient IDs and no external validation.
• Image size alone reaches about 0.62 AUROC on this data (a body-region confounder).

Requirements and licences

• These head weights: Apache-2.0.
• Using them requires nvidia/C-RADIOv2-H (NVIDIA, under its own licence) for the features, and Qwen3-8B plus the CLM action head only for custom options.
• Training data: FracAtlas, Abedeen et al., Scientific Data (2023), doi:10.6084/m9.figshare.22363012, CC BY 4.0.
• Built on Contrastive-LM/CLM (Apache-2.0) at the Nebius × NVIDIA hackathon, Berlin, October 2026.