paularoni/mtg-color-classifier

MTG Color Classifier

Looks at a piece of Magic: The Gathering card art and predicts which of the five Magic colors it belongs to: White, blUe, Black, Red, Green. It is multi-label: each color gets its own probability and its own decision threshold, so a picture can be several colors, or none (colorless). The model only ever sees the image, never the card's name, text or type.

Trained by fine-tuning an ImageNet-pretrained convnext_base.clip_laion2b_augreg_ft_in12k_in1k_384 (via timm) on the art crops of every unique illustration on Scryfall. Lands are labeled with their color identity (Forest -> G, a dual land -> both colors); every other card with its colors field.

Current model

Run	convnext_base_all_lands_r12_s42
Backbone	convnext_base.clip_laion2b_augreg_ft_in12k_in1k_384
Test macro F1 (nonland test cards, n=4,477)	0.620
Test macro AP	0.682
Exact match (all five colors right)	0.460
Promoted	2026-10-20T04:06:10+00:00
Trained with	torch 2.11.0+cu128, timm 1.0.29

Color	Test F1	Threshold
W	0.585	0.44
U	0.613	0.38
B	0.570	0.30
R	0.662	0.30
G	0.669	0.33

Thresholds were tuned per color on the validation split. Test cards never share an oracle_id with training cards, so the scores are on cards the model never saw.

Files

• model_bundle.pt: one dict with the weights (state_dict) and everything needed to use them: backbone, classes, thresholds, preprocess. Loads with torch.load(..., weights_only=True).
• model_info.json: the same fields without the weights.

Usage

import timm, torch
from huggingface_hub import hf_hub_download
from io import BytesIO
from PIL import Image
from torchvision import transforms as T

bundle = torch.load(hf_hub_download("paularoni/mtg-color-classifier", "model_bundle.pt"),
                    map_location="cpu", weights_only=True)
net = timm.create_model(bundle["backbone"], pretrained=False, num_classes=len(bundle["classes"]))
net.load_state_dict(bundle["state_dict"])
net.eval()

pp = bundle["preprocess"]
im = Image.open("art.jpg").convert("RGB")  # ideally color-managed to sRGB first
im = im.resize((pp["cache_size"], pp["cache_size"]), Image.Resampling.LANCZOS)
buf = BytesIO()  # JPEG round trip: the training images were JPEGs
im.save(buf, "JPEG", quality=pp["cache_jpeg_quality"], subsampling=pp["cache_jpeg_subsampling"])
im = Image.open(buf).convert("RGB")
x = T.Compose([T.Resize((pp["img_size"], pp["img_size"])), T.ToTensor(),
               T.Normalize(pp["mean"], pp["std"])])(im)
with torch.no_grad():
    probs = torch.sigmoid(net(x.unsqueeze(0)))[0].tolist()
colors = [c for c, p, t in zip(bundle["classes"], probs, bundle["thresholds"]) if p >= t]
print(dict(zip(bundle["classes"], probs)), "".join(colors) or "colorless")

Images are squashed to a square (no crop): color cues are often at the edges of the art. The two-step resize (Lanczos to 440, JPEG quality 95, then bilinear to 384) matches how the training images were prepared. Skipping the JPEG step shifts some probabilities by up to 0.1.

Limitations

• Art alone is often ambiguous: a macro F1 around 0.62 means many pictures get a wrong or missing color. Treat the probabilities as a guess, not a rule lookup.
• Trained on Magic card art; other pictures get an answer, but it means little.

Unofficial fan project, not affiliated with or endorsed by Wizards of the Coast. Card art is (c) Wizards of the Coast and its artists; card data and images from Scryfall.