jackstev/hw2-pool-ball-solid-stripe

Pool Ball Solid vs. Stripe Classifier (AutoGluon MultiModal)

Homework 2, Problem 2.

What it does

Takes a photo of a single pool ball on a blue table and predicts solid (0) or stripe (1).

Data

• Source: cmuchancel/hw1-pool-balls, revision 40c628cd92af. A classmate photographed 30 distinct balls from two sets.
• Splits used as published: train = 21 originals + 336 augmented copies (brightness, rotation, contrast, blur), validation = 4 originals, test = 5 originals. Copies come only from training photos, and no photo crosses splits.

Preprocessing

The dataset supplies 224 x 224 RGB images with EXIF orientation applied, aspect ratio preserved and gray (128,128,128) padding. AutoGluon then applies the backbone's own resize, center crop and ImageNet normalization. Use the same preparation on new photos.

Search

AutoGluon MultiModal 1.6.1, medium_quality preset, 12 runs in 3.6 minutes, capped at 240 s and 10 epochs per run, early stopping with optim.patience = 3. Seed 24679.

What varied: backbone (ResNet-18, MobileNetV3-Small, EfficientNet-B0), how much of it was trained via optim.peft (bit_fit, norm_fit, full), learning rate (0.0004, then 0.0001 and 0.001 around the leader), and whether the augmented copies were included.

Selection used validation cross-entropy rather than accuracy, because 4 validation photos give accuracy only five possible values.

Run	Backbone	peft	LR	Train rows	Val loss	Val acc
efficientnet_b0__None__lr0.001	efficientnet_b0	None	0.001	357	0.0803	1.00
efficientnet_b0__full	efficientnet_b0	None	0.0004	357	0.3689	0.75
ablation_originals_only	efficientnet_b0	None	0.001	21	0.3898	0.50
efficientnet_b0__None__lr0.0001	efficientnet_b0	None	0.0001	357	0.5856	0.75
resnet18__full	resnet18	None	0.0004	357	0.6425	0.50
resnet18__norm_fit	resnet18	norm_fit	0.0004	357	0.6756	0.50
resnet18__bit_fit	resnet18	bit_fit	0.0004	357	0.6765	0.50
mobilenetv3_small_100__full	mobilenetv3_small_100	None	0.0004	357	1.4883	0.75
efficientnet_b0__bit_fit	efficientnet_b0	bit_fit	0.0004	357	1.6278	0.75
efficientnet_b0__norm_fit	efficientnet_b0	norm_fit	0.0004	357	1.6315	0.75
mobilenetv3_small_100__bit_fit	mobilenetv3_small_100	bit_fit	0.0004	357	2.1502	0.75
mobilenetv3_small_100__norm_fit	mobilenetv3_small_100	norm_fit	0.0004	357	2.1502	0.75

Best model

efficientnet_b0 with full fine-tuning, every weight updated, learning rate 0.001, trained on 357 rows, so the author's augmented copies were used in training. Validation loss 0.080, validation accuracy 1.00.

Results on the test set (5 photos)

Metric	Value
Accuracy	1.000 (5/5)
Balanced accuracy	1.000
Macro F1	1.000
Solid recall / Stripe recall	1.00 / 1.00

95% Wilson interval on accuracy: [0.57, 1.00]. Per-photo predictions and the confusion matrix are in test_predictions.png; every run is logged in search_runs.csv.

Known failure modes and limitations

• Only 30 real balls from 2 sets, one table, similar lighting. Nothing here says how it behaves elsewhere.
• 5 test photos is a sanity check, not a measurement: one photo moves accuracy by 20 points.
• A solid ball's white number circle can look like a stripe's white area when it faces the camera.
• Heavy blur or contrast in the training copies can wash out the stripe band while the copy keeps the original label.
• The split is by photo, not by ball set, so this is not an evaluation on a completely unseen set.

Compute

Tesla T4, about 4 minutes for the whole search. Mixed precision is on and deterministic kernels are not forced, so a rerun with the same seed can still differ slightly.

How to use

import zipfile, pandas as pd
from huggingface_hub import hf_hub_download
from autogluon.multimodal import MultiModalPredictor
path = hf_hub_download("jackstev/hw2-pool-ball-solid-stripe", "autogluon_image_predictor_dir.zip")
zipfile.ZipFile(path).extractall("pool_ball_model")
predictor = MultiModalPredictor.load("pool_ball_model")
print(predictor.predict(pd.DataFrame({"image": ["/path/to/photo.png"]})))   # 0 = solid, 1 = stripe

Install autogluon.multimodal==1.6.1. Both artifacts use pickle internally, so only load ones you trust.

License

The dataset grants no reuse license, so I am not assigning one to these weights. Ask the dataset creator before using them outside class.

AI usage disclosure

I used Claude (Anthropic) to help write this notebook and draft this model card. I chose the dataset, ran the search, and checked the results and the card against the outputs.