leixiang25/handbag-attributes-dinov2

Handbag attribute recognizer (Model 1, DINOv2)

Predicts 6 attributes of a handbag (bag type, strap type, closure, structure, shape, view angle) from a line sketch. Part of a handbag sketch-to-render project (AI Systems course, Carnegie Mellon University).

Architecture. The image is cropped to the bag and resized to 224 px. Frozen facebook/dinov2-base features (CLS token + mean patch token, averaged with a mirrored copy, 1536-d) feed one scikit-learn pipeline per attribute: StandardScaler, PCA(128), LogisticRegression.

Training data. leixiang25/handbag-sketch-dataset-v2: 500 images (input: line sketch; front and three-quarter views) of 250 bags. Sketches are edge-traced line drawings of each product photo.

Evaluation. Grouped 5-fold cross-validation (a bag and its near-duplicates never appear in both training and test). Regularization C was chosen on these same folds, so scores may be very slightly optimistic (the fully nested comparison gave 0.56 mean). Baseline = always predicting the most common class. Lift = share of the gap between baseline and perfect that the model closes.

Attribute	Baseline	Accuracy	Macro F1	Lift
angle	0.50	0.96	0.96	0.93
bag_type	0.20	0.81	0.81	0.76
strap_type	0.28	0.74	0.71	0.64
closure	0.28	0.45	0.46	0.25
structure	0.44	0.52	0.51	0.14
shape	0.30	0.45	0.37	0.21

Mean accuracy over the 5 design attributes (excluding angle): 0.594 vs baseline 0.299.

Scope. Color, material, hardware and size are designer inputs, not predictions: a sketch carries no color, texture or metal finish, and every image is cropped to the bag, which removes absolute scale (a bag-type lookup predicted size better, 0.61, than the photo model, 0.53).

Backbone choice: on photos with the same CV, ImageNet ResNet-50, CLIP ViT-L/14 and DINOv2 ViT-B/14 were compared; DINOv2 was chosen for its stronger strap and closure recognition.

Limitations. Small training set (250 bags); sketches carry no color or texture, so material and hardware are hard to read from a sketch; material classes such as calfskin, smooth leather, lambskin and nappa overlap visually and are often confused. Best results on clean line drawings or studio photos with a plain light background.

Usage

import joblib
from huggingface_hub import hf_hub_download
B = joblib.load(hf_hub_download("leixiang25/handbag-attributes-dinov2", "heads.joblib"))
# features = 1536-d DINOv2 vector, see app.py in the Space for preprocessing
B["heads"]["bag_type"].predict_proba(features)

Requires scikit-learn==1.6.1.