shanexf/24679-garments-distilbert

Garment-category classifier — DistilBERT fine-tuned with AutoGluon MultiModal (24-679, Fall 2026)

Purpose

Classifies a short English product description of a garment into one of five categories: top, bottom, outerwear, dress, footwear. Built for the 24-679 "fine-tuning a small text model" homework. It is trained on 70 original descriptions written by one author in a deliberately consistent listing style, and is a classroom artefact rather than a general product-categorisation model.

Data

• Source dataset: leixiang25/24679-hw1-text-garments (commit 49d58b5a09420224a7d4d00fd983b9febd260f8f), a classmate's Homework 1 dataset: 100 original, fictional descriptions (164–208 characters) written while viewing retail photos, with no retailer text, brand names or personal data; no license assigned by the dataset card.
• Splits (published, kept fixed): train = 70 originals + 1289 synthetic rows (1359 rows), validation = 15 originals, test = 15 originals. Holdouts are original-only and no parent_id / message_id crosses a split (asserted in the notebook). Class balance in train originals: {'footwear': 14, 'dress': 14, 'top': 14, 'outerwear': 14, 'bottom': 14}.
• Training augmentation (in the dataset): interior character swaps, interior character deletions, within-sentence word swaps, WordNet synonym replacement (garment nouns protected), 5 draws per method, never stacked.

Preprocessing

Raw text strings are passed to AutoGluon's hf_text pipeline: distilbert-base-uncased WordPiece tokenizer (lower-casing, [CLS]/[SEP]), truncation/padding to at most 128 tokens. No cleaning, stemming or stop-word removal. Labels are integers 0–4 as in the dataset.

Training setup

• Framework: AutoGluon MultiModal 1.6.1 MultiModalPredictor(problem_type="multiclass", eval_metric="accuracy"), encoder distilbert-base-uncased (66 M parameters, 6 layers) with a new 5-way head; transformers 5.14.1, PyTorch 2.11.0+cu128.
• Fine-tuning modes compared on the validation split (AutoMM optim.peft; the head always trains):

	validation accuracy	validation macro F1	validation weighted F1	trainable_parameters	total_parameters	trainable %	fit_seconds
bit_fit	1	1	1	55301	66366725	0.0833264	25.8
norm_fit	1	1	1	65285	66366725	0.0983701	12.7
full	1	1	1	66366725	66366725	100	66

• Selected mode: bit_fit — 55,301 trainable of 66,366,725 parameters. Hyperparameters of the selected run (all explicit):

{
  "model.names": [
    "hf_text"
  ],
  "model.hf_text.checkpoint_name": "distilbert-base-uncased",
  "model.hf_text.max_text_len": 128,
  "model.hf_text.text_trivial_aug_maxscale": 0.0,
  "optim.peft": "bit_fit",
  "optim.lr_choice": "layerwise_decay",
  "optim.lr": 0.001,
  "optim.lr_decay": 0.9,
  "optim.weight_decay": 0.01,
  "optim.warmup_steps": 0.1,
  "optim.lr_schedule": "cosine_decay",
  "optim.max_epochs": 6,
  "optim.patience": 3,
  "optim.val_check_interval": 1.0,
  "optim.top_k": 1,
  "optim.top_k_average_method": "best",
  "env.per_gpu_batch_size": 16,
  "env.batch_size": 16
}

i.e. AdamW, learning rate 0.001 with layer-wise decay 0.9, 10 % linear warm-up then cosine decay, weight decay 0.01, batch size 16, up to 6 epochs with validation at the end of each epoch, early stopping with patience 3 on validation accuracy, single best checkpoint kept, AutoMM text augmentation off, mixed precision on GPU.

• Seed: 24679 (AutoGluon seed= plus Python / NumPy / PyTorch). Fit summary of the selected run: {'val_accuracy': 1.0, 'training_time': 25.142484664916992}; wall-clock 25.8 s within a 900 s limit (104.5 s for all three modes).
• Selection / evaluation: validation split selects the mode and the checkpoint; the test split was scored once, afterwards.

Metrics

Holdout scores (DistilBERT vs. TF-IDF + logistic-regression baseline trained on the same rows)

	accuracy	macro F1	weighted F1
('DistilBERT (fine-tuned)', 'validation')	1	1	1
('DistilBERT (fine-tuned)', 'test')	1	1	1
('TF-IDF + logistic regression', 'validation')	0.933	0.931	0.931
('TF-IDF + logistic regression', 'test')	1	1	1

Per-class precision / recall / F1 on the test split

	precision	recall	f1-score	support
top	1	1	1	3
bottom	1	1	1	3
outerwear	1	1	1	3
dress	1	1	1	3
footwear	1	1	1	3
accuracy	1	1	1	1
macro avg	1	1	1	15
weighted avg	1	1	1	15

• Test accuracy bootstrap 95 % CI: 1.000–1.000 (2000 resamples of 15 descriptions; one description = 6.7 points).

Robustness probes (diagnostics on the test descriptions, not the headline metric)

Accuracy under the dataset's noise methods applied to the test texts (5 draws each) and with garment nouns masked

probe	rows	distilbert_accuracy	baseline_accuracy	share_noun_hit
garment_nouns_masked	15	0.733	0.933	0.933
internal_character_deletions	75	0.96	0.987	0.16
internal_character_swaps	75	0.947	1	0.12
within_sentence_word_swap	75	1	0.987	0
clean test (reference)	15	1	1	0

Noise rows split by whether the perturbation altered a garment noun

noun_corrupted	rows	distilbert_accuracy	baseline_accuracy
noise missed the garment noun	204	0.985	0.99
noise hit the garment noun	21	0.81	1

Error analysis (excerpt; full table in error_analysis.csv)

• garment_nouns_masked — actual dress, predicted top (P = 0.97): “Fitted mini item in mahogany brown stretch jersey, with a scoop neckline, narrow Y-back straps and a supportive inner shelf, finished with smooth bound edges and a clean straight hem.”
• internal_character_deletions — actual dress, predicted top (P = 0.93): “Fitted high-nck mini dres in black buttery strech jersey, with racerback shoulders, curved bust seams and an integrated short liner, finished with bonded edges and a clean above-knee hem.”
• internal_character_swaps — actual dress, predicted top (P = 0.90): “Fitted high-neck mini drses in black buttery stretch jersey, with racerback shoulders, curved bust seams and an intgerated shrot liner, finished with bonded edges and a clean above-knee hem.”
• internal_character_deletions — actual dress, predicted top (P = 0.86): “Fitted high-neck mini dess in black buttery stretch jersey, with racerback shoulders, curved bust sams and an integrated short liner, finished with bonded edges and a cean above-knee hem.”
• garment_nouns_masked — actual dress, predicted top (P = 0.70): “Fitted high-neck mini item in black buttery stretch jersey, with racerback shoulders, curved bust seams and an integrated short liner, finished with bonded edges and a clean above-knee hem.”
• internal_character_swaps — actual bottom, predicted top (P = 0.63): “High-rise cropped lgegings in black buttery stretch jersey, with a seamless front, sculpted back seams and a close second-skin fit, finsihed wtih a wide pull-on waistband and calf-length hems.”
• internal_character_swaps — actual bottom, predicted top (P = 0.63): “High-rise cropped leggings in black buttery stretch jersey, with a seamless front, sculpetd back seams and a close second-skin fit, finished with a wide pull-on waitsband and calf-length hems.”
• internal_character_deletions — actual bottom, predicted top (P = 0.51): “High-rise cropped leggings in black buttery stretch jersey, with a seamless front, sculped back seams and a close second-skin fit, finished with a wide pullon waitband and calf-length hems.”

Hypotheses examined in the notebook: (1) the garment noun carries most of the signal, so masking it costs far more than noise that misses it; (2) character noise that hits the noun breaks WordPiece tokenisation and flips the prediction; (3) residual confusions sit on the top/outerwear and top/dress boundaries, while footwear is stable because its vocabulary is disjoint; (4) word swaps are nearly harmless because the token set is unchanged.

Limitations and ethical notes

• Single author, single template. All descriptions were written by one person in one consistent format; the model has learned that format and may not transfer to real retailer listings, other languages, or terse / noisy user text.
• Easy by design. The dataset card states that classifiers can rely on garment nouns alone; the probes above quantify how much this model does.
• Tiny holdouts. 15 validation and 15 test descriptions; a single error moves accuracy by 6.7 points. Synthetic training rows add variation, not new garments.
• No personal data, no brands. Descriptions are fictional; the model should not be used for any consequential decision (pricing, moderation, compliance) and encodes one author's garment vocabulary and conventions.
• GPU fine-tuning is not bit-reproducible; reruns with the same seed can differ slightly.

License

The dataset card assigns no license and the data were shared for 24-679 coursework; this model is released under the same classroom-use-only terms (license: other). distilbert-base-uncased itself is Apache-2.0.

Hardware and compute budget

Linux-6.6.122+-x86_64-with-glibc2.39, Python 3.13.15, 2 CPU cores, 13.6 GB RAM, GPU: Tesla T4. Fine-tuning budget 900 s per mode (selected run 25.8 s, all three modes 104.5 s, ≤ 6 epochs each); probe inference on 240 rows. Run started 2026-09-21T20:28:48+00:00.

How to use

Install automm/requirements.txt (Python 3.13.15), then:

from pathlib import Path
import pandas
from huggingface_hub import snapshot_download
from autogluon.multimodal import MultiModalPredictor

snapshot = Path(snapshot_download(repo_id="shanexf/24679-garments-distilbert", allow_patterns=["automm/*"]))
model = MultiModalPredictor.load(str(snapshot / "automm"))
model.set_num_gpus(0)
labels = {0: 'top', 1: 'bottom', 2: 'outerwear', 3: 'dress', 4: 'footwear'}
frame = pandas.DataFrame({"text": ["Relaxed crew-neck sweater in heather grey merino, with dropped shoulders and ribbed cuffs, finished with a straight hem."]})
print(labels[int(model.predict(frame)[0])], model.predict_proba(frame))

Pin revision= to the commit recorded in metrics.json to reload exactly this version. Partial-tuning modes may re-download the distilbert-base-uncased backbone files during loading.

AI usage disclosure

Claude (Anthropic) was used to adapt the 24-679 lecture fine-tuning notebook to this text dataset, to implement the robustness probes, and to draft the model-card template. The author ran the notebook end to end, checked the outputs, and wrote the error-analysis conclusions. The garment descriptions were written by a classmate; none were generated by AI.

Author

shanexf — 24-679, Carnegie Mellon University, Fall 2026.