indoor-outdoor-distilbert

distilbert-base-uncased fine-tuned with AutoGluon MultiModalPredictor to classify descriptions of places as indoor or outdoor. Selected from the three optim.peft settings covered in the Week 4 lecture.

Built for Homework 2 of 24-679 Designing with AI.

1. Purpose

Given a short description of a place, predict indoor or outdoor.

Intended use: coursework, and as the fine-tuning half of a fine-tuning-versus-prompting comparison. Out of scope: anything real. The evaluation set is 31 sentences from one annotator.

2. Data origin and splits

Source: srivathsanb14/indoor-outdoor-text, collected by a classmate for HW1. I am not the dataset author and did not contribute to its collection.

Split	Rows	Used for
train	1072	Fine-tuning
validation	16	Selecting the peft setting and early stopping
test	15	Untouched until the final evaluation

Most training rows are augmented variants of a smaller set of original sentences, so the effective sample size is well below 1072.

3. Method

The lecture's three optim.peft settings, run identically apart from the learning rate:

optim.peft	What updates in DistilBERT
bit_fit	Bias terms throughout the encoder. Main weight matrices stay frozen.
norm_fit	LayerNorm scales and offsets, plus other biases. Main weight matrices stay frozen.
None	Everything, including embeddings, attention and feed-forward weights.

Selected: peft = bit_fit.

peft	Learning rate	Validation macro-F1	Epochs run	Time
bit_fit	1e-03	1.0000	1	30 s
norm_fit	1e-03	1.0000	1	19 s
None	2e-05	0.9373	4	86 s

The learning rates differ by setting on purpose: a full fine-tune of a pretrained transformer needs a small step or it destroys what pretraining learned, while a bias-only fit updates so few parameters that the same rate would barely move the model. A single shared rate would have compared rates, not settings.

4. Training setup

Base model	distilbert-base-uncased
Library	AutoGluon MultiModalPredictor 1.6.3
Objective	eval_metric='f1_macro'
Epochs	up to 6, 1 run (early stopping, patience 3)
Learning rate	1e-03, 10 % warmup, cosine decay
Batch size	16 · max sequence length 128
Hardware	Tesla T4
Wall clock	30 s
Seed	20260922

5. Metrics

Metric	Value
Validation macro-F1 (selection, n=16)	1.0000
Test accuracy (n=15)	0.933
Test macro-F1 (n=15)	0.933
Validation + test accuracy (n=31)	0.968
Validation + test macro-F1 (n=31)	0.968
Errors on validation + test	1 of 31

Both n values are small. A 95 % confidence interval on 31 rows is roughly ±17 percentage points, so this model and another within that range are not distinguishable by these numbers.

6. Leakage audit

Training rows are augmented variants of a smaller original set, so an evaluation sentence may have a near-twin in training. Measured by TF-IDF cosine similarity to the nearest training sentence:

Exact duplicates in evaluation	0 of 31
Mean nearest-neighbour similarity	0.204
Evaluation rows above 0.90 similarity	0 of 31

The more rows above 0.90, the more the headline accuracy should be read as "recognises sentences like the ones it saw" rather than "understands indoor versus outdoor".

7. Limitations

1. The task may be close to saturated. Indoor and outdoor vocabulary barely overlap. A high score here is substantially a property of the dataset.
2. 31 evaluation rows. Too few to rank this model against another.
3. Single-annotator labels, with no independent check, and no defined convention for transitional spaces such as porches, greenhouses or covered markets.
4. Augmentation-shaped coverage. The model has seen a narrow neighbourhood around each original sentence thoroughly and everything else not at all.
5. English only, one annotator's register and vocabulary.
6. DistilBERT inheritance. Whatever biases the pretraining corpus carries are carried here, and a frozen-weight peft setting inherits them almost entirely.

8. Ethical considerations

The data describes places, not people, and no personal information is involved. The realistic harm is over-trust in a small model: an indoor/outdoor call from a classifier trained on a few hundred augmented sentences carries no authority outside this coursework.

9. License

CC-BY-4.0, matching the source dataset's terms. Attribution to the dataset author for the underlying text. Base model distilbert-base-uncased is Apache-2.0.

10. AI usage disclosure

Generative AI (Claude, Anthropic) was used as a coding and writing assistant: it drafted the AutoGluon configuration, the leakage audit, the plotting code and the prose of this card. The methodological choices — running all three peft settings rather than assuming one, separating the learning rates, auditing the evaluation set for near-duplicates, and reporting both n values — were reviewed and accepted by me, and every cell was executed and checked before upload. I am responsible for the content.

11. How to use

import pandas as pd
from huggingface_hub import snapshot_download
from autogluon.multimodal import MultiModalPredictor

predictor = MultiModalPredictor.load(snapshot_download("sunkaiwen/indoor-outdoor-distilbert") + "/predictor")
print(predictor.predict(pd.DataFrame({"text": ["A long corridor lined with lockers."]})))