toyin88/u2t01-bert-ner

CoNLL-2003 named entity recognition — feature adaptation

Part of U2T01: Adapting BERT for NLP tasks. One of four models, each one a different head on the same bert-base-uncased body, each delivered with the adaptation method that its own measurements justified.

This repository is not a fine-tuned transformer. BERT was used as a frozen feature extractor, so there is no task head inside the model weights. The repo contains the unmodified bert-base-uncased body, the fitted scikit-learn probe as probe.joblib, and inference.py showing the two-step call. AutoModelFor... will load the body but will not classify.

What this model does

Predicts O, B/I-PER, B/I-ORG, B/I-LOC, B/I-MISC (IOB2).

How it was adapted

Feature-based adaptation. The BERT body was frozen completely — zero of its parameters were updated. Its hidden states were extracted once and a classical classifier was fitted on top.

Base model	bert-base-uncased
Adaptation	feature
Trainable parameters	0 of 109,482,240 (0.00%)
Encoder layers trained	0 of 12
Head learning rate	0.001
Encoder learning rate	2e-05
Epochs	3.0
Batch size	32
Max sequence length	128
Seed	42
Training time	0.3 min on Tesla T4

A freshly initialised head and pretrained encoder layers were given separate learning rates through two optimiser parameter groups; a single learning rate for both silently cripples this kind of run.

Training data

CoNLL-2003 (lhoestq/conll2003). Training examples used: 1500.

Evaluation

Metric	Test	Validation
accuracy	95.70	96.75
macro_f1	82.34	84.72
f1	78.01	85.52
precision	74.52	83.50
recall	81.84	87.64
token_accuracy	95.70	96.75

Metric choice is argued in the project report. In short: entity-level F1 for NER because ~83% of CoNLL tokens are O and token accuracy would flatter a model that predicts nothing; token accuracy for POS because there is no dominant class; EM and F1 for SQuAD because each covers the other's blind spot.

Why this method, and what lost

Run	Method	Trainable params	f1 (test)
ner_feature_mini	feature	0	78.01
ner_full_mini	full	85,061,385	69.19
ner_partial_mini	partial	28,358,409	68.33

Decision rule: best primary metric on test, and where two methods fall inside a ±1 point band — smaller than the ±1–3 points a different random seed moves results by — the cheaper one wins.

Intended use and limitations

Intended for research and coursework: benchmarking adaptation strategies and as a starting point for related English tasks.

Not intended for production decisions about people. Concretely:

• English only, and the training text is narrow. AG News is 2004 newswire; CoNLL-2003 is Reuters newswire, so it under-performs on social media, names outside Western European conventions, and any entity that became prominent after 2003. UD EWT is web English (blogs, reviews, emails). SQuAD contexts are Wikipedia paragraphs, and the model always returns a span — SQuAD v1.1 contains no unanswerable questions, so it cannot say "I don't know".
• Single run, single seed. Re-running with a different seed moves results by roughly ±1–3 points. Differences smaller than that are not meaningful.
• Inherited bias. bert-base-uncased carries the associations of its pretraining corpus, and nothing here mitigates them.
• Sub-sampled training data, so these numbers sit below published full-data results. Every adaptation method on this task saw exactly the same data, so the comparison that chose this method is unaffected.

Reproduce it

git clone <this project>
pip install -r requirements.txt
python -m u2t01.cli run configs_mini/ner_feature.yaml

Fixed seed (42), pinned dependencies, and one YAML file per experiment: config + seed + requirements.txt fully determine a run.

References

• Devlin et al., BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, 2018 — https://arxiv.org/abs/1810.04805 (§5.3 is the feature-based vs fine-tuning comparison this project mirrors)
• Tunstall, von Werra & Wolf, Natural Language Processing with Transformers, O'Reilly, chapters 1–3
• Hugging Face, Fine-tune a pretrained model — https://huggingface.co/docs/transformers/training
• Dataset: lhoestq/conll2003

Card generated 2026-09-27 from results/ner_feature_mini.json.