inovruzova/polygraf-ner-stage3-model

Polygraf / NER Stage 3: First Model

google/electra-small-discriminator fine-tuned end to end for token classification on inovruzova/polygraf-ner-stage2-expanded. All layers were updated, not only the classification head.

This is the Stage 3 output of the Polygraf Applied NLP / NER technical task. It is a first model, published with its faults documented rather than tuned away.

Labels

Eight entity types, seventeen tags including O.

PERSON, ORGANIZATION, LOCATION, TIMEDATE, PRODUCT, WORKOFART, JOB, AMOUNT

The full annotation policy the labels follow is on the dataset card.

Usage

from transformers import pipeline

ner = pipeline(
    "token-classification",
    model="inovruzova/polygraf-ner-stage3-model",
    aggregation_strategy="simple",
)

ner("The software engineer at Google filed the patent in March 2024.")

Model facts

Base model	google/electra-small-discriminator
Parameters	13,487,377
Size on disk	52.2 MB
CPU memory at inference	[MEASURE] MB
Throughput on CPU	16.3 records per second, 61.5 ms per record

Throughput measured one passage at a time on CPU, after warm-up, over 100 passages from the test split.

Training

Hardware	NVIDIA RTX 4070 Laptop GPU, 8 GB VRAM
Learning rate	1e-4
Batch size	8
Epochs	30
Weight decay	0.01
Maximum sequence length	512
Seed	202608
Runtime	131 seconds

No validation split, following the task instruction, and therefore no checkpoint selection.

An earlier run used the Hugging Face defaults of 5e-5, batch 16 and 10 epochs and reached test F1 0.491. It was undertrained rather than overfitted: train F1 was 0.583 and the loss was still falling at the last step. The settings above were chosen from that evidence, all of which comes from the training split. 5e-5 suits BERT base at 110M parameters; this model is 13.5M.

Results

Micro-averaged F1: 0.745 on test, 0.999 on train.

Test, per label

Label	Precision	Recall	F1	Support
TIMEDATE	0.831	0.920	0.873	112
PERSON	0.805	0.901	0.850	101
AMOUNT	0.809	0.821	0.815	67
LOCATION	0.783	0.837	0.809	86
JOB	0.719	0.708	0.713	65
PRODUCT	0.578	0.673	0.622	55
ORGANIZATION	0.500	0.600	0.545	65
WORKOFART	0.480	0.490	0.485	49
micro avg	0.715	0.778	0.745	600

On train, every label scores between 0.997 and 1.000.

Limitations

The model memorises its training set. Train 0.999 against test 0.745 is a gap of 0.254. This is left uncorrected rather than fixed by shortening training, because with no validation split, shortening until the test gap narrows would be selecting a training length on test performance.

Three labels are much weaker than the rest. WORKOFART, ORGANIZATION and PRODUCT are exactly the labels governed by the four policy rules that need knowledge outside the token sequence: whether a spec-like token belongs to a product's commercial name, whether a brand is acting as a company, whether a title refers to a game or a film, and whether a place name stands for the institution it houses. Scarcity does not explain it; ORGANIZATION is the fourth most common label in the dataset.

The model is uncased. It never sees capitalisation, which is the strongest surface clue that a word is a name in English. Every model below about 35M parameters is uncased and the smallest cased one is five times larger, so this is the cost of choosing the smallest reasonable model.

It over-predicts. Recall exceeds precision on seven of eight labels.

Test metrics are agreement, not accuracy. The test labels were themselves measured at 98.3 percent span precision in a 40-record hand audit, so these numbers compare the model against an imperfect reference.

Two entities in the training data cannot be represented as token tags, because the policy places a currency symbol outside the span and BERT-family tokenizers split $ from a following digit but not £.

Known failure modes

Ten patterns were found by running the model on 32 constructed samples outside the dataset, each written to exercise one policy rule. Examples in full are in the project report. In summary: generic nouns labeled as named entities; vague time expressions labeled; boundaries falling inside words; single entities split into two differently labeled spans; entities missed; possessives dropped from establishment names; dish preparations labeled; nationality adjectives kept inside spans inconsistently; adjacent time expressions merged; adverbial phrases labeled as jobs.

At least one of these is inherited rather than learned wrongly. The training data itself contains nationality adjectives treated inconsistently, so the model reproduces that faithfully.

Related repositories

Stage	Artefact
1	inovruzova/polygraf-ner-stage1-cleaned
2	inovruzova/polygraf-ner-stage2-expanded

Licensing

The model weights are released under Apache 2.0, matching the base model.

The training data is a collection of corpora under different terms, including share-alike obligations from ODbL and CC BY-SA. Those obligations attach to the dataset rather than to a set of trained weights, but anyone reusing this model should read the dataset card.

Produced during the Polygraf Applied NLP / NER technical task.