tolmachf/tolmachv1.0

IELTS Writing Task 2 band scorer

A ModernBERT-base regressor fine-tuned to predict the five IELTS Writing Task 2 band scores. Input is the (question, essay) pair; output is five values in the 1–9 band range: Overall, Task Response, Coherence & Cohesion, Lexical Resource, and Grammatical Range & Accuracy.

Training data

114,349 essays from ndtran0101/writing9-ielts-essays. The full corpus holds 165,800 essays after cleaning, of which 163,286 carry labels for all five criteria. Evaluation uses a held-out split of 6,000 essays; synthetic (band9_augmented) rows are excluded from both.

Evaluation

Held-out split of the same source, raw predictions without calibration. ±0.5 is the share of predictions landing within half a band of the human label.

Criterion	MAE	±0.5	±1.0	Pearson r
Overall	0.658	50.3%	80.0%	0.772
Task Response	1.026	32.8%	58.3%	0.570
Coherence & Cohesion	0.847	43.0%	69.0%	0.602
Lexical Resource	0.717	54.2%	76.1%	0.802
Grammar	0.892	52.8%	70.8%	0.773

A TF-IDF + Ridge baseline trained on the identical split reaches Overall MAE 0.865 / r 0.625. The transformer wins on all five criteria; the widest gap is Grammar, where within-half-a-band accuracy triples (17.5% → 52.8%) — a bag-of-words model is blind to grammar by construction.

Context length was chosen empirically. At 448 tokens 15.8% of essays are truncated, losing the conclusion on average; the resulting damage falls specifically on the two discourse-level criteria. ModernBERT removes truncation entirely and beats a roberta-base/512 variant on all ten figures. The p99 length of a question+essay pair is 581 tokens, so max_len is set to 768 rather than ModernBERT's full 8192.

Limitations

Read these before using the model for anything that matters.

• Scale compression. Weak essays are scored up, strong ones scored down: +0.46 on bands 4–5.5 and −0.83 on bands 8–9. This is not a training bug — any MSE-minimising predictor with r < 1 must shrink toward the mean. calibration.json carries a variance-matching correction that cuts the skew to +0.30 / −0.56, at the cost of MAE 0.658 → 0.681. It was accepted for the Overall band only; for the other four criteria the MAE it costs exceeds what it fixes.
• Single-source evaluation. Training and test data come from one dataset. Performance outside that source has not been measured, and r = 0.77 should not be assumed to transfer.
• Domain shift on machine-written text. On essays written by an LLM the model scores roughly one band below a strong LLM examiner. It learned from human student writing, and generated text is partly out of distribution.
• Weakest criterion is Task Response (r = 0.57). Judging whether an essay actually answers the prompt requires understanding the prompt, and 114k noisily-labelled examples do not get there.
• Annotation noise ceiling. Measured on 84 pairs of byte-identical essays that received independent scores, human raters disagree by 0.393 bands on Overall — a single-rater error of about 0.278. The model is 2.4× worse than one human rater on Overall, and about 6× worse on Lexical and Grammar. The ceiling has not been reached on any criterion, but the ordering of criteria by model quality matches their ordering by label noise exactly.

Do not use this as the sole judge in any decision with consequences for a person. In the pipeline it was built for, it is one noisy signal among several and an LLM examiner makes the call.

Files

File	Purpose
model.pt	state_dict of the regressor
meta.json	base model, max_len, target order, eval report
modeling_band_regressor.py	model class + loader (required — see below)
calibration.json	scale-matching and domain correction coefficients
tokenizer.json, tokenizer_config.json	ModernBERT tokenizer

Usage

This checkpoint is a plain state_dict, not a transformers model, so AutoModel.from_pretrained will not open it. Use the bundled loader:

from huggingface_hub import snapshot_download
import sys

path = snapshot_download("CHANGE_ME/tolmach-ielts-scorer")
sys.path.insert(0, path)

from modeling_band_regressor import score

print(score(
    "Some people believe that unpaid community service should be a compulsory "
    "part of high school programmes. To what extent do you agree?",
    "Mandating community service in secondary education has become ...",
))
# {'band_overall': 5.0, 'band_task': 5.0, 'band_coherence': 5.0,
#  'band_lexical': 8.5, 'band_grammar': 8.5}

Use score_many(questions, essays) for batches. Outputs are rounded to the nearest half band, since IELTS does not issue intermediate values; pass through score_many and skip the rounding if you need the raw regression output.

That example output is worth reading as a warning rather than a demo: on a 150-word answer the model correctly punishes Task Response and Coherence for being far under the 250-word requirement, while still rating vocabulary and grammar at 8.5. The criteria move independently, and a high Lexical score says nothing about whether the essay answered the question.

About calibration.json

scale_matching holds the variance-matching correction fitted on student writing: band = mu_g + (prediction - mu_p) * scale.

domain holds a separate linear correction fitted on 146 machine-written essays against LLM-examiner scores. Two of its five criteria are degenerate and flagged as such in the file: band_coherence has a slope of −0.04 and band_task a slope of 0.14, meaning they predict a near-constant regardless of input. Their apparent MAE improvement is the model output being discarded, not the model getting better. Use band_overall, band_lexical and band_grammar from that block, and ignore the other two.

License

The training corpus ndtran0101/writing9-ielts-essays is published on the Hub with an unspecified license, so no permissive license is claimed here. If you intend to use these weights commercially, clear the rights on the source data first.

IELTS is a registered trademark of the British Council, IDP: IELTS Australia and Cambridge Assessment English. This model is an independent research artefact with no affiliation to, or endorsement by, any of them, and its output is not an official IELTS score.