freelawproject/caselaw-block-tagger-wordlevel

Case-law block tagger — word-level

A token-classification model that labels the structural components of a U.S. case-law opinion: given the OCR'd text of an opinion as lightly-marked-up HTML, it tags the caption and headmatter elements and the opinion skeleton as character spans.

Classes (12, BIO): party, separator, docketnumber, court, attorneys, judges, datefiled, otherdate, history, disposition, author, heading. Everything else (body prose) is O.

This is the second release of the tagger. It continues freelawproject/caselaw-block-tagger (the first release, called "the previous model" below) and must be decoded differently: one prediction per word, not per token. See Differences from the previous model and Usage.

Built by the Free Law Project for structuring scanned case-law reporters; also applicable to born-digital opinion text serialized to the same input format.

Differences from the previous model

The previous model was trained with a label on every subword piece and decoded per token. On real scanned opinions that produced a family of faults with one root cause: the byte-level tokenizer cuts words into pieces (TR|UST, R|HO|DE) and the pieces of one word could carry different labels. Spans then opened inside the first word of a paragraph, flipped inside capitalised words, flickered word by word on long lines, and dispositions came back in fragments. A second gap: the training text held raw & while the scanning pipeline sends &, a form the model had never seen.

	previous model	this model
labels at training	every subword piece	the first piece of each word; the rest unsupervised
decoding	per token, BIO runs over all pieces	per word: read the label of each word's first piece, extend spans to whole words
back-to-back same-label spans	separate	shown to the model as one span; it returns one
&amp; / &lt; / &gt;	never seen	an entity-escaped twin of every training window that holds &, < or >
big-set labels	model-seeded, human-reviewed once	the same pages, re-cleaned by scripted rules and a second human review (about one span in eleven changed: fragments, cross-word cuts, vote-line and disposition conventions)
multi-docket captions	2 training windows with a run of docket lines	plus 650 synthetic captions covering five docket-line shapes
post-processing	none	a disposition heal: several disposition spans inside one paragraph become one span over the paragraph
initialization	golden-set fine-tune	the previous model

What stayed the same: the architecture, the input format, the 12 classes and label ids, the context length, and the evaluation sets.

Model details

• Architecture: ModernBERT-large (0.4B), the previous model's weights continued; ultimately fine-tuned from CaseLawModernBERT-large (continued-pretrained on 13B words of U.S. court opinions), 8,192-token context.
• Input format: one court case per sequence, as minimal HTML — <p> per text block, plus <blockquote>, <em>, <sup>; footnote content omitted. One \n between blocks. Ampersands and angle brackets may be raw or escaped (&); both were trained. The model never generates text; it emits per-token BIO labels (12 classes × begin/inside + outside = 25 label ids), of which only the label on the first token of each word is meaningful.
• Training data: the previous model's data — about 4,400 pages of scanned U.S. reporter volumes (16 volumes across 8 reporter series), OCR'd through a multi-engine pipeline, with a hand-annotated golden set (about 680 pages) and a model-seeded, human-reviewed extension (about 3,900 pages) — with the extension's labels cleaned and reviewed again for this release, plus the synthetic multi-docket captions. Augmentations: random block-boundary drops (OCR segmentation robustness), styling dropout (<em>/<sup>-stripped copies) and entity-escaped twins. 6,933 training windows after augmentation.
• Training: two continued-training stages from the previous model, both with word-level labels, lr 3e-5 then 1e-5, bf16, best epoch by span-F1 on a mixed validation set, early stopping. The first stage (reviewed gold + entity twins) ran 18 epochs with the best at 10; the second (plus the synthetic docket captions) 5 epochs with the best at 2.

Evaluation

Span-level F1: exact (start, end, label) sets, gold and predictions normalized identically (whitespace/markup edge anchoring; the "normalized" column additionally ignores edge punctuation). Gold spans are merged the way the model sees them (back-to-back same-label spans count as one). Predictions are decoded per word and healed as described under Usage; no other post-processing. All evaluation volumes are unseen in training. The previous model is scored with its own per-token decoding, as deployed.

eval set	previous model	this model
golden validation (39 cases, hand-annotated, born-digital)	0.968 / 0.975	0.999 / 0.999
big-set validation (3 volumes, 142 cases, human-reviewed gold)	0.929 / 0.935	0.969 / 0.970
scanning-portal set (13 opinions, 21 windows, in the pipeline's own input shape, reviewed gold)	0.830 / 0.901	0.974 / 0.981

Strict / normalized. The gain is precision: on the portal set the previous model returned 435 spans for 363 gold (precision 0.761), this one 356 (precision 0.983). Recall moved from 0.912 to 0.964.

Per class on big-set validation (strict, previous → this model): disposition 0.801 → 0.958 · history 0.815 → 0.933 · judges 0.865 → 0.952 · docketnumber 0.949 → 0.986 · heading 0.938 → 0.964 · court 0.963 → 0.977 · author 0.968 → 0.976 · attorneys 0.984 → 0.988 · datefiled 0.982 → 0.977 · party 0.990 → 0.990 · separator 0.994 → 0.994 · otherdate 0.842 → 0.831 (28 gold spans).

On the portal set (strict): history 0.486 → 0.963 · attorneys 0.741 → 1.000 · judges 0.743 → 0.929 · heading 0.804 → 0.961 · docketnumber 0.973 → 1.000 · party 0.963 → 1.000 · court 0.923 → 1.000 · datefiled 0.889 → 1.000 · author 0.909 → 1.000 · disposition 0.800 → 0.815. Several portal classes have 4 to 18 gold spans, so one span moves a class by 5 to 25 points; big-set validation is the stable measure.

Caption classes (party, separator, docketnumber, court, datefiled, otherdate — typically the primary extraction targets), micro-F1 over the six, strict / normalized: big-set validation previous model 0.971 / 0.973, this model 0.977 / 0.977 (627 gold spans); portal set previous model 0.959 / 0.971, this model 1.000 / 1.000 (121 gold spans).

The held-out test set of the previous release has not been scored for this one yet.

Usage

The label on a word's first token is the word's label; the labels on its other pieces were never supervised and must be ignored. Decode per word, then heal dispositions:

import re
from transformers import AutoModelForTokenClassification, AutoTokenizer

repo = "freelawproject/caselaw-block-tagger-wordlevel"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForTokenClassification.from_pretrained(repo)
names = [model.config.id2label[i] for i in range(len(model.config.id2label))]

text = (
    "<p>Jane ROE, Appellant,</p>\n"
    "<p>v.</p>\n"
    "<p>STATE of Example, Appellee.</p>\n"
    "<p>Nos. 24-123, 24-124</p>\n"
    "<p>Court of Appeals of Example.</p>\n"
    "<p>January 1, 2026</p>\n"
    "<p>Affirmed.</p>"
)
enc = tok(text, return_tensors="pt", return_offsets_mapping=True)
logits = model(**{k: v for k, v in enc.items() if k != "offset_mapping"}).logits
ids = logits.argmax(-1)[0].tolist()
offsets = enc["offset_mapping"][0].tolist()

MARKUP = re.compile(r"</?(?:p|blockquote|footnotes|em|sup)>")
PAGE_MARK = re.compile(r"^(?:↓\d+|\d{2,4}(?=[(\[]))")   # a page number glued to a word

def decode_word_spans(text, offsets, ids, names):
    mask = bytearray(ch not in " \n\t\r" for ch in text)
    for m in MARKUP.finditer(text):
        mask[m.start():m.end()] = bytes(m.end() - m.start())
    words, start = [], None                     # (start, end) of every word
    for i, on in enumerate(mask):
        if on and start is None: start = i
        elif not on and start is not None: words.append((start, i)); start = None
    if start is not None: words.append((start, len(text)))
    split = []
    for s, e in words:
        m = PAGE_MARK.match(text[s:e])
        split += [(s, s + m.end()), (s + m.end(), e)] if m and m.end() < e - s else [(s, e)]
    word_of = [-1] * len(text)
    for k, (s, e) in enumerate(split):
        for c in range(s, e): word_of[c] = k
    head = [None] * len(split)                  # first token of each word
    for t, (ts, te) in enumerate(offsets):
        first = next((c for c in range(ts, min(te, len(text))) if mask[c]), None)
        if first is not None and head[word_of[first]] is None: head[word_of[first]] = t
    spans, cur = [], None
    for w, (s, e) in enumerate(split):
        t = head[w]
        if t is None: continue
        tag = names[ids[t]]
        if tag == "O": cur = None; continue
        prefix, label = tag.split("-", 1)
        if prefix == "I" and cur and cur["label"] == label: cur["end"] = e; continue
        cur = {"label": label, "start": s, "end": e}; spans.append(cur)
    return spans

spans = decode_word_spans(text, offsets, ids, names)
for s in spans:
    print(s["label"], repr(text[s["start"]:s["end"]]))

Disposition heal. If two or more disposition spans fall inside one <p> block, replace them with one span over the block's inner text (the sentence came back in pieces; a dispositive paragraph is dispositive as a whole). The evaluation above includes this step.

For best results segment the input at case boundaries (one case per sequence, ≤8k tokens) and strip footnote content — the model was trained that way. Decoding per token, as for the previous model, is not supported for these weights.

Limitations

• Trained on U.S. reporter typography; other layout traditions (official state reporters, foreign case law) are untested.
• disposition remains the hardest class: a dispositive sentence inside a long paragraph of argument can be missed, and single-word rulings on their own line (Affirmed.) are tagged with the period where some annotation conventions leave it out.
• Single-letter and digit-only section headings (C, (1)) are inconsistently tagged; otherdate has thin support; signature-block dates (DATED this 24th day of …) may be tagged datefiled.
• Citations are deliberately NOT a class — they are handled by a separate model.
• Input must follow the canonical serialization above; markup tokens are label-masked during training, so unexpected markup degrades output.