CoLMLM-Fact-Span-Annotator

The fact-span annotator used to build the training corpora for Co-LMLM: Continuous-Query Limited Memory Language Models.

Co-LMLM is trained on text in which factual spans are wrapped in <FACT>...</FACT> tags. Producing those tags with a frontier LLM is far too expensive to run over a pretraining-scale corpus, so this model distills the span-marking half of that annotation into a ModernBERT-large token classifier: it labels every token O / B / I and thereby marks the fact spans directly.

It is the first stage of a two-stage annotation pipeline. The second stage, CoLMLM-Question-Generator, writes a question and a paraphrased answer for each span this model marks.

Model details

Base model	answerdotai/ModernBERT-large
Head	3-way token classification — O (0), B (1), I (2)
Precision	bfloat16
Training sequence length	4096 tokens (the backbone itself supports 8192)

Usage

For more details and the full annotation pipeline, see the code repository:

👉 github.com/lil-lab/Co-LMLM

Standalone, the model is an ordinary token classifier. Decoding its BIO output back into character spans:

import torch
from transformers import AutoModelForTokenClassification, AutoTokenizer

model_id = "lil-lab/CoLMLM-Fact-Span-Annotator"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForTokenClassification.from_pretrained(
    model_id, dtype=torch.bfloat16,
    # attn_implementation="flash_attention_2",  # faster, if flash-attn is installed
).eval()

text = "Marie Curie was born in Warsaw in 1867 and won two Nobel Prizes."

encoded = tokenizer(text, return_tensors="pt", return_offsets_mapping=True,
                    truncation=True, max_length=4096)
offsets = encoded.pop("offset_mapping")[0].tolist()
with torch.no_grad():
    predictions = model(**encoded).logits[0].argmax(-1).tolist()

spans, start, end = [], None, None
for (char_start, char_end), label in zip(offsets, predictions):
    if char_start == char_end:  # special token
        continue
    tag = model.config.id2label[label]
    if tag == "B":
        if start is not None:
            spans.append((start, end))
        start, end = char_start, char_end
    elif tag == "I" and start is not None:
        end = char_end
    else:
        if start is not None:
            spans.append((start, end))
        start = None
if start is not None:
    spans.append((start, end))

print([text[s:e].strip() for s, e in spans])
# ['Warsaw', '1867', 'two Nobel Prizes']

This model is part of the Co-LMLM collection.

Citation

@misc{feldman2026colmlmcontinuousquerylimitedmemory,
      title={Co-LMLM: Continuous-Query Limited Memory Language Models},
      author={Yair Feldman and Linxi Zhao and Nathan Godey and Dongyoung Go and Yilun Hua and Kilian Q. Weinberger and Jennifer J. Sun and Yoav Artzi},
      year={2026},
      eprint={2607.07707},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2607.07707},
}