CoLMLM-Question-Generator

The question generator used to build the training corpora for Co-LMLM: Continuous-Query Limited Memory Language Models.

Co-LMLM is trained on text in which each factual span carries the question it answers. Producing those questions with a frontier LLM is far too expensive to run over a pretraining-scale corpus, so this model distills that step: given a document whose fact spans are already marked and numbered, and the id of one of them, it emits the question that span answers plus a paraphrased answer.

It is the second stage of a two-stage annotation pipeline. The first stage, CoLMLM-Fact-Span-Annotator, marks the spans this model is asked about.

This repository contains a LoRA adapter, not a standalone model — the base weights are loaded from Qwen/Qwen2.5-1.5B-Instruct at inference time.

Model details

Base model	Qwen/Qwen2.5-1.5B-Instruct
Adaptation	LoRA, r=64, α=64, dropout 0.05, on all linear projections (q,k,v,o,gate,up,down)
Also trained	embeddings for the 8 added annotation tokens (<FACT>, </FACT>, <FACT_ID>, <QUESTION>, </QUESTION>, <ANSWER>, </ANSWER>, <DOC_SEP>)
Precision	bfloat16
Sequence length	8192 tokens

Prompt format

The user message is the numbered document, then <DOC_SEP>, then the question for one fact id. The chat template is Qwen's default (no system prompt is supplied, so Qwen's default system block is used — matching training).

<document with <FACT>N<FACT_ID>span</FACT> tags><DOC_SEP>

What are the question and paraphrased answer for <FACT>N<FACT_ID>?

The model responds with <QUESTION>...</QUESTION><ANSWER>...</ANSWER>.

Usage

For more details and the full annotation pipeline, see the code repository:

👉 github.com/lil-lab/Co-LMLM

Standalone:

import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

adapter_id = "lil-lab/CoLMLM-Question-Generator"
base_id = "Qwen/Qwen2.5-1.5B-Instruct"

tokenizer = AutoTokenizer.from_pretrained(adapter_id)
model = AutoModelForCausalLM.from_pretrained(base_id, dtype=torch.bfloat16)
model = PeftModel.from_pretrained(model, adapter_id).eval()

context = ("Marie Curie was born in <FACT>1<FACT_ID>Warsaw</FACT> in "
           "<FACT>2<FACT_ID>1867</FACT> and won <FACT>3<FACT_ID>two</FACT> Nobel Prizes.")
fact_id = 1

user = (f"{context}<DOC_SEP>\n\n"
        f"What are the question and paraphrased answer for <FACT>{fact_id}<FACT_ID>?\n")
prompt = tokenizer.apply_chat_template([{"role": "user", "content": user}],
                                       add_generation_prompt=True, tokenize=False)

inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
with torch.no_grad():
    output = model.generate(**inputs, max_new_tokens=64, do_sample=False)
print(tokenizer.decode(output[0, inputs["input_ids"].shape[1]:], skip_special_tokens=False))
# <QUESTION>Where was Marie Curie born?</QUESTION><ANSWER>Warsaw</ANSWER><|im_end|>

This model is part of the Co-LMLM collection.

Citation

@misc{feldman2026colmlmcontinuousquerylimitedmemory,
      title={Co-LMLM: Continuous-Query Limited Memory Language Models},
      author={Yair Feldman and Linxi Zhao and Nathan Godey and Dongyoung Go and Yilun Hua and Kilian Q. Weinberger and Jennifer J. Sun and Yoav Artzi},
      year={2026},
      eprint={2607.07707},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2607.07707},
}