DocSEAM-Qwen3.5-9B

DocSEAM-Qwen3.5-9B is a multi-page document vision-language model based on Qwen3.5-9B. It inserts Document Selective Enrichment via Asymmetric Masking (DocSEAM) after Qwen's image-token merger and before the language model. Each page is enriched with its previous and next page while keeping the visual token count unchanged.

The checkpoint is a standalone merged model: users do not need the Stage-A DocSEAM checkpoint, the Stage-B LoRA adapter, or LLaMA-Factory for inference.

Transformers inference

Requires Python 3.11+, PyTorch 2.6+, and Transformers 5.6+.

pip install "torch>=2.6.0" "transformers>=5.6.0" "accelerate>=1.11.0" pillow safetensors

import torch
from PIL import Image
from transformers import AutoModelForImageTextToText, AutoProcessor

model_id = "mossbee/DocSEAM-Qwen3.5-9B"
processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForImageTextToText.from_pretrained(
    model_id,
    trust_remote_code=True,
    dtype=torch.bfloat16,
    device_map="auto",
    use_safetensors=True,
).eval()

# Keep pages in their original reading order. DocSEAM treats adjacent images as
# adjacent pages of the same document.
pages = [Image.open("page_1.png").convert("RGB"),
         Image.open("page_2.png").convert("RGB")]
messages = [{
    "role": "user",
    "content": [
        *[{"type": "image", "image": page,
           "min_pixels": 32768, "max_pixels": 524288} for page in pages],
        {"type": "text", "text": "What continues across these two pages?"},
    ],
}]

text = processor.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
    enable_thinking=False,
)
inputs = processor(
    text=[text], images=pages, padding=True, return_tensors="pt"
).to(model.device)

with torch.inference_mode():
    output_ids = model.generate(**inputs, max_new_tokens=2048, do_sample=False)

generated = output_ids[:, inputs.input_ids.shape[1]:]
answer = processor.batch_decode(
    generated, skip_special_tokens=True, clean_up_tokenization_spaces=False
)[0]
print(answer)

The model automatically infers the number of pages in each batch item from the processed prompt. Do not mix pages from different documents in one prompt.

Architecture

DocSEAM uses two post-merger transformer layers over local page triplets. An asymmetric attention mask blocks direct interaction between the previous and next pages, while four learned directional gates control cross-page information flow. The center page can read both neighbors; only the enriched center-page tokens are returned. First and last pages use themselves as the missing neighbor. The released checkpoint uses content-frontier positional features.

Training

Training used two stages:

1. Stage A trained the DocSEAM module while freezing the Qwen3.5-9B backbone.
2. Stage B froze DocSEAM and trained rank-16 LoRA adapters plus the visual merger on multi-page document reasoning and document-understanding data.

The public weights merge the Stage-B adapter into the backbone and include the Stage-A DocSEAM parameters under model.docseam.

Limitations

• Pages must be supplied in reading order.
• DocSEAM models local adjacency with a three-page window; it is not a document retriever and does not replace long-range reasoning in the language model.
• Performance may degrade on documents, languages, layouts, or image quality unlike the training data.
• Outputs can be incorrect or fabricated. Verify answers before consequential use.
• The custom architecture requires trust_remote_code=True.

Authors

Hoang-Nhat Nguyen and Chen Change Loy, S-Lab, Nanyang Technological University, Singapore.

License

Apache License 2.0. This release is derived from Qwen3.5-9B; see the included license and the base model card.

Citation

The accompanying paper and final citation will be added when publicly available.