mayocream/koharu-text-sam-ts-l

Koharu Text SAM-TS-L

koharu-text-sam-ts-l is a ViT-L Hi-SAM/SAM-TS TextSeg model fine-tuned for binary text segmentation on manga pages. It detects text pixels rather than speech balloons, panels, or individual text lines.

The model was fine-tuned from the SAM-TS-L TextSeg checkpoint from Hi-SAM. The released weights use SafeTensors only; no pickle checkpoint is required for the full model.

Files

File	Contents
model.safetensors	Complete merged model state: 728 tensors, 338,934,897 elements
adapter_model.safetensors	Fine-tuned trainable state only: 366 tensors, 30,650,149 elements
config.json	Architecture, preprocessing, and training configuration
evaluation.json	Aggregate book-disjoint evaluation results
inference.py	Minimal loading and full-page inference example

Use model.safetensors unless you specifically need the smaller adapter state. The full file loads strictly into a freshly constructed SAM-TS-L model and does not require the original SAM or TextSeg .pth files.

Results

Metrics use micro-averaged binary pixel counts. HR refers to the high-resolution SAM-TS output.

Split	Model	HR IoU	HR F-score	HR precision	HR recall
Validation, 50 pages / 5 books	Original TextSeg	0.4138	0.5854	0.8788	0.4388
Validation, 50 pages / 5 books	Koharu Text SAM-TS-L	0.7062	0.8278	0.8513	0.8056
Test, 40 pages / 4 unseen books	Original TextSeg	0.3461	0.5142	0.8365	0.3712
Test, 40 pages / 4 unseen books	Koharu Text SAM-TS-L	0.6285	0.7719	0.7786	0.7653

The fine-tuned model improved HR IoU on 39 of the 40 test pages. Test HR IoU ranged from 0.5616 to 0.6977 across the four held-out books.

Training data

Training used the masks from Mask Dataset for: Unconstrained Text Detection in Manga: a New Dataset and Baseline by Julian del Gobbo and Rosana Matuk Herrera, licensed CC BY 4.0, paired locally with authorized Manga109 images. Manga109 images are not included in this model repository.

The split was deterministic and book-disjoint:

• 36 training books: 359 full pages
• 5 validation books: 50 full pages
• 4 test books: 40 full pages
• one empty-mask training page was excluded

The training stream contained 1,795 entries: 1,436 native 1024×1024 crops (80%) and 359 full pages resized and padded to 1024×1024 (20%). This model was not trained with the mayocream/manga109-segmentation dataset.

Training configuration

• 20 epochs; best checkpoint at epoch 20
• AdamW, weight decay 0.05
• peak learning rate 1e-5
• one-epoch warmup followed by cosine decay to 1e-6
• batch size 16, BF16
• NVIDIA B300 SXM6 AC
• PyTorch 2.9.1 + CUDA 13.0

Usage

Clone Hi-SAM and install its dependencies, plus huggingface_hub and safetensors. The included command-line example loads the complete SafeTensors state and writes a binary PNG mask:

python inference.py input-page.jpg output-mask.png --hi-sam-root /path/to/Hi-SAM

Equivalent model-loading code:

from types import SimpleNamespace

from huggingface_hub import hf_hub_download
from safetensors.torch import load_file
from hi_sam.modeling.build import model_registry

args = SimpleNamespace(
    checkpoint=None,
    model_type="vit_l",
    attn_layers=1,
    prompt_len=12,
    hier_det=False,
)
model = model_registry["vit_l"](args=args)
weights = hf_hub_download(
    "mayocream/koharu-text-sam-ts-l", "model.safetensors"
)
model.load_state_dict(load_file(weights, device="cpu"), strict=True)
model = model.eval().cuda()

Inputs are RGB tensors in the 0–255 range. Resize the page so its longest side is 1024 pixels, pad the bottom and right sides to 1024×1024 with RGB value 128, and crop away the padding from the predicted mask before resizing it back to the original page size. inference.py implements this exact preprocessing.

Limitations

• Evaluation covers 45 books and only the first ten pages of each book.
• The model is strongest on clearly rendered Japanese manga text. Tiny text, stylized sound effects, dense pages, and text-like artwork remain difficult.
• The output is binary text segmentation. It does not assign instances, recognize characters, or associate text with speech balloons.
• Manga and comic styles outside Manga109 may have a larger domain gap.
• Hi-SAM is custom research code, so this repository is not directly compatible with transformers.AutoModel or the hosted Hugging Face inference widget.

Reproducibility and integrity

The complete SafeTensors file was loaded with strict=True and reevaluated on all validation and test pages. It reproduced the original fine-tuned checkpoint metrics exactly in the local CUDA 13 evaluation environment.

Artifact	SHA-256
model.safetensors	bcd9525291677f467f0603509a0ca3df35711b4e3417cefce8da6bfc97164f45
adapter_model.safetensors	78fd7140e5f8728c8d07f97163535c649390077811aad90e0e418c80504d8242
source epoch-20 checkpoint	ebe1ed2bec50659dd4d22bd9fdb017117c15d5bc2acbef4588dfe3baf4d723f8

License and attribution

The model is released under Apache License 2.0, matching Hi-SAM and Segment Anything. The training masks are CC BY 4.0 and are attributed above. Use of Manga109 imagery remains subject to the Manga109 terms; no Manga109 images are redistributed here.