chinese-clip-ft-sinonom

Code, data setup and full reproduction steps: https://github.com/vo-hoang-kh4ng/comparing-character-glyph-images-Sino-N-m (the README.md there walks the whole pipeline, with the expected value to check after each step; the write-up is report/paper.pdf.)

Chinese-CLIP ViT-B/16 image tower fine-tuned to retrieve Sino-Nôm (Hán-Nôm) glyph images across typefaces. The training objective is typeface invariance: the same Unicode codepoint rendered in different fonts should map to the same vector.

Built for a course project on similarity search over a 26,044-glyph Sino-Nôm corpus, where the query is a handwritten scan and the corpus is printed regular script — a distribution gap that zero-shot models handle poorly.

Results

Measured on 59 real handwritten scans whose labels are decoded from filenames and checked by a human reader. No pixel of the evaluation set entered training (enforced by an assertion, not by convention).

Whole-corpus retrieval over all 26,044 characters:

model	hit@1	hit@10	hit@20	MRR	95% CI of hit@1
chinese-clip zero-shot (this base)	0.0169	0.0678	—	0.0327	—
chinese-clip-large zero-shot	0.0847	0.1864	0.2034	0.1143	[0.028, 0.187]
this model	0.5932	0.7966	0.8644	0.6691	[0.457, 0.719]

The two confidence intervals are disjoint. Random baseline is 4.7e-4, so hit@1 is ~1250x chance.

Ranking within a same-reading candidate group (median 7 candidates): top-1 accuracy 0.8136 versus 0.4746 for the zero-shot large model and 0.4407 for a colour-histogram baseline.

Not memorisation

2,604 Unicode classes were held out by class — every view removed, not a random image split.

	trained classes	held-out classes
before fine-tuning	0.7938	0.8041
after fine-tuning	0.9954	0.9939

Equal. SupCon has no learnable class centroids, so there is nothing to memorise; what transfers is a comparison function, and it carries over to characters never seen.

Usage

The architecture and embedding width are unchanged from the base model, so the state dict loads into ChineseCLIPModel directly.

import torch
from transformers import AutoImageProcessor, ChineseCLIPModel
from huggingface_hub import hf_hub_download

REPO = "vohoangkh4ng/chinese-clip-ft-sinonom"
model = ChineseCLIPModel.from_pretrained("OFA-Sys/chinese-clip-vit-base-patch16")
state = torch.load(hf_hub_download(REPO, "best_fp16.pt"), map_location="cpu")["model"]
missing, _ = model.load_state_dict(state, strict=False)
assert not [k for k in missing if not k.startswith(("text_model", "text_projection"))]

processor = AutoImageProcessor.from_pretrained(
    "OFA-Sys/chinese-clip-vit-base-patch16", use_fast=False)
model = model.half().eval()

inputs = processor(images=[img], return_tensors="pt")   # img: PIL, RGB
with torch.no_grad():
    v = model.get_image_features(pixel_values=inputs["pixel_values"].half())
v = torch.nn.functional.normalize(v.float(), dim=1)     # cosine == dot product

strict=False is required because the text tower is dropped; the assertion above still fails loudly if any vision key is missing, so a partial load cannot pass silently.

Weights are stored in fp16. This is not an approximation — the inference path casts to fp16 right after loading, so storing fp16 only performs a cast that always happened. Verified by rebuilding all 26,044 corpus vectors from both precisions: maximum absolute difference 0.0.

Training

objective	SupCon (supervised InfoNCE), temperature 0.07
labels	Unicode codepoint only
sampling	P×K: 24 classes × 2 views per batch
data	141,981 images across 23,440 classes, rendered through 7 Hán-Nôm fonts
augmentation	scan-degradation applied online, p = 0.6, strength 1.0
optimiser	AdamW, lr 1e-5, 300-step warmup, cosine, bf16, gradient checkpointing
schedule	6 epochs, best at epoch 5; ~4 min/epoch, 3.7 GB VRAM

Labels are deliberately only the codepoint — not radical or stroke count. Training on a linguistic label and then reporting that label's metric measures the training set rather than retrieval quality.

The degradation model is calibrated against measurements of real scans (edge, contrast, stroke thickness, fill, noise, background) rather than tuned by eye.

P×K sampling is required, not a refinement. A random batch of 48 drawn from 23,440 classes contains ~0.05 positive pairs in expectation, so nearly every batch would have nothing to pull together.

Limitations

• One run, one seed. Per-epoch hit@1 varied 0.51–0.59, which is the size of the measurement noise, so read 0.5932 as "around 0.55".
• n = 59 is the whole real-scan evaluation set. Only the zero-shot-vs-fine-tuned gap is large enough to clear that interval; smaller comparisons are not resolvable.
• Backbone is ViT-B/16, not large — a VRAM constraint during training, not a modelling choice. Fine-tuning large is untested.
• Rendered images cannot evaluate this model; they are its training distribution. Clean and degraded held-out renders both saturate above 0.98. Use real scans.
• Queries are handwritten semi-cursive and the corpus is printed regular script. Every number above describes that one pair of distributions.
• Fonts cover printed styles only. Real brush calligraphy differs structurally, not just in noise, and image degradation cannot synthesise it.

Failure worth recording

An ArcFace head over 23,440 classes was tried first and collapsed: render→corpus hit@1 fell 0.79 → 0.0035 on seen classes and 0.80 → 0.0050 on unseen after one epoch. Falling on both sides is collapse, not overfitting.

The cause is the data shape, not a hyperparameter: 23,440 classes with ~6 images each means a randomly initialised head never organises, so it back-propagates noise — and Adam normalises by gradient magnitude, so "lr = 1e-5" still destroys the pretrained structure within ~3,000 steps. Gradient clipping does not help; clipped noise is still noise.

Anyone retrying metric learning at this class-to-image ratio should expect the same.