phonsobon/khmer-images_captioning

Khmer Image Captioning (Show, Attend and Tell)

A from-scratch image captioning model for Khmer, following the classic Show, Attend and Tell architecture:

• Encoder: ResNet-101 pretrained on ImageNet, used only as a frozen feature extractor (no fine-tuning) producing a 14x14 grid of 2048-dim region features.
• Decoder: an LSTM with Bahdanau (additive) attention, embedding table, and gating layers, trained from scratch on Khmer captions.
• Tokenization: word-level, using khmer-nltk for Khmer word segmentation.

Trained on phonsobon/khmer_images_captioning_v2.

This is not a standard transformers model — it uses custom PyTorch code (modeling_khmer_captioning.py in this repo) rather than AutoModel.

Files

• config.json — architecture hyperparameters
• decoder.pt — trained decoder weights (state_dict)
• vocab.json — word-level vocabulary (itos list)
• modeling_khmer_captioning.py — model classes + load_model() / caption_image() helpers

The encoder is not included — it's just off-the-shelf frozen ImageNet ResNet-101 weights, downloaded automatically via torchvision when you load the model.

Usage

pip install torch torchvision pillow huggingface_hub

from huggingface_hub import snapshot_download
import sys

repo_dir = snapshot_download("phonsobon/khmer-images_captioning")
sys.path.insert(0, repo_dir)

from modeling_khmer_captioning import load_model, caption_image

encoder, decoder, itos, stoi, cfg = load_model(repo_dir)
caption = caption_image("your_image.jpg", encoder, decoder, itos, stoi, cfg, beam_size=3)
print(caption)

Limitations

• Trained on a relatively small dataset (~3.7k image-caption pairs), so generalization outside that domain will be limited.
• Khmer text has no spaces between words; captions are segmented with khmer-nltk, whose probabilistic tokenizer won't always agree exactly with human segmentation, which affects BLEU-style evaluation more than it affects readability.