lazyyawn07/flickr8k-image-captioning

Flickr8k Image Caption Generator

A CNN encoder + LSTM decoder image captioning model, trained from scratch (decoder) on top of a frozen, ImageNet-pretrained ResNet50 encoder, on the Flickr8k dataset.

Architecture

• Encoder: ResNet50 (ImageNet-pretrained, frozen) → 2048-d feature vector → linear projection to embed_size.
• Decoder: single-layer LSTM. The image feature occupies the sequence's "zeroth" timestep; the decoder then generates the caption token by token.
• Inference: beam search (beam_width=3 by default).

No attention mechanism — the decoder conditions on one pooled global image feature per caption, not a spatial feature map.

Training

• Dataset split at the image level (not caption level) into train/val/test (80/10/10, seed 42), so no image's captions leak across splits.
• Image features cached (pre-extracted via the frozen ResNet50) rather than recomputed every epoch.
• Optimizer: Adam, lr=3e-4, gradient clipping at norm 1.0.
• ReduceLROnPlateau LR scheduling + early stopping on validation loss.
• Trained for 35 epochs (stopped manually once val-loss improvement flattened while train/val gap widened).

Evaluation

Evaluated on the held-out test split (809 images, never seen during training or checkpoint selection), one beam-search-generated caption per image compared against that image's 5 human reference captions:

Metric	Score
BLEU-1	0.596
BLEU-2	0.396
BLEU-3	0.268
BLEU-4	0.183
ROUGE-1	0.463
ROUGE-2	0.227
ROUGE-L	0.432

These numbers are in the expected range for a non-attention CNN→LSTM captioner on Flickr8k.

Checkpoint contents

best_model.pth is a single torch.save dict containing:

• model_state_dict — full CNNtoRNN weights (encoder projection + decoder LSTM; the frozen ResNet50 backbone itself is not included and is reconstructed from torchvision's ImageNet weights on load).
• model_config — embed_size, hidden_size, vocab_size, num_layers, needed to reconstruct the model before loading the state dict.
• vocab_stoi / vocab_itos — the trained vocabulary (word ↔ id mappings).
• epoch, val_loss — metadata about the checkpoint.

Usage

import torch
from huggingface_hub import hf_hub_download

ckpt_path = hf_hub_download(repo_id="lazyyawn07/flickr8k-image-captioning", filename="best_model.pth")
checkpoint = torch.load(ckpt_path, map_location="cpu")

# Rebuild with the project's CNNtoRNN / Vocabulary classes:
# https://github.com/<your-repo>  (see models/model.py, data/vocab.py)

Full training/inference code (data pipeline, model definitions, FastAPI + Gradio apps, Dockerfile) is in the accompanying project repository.