🖼️ When ViT meets GPT-2 📝
An image captioning model ViT-GPT2 by combining the ViT model and a French GPT2 model.
Part of the Huggingface JAX/Flax event.
The GPT2 model source code is modified so it can accept an encoder's output. The pretained weights of both models are loaded, with a set of randomly initialized cross-attention weigths. The model is trained on 65000 images from the COCO dataset for about 1500 steps (batch_size=256), with the original English cpationis being translated to French for training purpose.
A HuggingFace Space demo for this model: 🖼️ French Image Captioning Demo 📝