first commit

Browse files

Files changed (3) hide show

README.md +103 -0
config.json +316 -0
logs/events.out.tfevents.1626777663.t1v-n-e32efd45-w-0.1846718.3.v2 +3 -0

README.md ADDED Viewed

	@@ -0,0 +1,103 @@

+# CLIP-Vision-mBART50 Seq2Seq Encoder-Decoder Model
+Pretrained CLIP-Vision-mBART50 pre-trained on subset of translated Conceptual-12M image-text pairs using a seq2seq model training objective. 2.5M cleaned English image-text pairs are translated using Marian Model for respective languages to 2.5M examples each in English, French, German and Spanish. We trained CLIP-Vision-mBART50 model during community week hosted by Huggingface 🤗 using JAX/Flax.
+## Model description
+CLIP-Vision-mBART50 is a modified mBART50 model which takes in visual embeddings from CLIP-Vision transformer and concatenates them with mBART textual embeddings before passing them to the self-attention layers of mBART. This is done for deep cross-modal interaction between the two modes.
+## Intended uses & limitations❗️
+You can use the raw model for encoder decoder network where you want the encoder to encode images and decoder to decode text.
+Note that this model is primarily aimed at being fine-tuned on tasks like multi-lingual/mono-lingual image captioning.
+### How to use❓
+You will need to clone the model from [here](https://github.com/gchhablani/multilingual-image-captioning). An example of usage is shown below:
+```python
+>>> from torchvision.io import read_image
+>>> import numpy as  np
+>>> import os
+>>> from transformers import CLIPProcessor, MBart50TokenizerFast
+>>> from model.flax_clip_vision_mbart.modeling_clip_vision_mbart import FlaxCLIPVisionMBartForConditionalGeneration
+>>> image_path = os.path.join('images/val2014', os.listdir('images/val2014')[0])
+>>> img = read_image(image_path) # reading image
+>>> clip_processor = CLIPProcessor.from_pretrained('openai/clip-vit-base-patch32')
+>>> clip_outputs = clip_processor(images=img)
+>>> clip_outputs['pixel_values'][0] = clip_outputs['pixel_values'][0].transpose(1,2,0) # Need to transpose images as model expected channel last images.
+>>> tokenizer = MBart50TokenizerFast.from_pretrained('facebook/mbart-large-50"')
+>>> model = FlaxCLIPVisionBertForMaskedLM.from_pretrained('flax-community/clip-vit-base-patch32_mbart-large-50')
+>>> output_ids = model.generate(batch["pixel_values"], forced_bos_token_id=tokenizer.lang_code_to_id["en_XX"], num_beams=4, max_length=64).sequences  # "en_XX is the language code in which you want the translation
+# en_XX: English, fr_XX: French, es_XX: Spanish, de_DE: Deutsch
+>>> output_string = tokenizer.batch_decode(output_ids.reshape(-1, 64), skip_special_tokens=True, max_length=64)
+>>> output_string # relevant caption
+```
+## Training data 🏋🏻‍♂️
+The Multi-lingual image captioning model was trained on a subset of Conceptual 12M dataset by Google:
+<br>
+<br>
+[Conceptual 12M](https://github.com/google-research-datasets/conceptual-12m), Introduced by Changpinyo et al. in [Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts](https://arxiv.org/abs/2102.08981).
+The translated dataset can be downloaded from [conceptual-12m-multilingual-marian](https://huggingface.co/datasets/flax-community/conceptual-12m-multilingual-marian). We do not provide images as we do not own any of them. One can download images from the `image_url` section of the original Conceptual 12M dataset.
+## Data Cleaning 🧹
+Though the original dataset contains 12M image-text pairs, a lot of the URLs are invalid now, and in some cases, images are corrupt or broken. We remove such examples from our data, which leaves us with approximately 10M image-text pairs.
+#### **Train set:**
+Total data: <br>
+10010625 captions  <br>
+2502656 images br><br>
+Language-wise distribution: <br>
+English: 2502656 captions<br>
+Spanish: 2502656 captions<br>
+Deutsch: 2502656 captions<br>
+French: 2502656 captions<br>
+#### **Validation set**
+Total data: <br>
+110592 captions  <br>
+27648 images <br><br>
+Language-wise distribution: <br>
+English: 27648 captions<br>
+Spanish: 27648 captions<br>
+Deutsch: 27648 captions<br>
+French: 27648 captions<br>
+## Training procedure 👨🏻‍💻
+### Training
+The model was trained on Google Cloud Engine TPUv3-8 machine (with 335 GB of RAM, 1000 GB of hard drive, 96 CPU cores) **8 v3 TPU cores** for 42K steps with a batch size of 128 and a sequence length of 128. The
+optimizer used is Adam with a learning rate of 3e-4, β1 = 0.9, β2 = 0.98 and
+ε = 1e-8, a weight decay of 0.01, learning rate warmup for 1,000 steps and linear decay of the learning
+rate after.
+We tracked experiments using Tensorboard which can be found in `Training Metrics` tab.
+#### **Pretraining Results 📊**
+Our model reached **eval loss of ~2.6** around ~70K steps. Here are the BLEU^ scores for different languages:
+|Language                  |BLEU-1|BLEU-2|BLEU-3|BLEU-4|
+|--------------------------|------|------|------|------|
+|English                   | 0.163| 0.127| 0.10 | 0.081|
+|Spanish                   | 0.171| 0.133| 0.114| 0.082|
+|German                    | 0.165| 0.129| 0.103| 0.077|
+|French                    | 0.162| 0.124| 0.104| 0.073|
+^BLEU scores are out of 1
+## **App Demo**
+You can try out our model on 🤗 Huggingface's spaces 🪐 :
+[Streamlit app of Multi-lingual Image Captioning model on Huggingface Spaces](https://huggingface.co/spaces/flax-community/multilingual-image-captioning)
+## Team Members
+  - Bhavitvya Malik [@bhavitvyamalik](https://github.com/bhavitvyamalik)
+  - Gunjan Chhablani [@gchhablani](https://github.com/gchhablani)
+## Credits
+  Thanks to Huggingface 🤗 & Google JAX/FLAX team for such a wonderful community week. Big thanks to [@patrickvonplaten](https://github.com/patrickvonplaten) and [@patil-suraj](https://github.com/patil-suraj) for helping us with our solution during the community week.
+<img src=https://pbs.twimg.com/media/E443fPjX0AY1BsR.jpg:large>

config.json ADDED Viewed

	@@ -0,0 +1,316 @@

+{
+  "architectures": [
+    "CLIPVisionMBartForConditionalGeneration"
+  ],
+  "clip_vision_config": {
+    "_name_or_path": "",
+    "add_cross_attention": false,
+    "architectures": [
+      "CLIPModel"
+    ],
+    "attention_dropout": 0.0,
+    "bad_words_ids": null,
+    "bos_token_id": null,
+    "chunk_size_feed_forward": 0,
+    "decoder_start_token_id": null,
+    "diversity_penalty": 0.0,
+    "do_sample": false,
+    "dropout": 0.0,
+    "early_stopping": false,
+    "encoder_no_repeat_ngram_size": 0,
+    "eos_token_id": null,
+    "finetuning_task": null,
+    "forced_bos_token_id": null,
+    "forced_eos_token_id": null,
+    "gradient_checkpointing": false,
+    "hidden_act": "quick_gelu",
+    "hidden_size": 768,
+    "id2label": {
+      "0": "LABEL_0",
+      "1": "LABEL_1"
+    },
+    "image_size": 224,
+    "initializer_factor": 1.0,
+    "initializer_range": 0.02,
+    "intermediate_size": 3072,
+    "is_decoder": false,
+    "is_encoder_decoder": false,
+    "label2id": {
+      "LABEL_0": 0,
+      "LABEL_1": 1
+    },
+    "layer_norm_eps": 1e-05,
+    "length_penalty": 1.0,
+    "max_length": 20,
+    "min_length": 0,
+    "model_type": "clip_vision_model",
+    "no_repeat_ngram_size": 0,
+    "num_attention_heads": 12,
+    "num_beam_groups": 1,
+    "num_beams": 1,
+    "num_hidden_layers": 12,
+    "num_return_sequences": 1,
+    "output_attentions": false,
+    "output_hidden_states": false,
+    "output_scores": false,
+    "pad_token_id": null,
+    "patch_size": 32,
+    "prefix": null,
+    "problem_type": null,
+    "projection_dim": 512,
+    "pruned_heads": {},
+    "remove_invalid_values": false,
+    "repetition_penalty": 1.0,
+    "return_dict": true,
+    "return_dict_in_generate": false,
+    "sep_token_id": null,
+    "task_specific_params": null,
+    "temperature": 1.0,
+    "text_config": {
+      "_name_or_path": "",
+      "add_cross_attention": false,
+      "architectures": null,
+      "attention_dropout": 0.0,
+      "bad_words_ids": null,
+      "bos_token_id": 0,
+      "chunk_size_feed_forward": 0,
+      "decoder_start_token_id": null,
+      "diversity_penalty": 0.0,
+      "do_sample": false,
+      "dropout": 0.0,
+      "early_stopping": false,
+      "encoder_no_repeat_ngram_size": 0,
+      "eos_token_id": 2,
+      "finetuning_task": null,
+      "forced_bos_token_id": null,
+      "forced_eos_token_id": null,
+      "gradient_checkpointing": false,
+      "hidden_act": "quick_gelu",
+      "hidden_size": 512,
+      "id2label": {
+        "0": "LABEL_0",
+        "1": "LABEL_1"
+      },
+      "initializer_factor": 1.0,
+      "initializer_range": 0.02,
+      "intermediate_size": 2048,
+      "is_decoder": false,
+      "is_encoder_decoder": false,
+      "label2id": {
+        "LABEL_0": 0,
+        "LABEL_1": 1
+      },
+      "layer_norm_eps": 1e-05,
+      "length_penalty": 1.0,
+      "max_length": 20,
+      "max_position_embeddings": 77,
+      "min_length": 0,
+      "model_type": "clip_text_model",
+      "no_repeat_ngram_size": 0,
+      "num_attention_heads": 8,
+      "num_beam_groups": 1,
+      "num_beams": 1,
+      "num_hidden_layers": 12,
+      "num_return_sequences": 1,
+      "output_attentions": false,
+      "output_hidden_states": false,
+      "output_scores": false,
+      "pad_token_id": 1,
+      "prefix": null,
+      "problem_type": null,
+      "pruned_heads": {},
+      "remove_invalid_values": false,
+      "repetition_penalty": 1.0,
+      "return_dict": true,
+      "return_dict_in_generate": false,
+      "sep_token_id": null,
+      "task_specific_params": null,
+      "temperature": 1.0,
+      "tie_encoder_decoder": false,
+      "tie_word_embeddings": true,
+      "tokenizer_class": null,
+      "top_k": 50,
+      "top_p": 1.0,
+      "torchscript": false,
+      "transformers_version": "4.7.0.dev0",
+      "use_bfloat16": false,
+      "vocab_size": 49408
+    },
+    "text_config_dict": null,
+    "tie_encoder_decoder": false,
+    "tie_word_embeddings": true,
+    "tokenizer_class": null,
+    "top_k": 50,
+    "top_p": 1.0,
+    "torch_dtype": null,
+    "torchscript": false,
+    "transformers_version": "4.9.0.dev0",
+    "use_bfloat16": false,
+    "vision_config": {
+      "_name_or_path": "",
+      "add_cross_attention": false,
+      "architectures": null,
+      "attention_dropout": 0.0,
+      "bad_words_ids": null,
+      "bos_token_id": null,
+      "chunk_size_feed_forward": 0,
+      "decoder_start_token_id": null,
+      "diversity_penalty": 0.0,
+      "do_sample": false,
+      "dropout": 0.0,
+      "early_stopping": false,
+      "encoder_no_repeat_ngram_size": 0,
+      "eos_token_id": null,
+      "finetuning_task": null,
+      "forced_bos_token_id": null,
+      "forced_eos_token_id": null,
+      "gradient_checkpointing": false,
+      "hidden_act": "quick_gelu",
+      "hidden_size": 768,
+      "id2label": {
+        "0": "LABEL_0",
+        "1": "LABEL_1"
+      },
+      "image_size": 224,
+      "initializer_factor": 1.0,
+      "initializer_range": 0.02,
+      "intermediate_size": 3072,
+      "is_decoder": false,
+      "is_encoder_decoder": false,
+      "label2id": {
+        "LABEL_0": 0,
+        "LABEL_1": 1
+      },
+      "layer_norm_eps": 1e-05,
+      "length_penalty": 1.0,
+      "max_length": 20,
+      "min_length": 0,
+      "model_type": "clip_vision_model",
+      "no_repeat_ngram_size": 0,
+      "num_attention_heads": 12,
+      "num_beam_groups": 1,
+      "num_beams": 1,
+      "num_hidden_layers": 12,
+      "num_return_sequences": 1,
+      "output_attentions": false,
+      "output_hidden_states": false,
+      "output_scores": false,
+      "pad_token_id": null,
+      "patch_size": 32,
+      "prefix": null,
+      "problem_type": null,
+      "pruned_heads": {},
+      "remove_invalid_values": false,
+      "repetition_penalty": 1.0,
+      "return_dict": true,
+      "return_dict_in_generate": false,
+      "sep_token_id": null,
+      "task_specific_params": null,
+      "temperature": 1.0,
+      "tie_encoder_decoder": false,
+      "tie_word_embeddings": true,
+      "tokenizer_class": null,
+      "top_k": 50,
+      "top_p": 1.0,
+      "torchscript": false,
+      "transformers_version": "4.7.0.dev0",
+      "use_bfloat16": false
+    },
+    "vision_config_dict": null
+  },
+  "is_encoder_decoder": true,
+  "mbart_config": {
+    "_name_or_path": "/home/suraj/projects/mbart-50/hf_models/mbart-50-large",
+    "_num_labels": 3,
+    "activation_dropout": 0.0,
+    "activation_function": "gelu",
+    "add_bias_logits": false,
+    "add_cross_attention": false,
+    "add_final_layer_norm": true,
+    "architectures": [
+      "MBartForConditionalGeneration"
+    ],
+    "attention_dropout": 0.0,
+    "bad_words_ids": null,
+    "bos_token_id": 0,
+    "chunk_size_feed_forward": 0,
+    "classif_dropout": 0.0,
+    "classifier_dropout": 0.0,
+    "d_model": 1024,
+    "decoder_attention_heads": 16,
+    "decoder_ffn_dim": 4096,
+    "decoder_layerdrop": 0.0,
+    "decoder_layers": 12,
+    "decoder_start_token_id": 2,
+    "diversity_penalty": 0.0,
+    "do_sample": false,
+    "dropout": 0.1,
+    "early_stopping": true,
+    "encoder_attention_heads": 16,
+    "encoder_ffn_dim": 4096,
+    "encoder_layerdrop": 0.0,
+    "encoder_layers": 12,
+    "encoder_no_repeat_ngram_size": 0,
+    "eos_token_id": 2,
+    "finetuning_task": null,
+    "forced_bos_token_id": null,
+    "forced_eos_token_id": 2,
+    "gradient_checkpointing": false,
+    "id2label": {
+      "0": "LABEL_0",
+      "1": "LABEL_1",
+      "2": "LABEL_2"
+    },
+    "init_std": 0.02,
+    "is_decoder": false,
+    "is_encoder_decoder": true,
+    "label2id": {
+      "LABEL_0": 0,
+      "LABEL_1": 1,
+      "LABEL_2": 2
+    },
+    "length_penalty": 1.0,
+    "max_length": 200,
+    "max_position_embeddings": 1024,
+    "min_length": 0,
+    "model_type": "mbart",
+    "no_repeat_ngram_size": 0,
+    "normalize_before": true,
+    "normalize_embedding": true,
+    "num_beam_groups": 1,
+    "num_beams": 5,
+    "num_hidden_layers": 12,
+    "num_return_sequences": 1,
+    "output_attentions": false,
+    "output_hidden_states": false,
+    "output_past": true,
+    "output_scores": false,
+    "pad_token_id": 1,
+    "prefix": null,
+    "problem_type": null,
+    "pruned_heads": {},
+    "remove_invalid_values": false,
+    "repetition_penalty": 1.0,
+    "return_dict": true,
+    "return_dict_in_generate": false,
+    "scale_embedding": true,
+    "sep_token_id": null,
+    "static_position_embeddings": false,
+    "task_specific_params": null,
+    "temperature": 1.0,
+    "tie_encoder_decoder": false,
+    "tie_word_embeddings": true,
+    "tokenizer_class": "MBart50Tokenizer",
+    "top_k": 50,
+    "top_p": 1.0,
+    "torch_dtype": null,
+    "torchscript": false,
+    "transformers_version": "4.9.0.dev0",
+    "use_bfloat16": false,
+    "use_cache": true,
+    "vocab_size": 250054
+  },
+  "model_type": "clip-vision-mbart",
+  "seed": 42,
+  "transformers_version": null
+}

logs/events.out.tfevents.1626777663.t1v-n-e32efd45-w-0.1846718.3.v2 ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:3e6c4ab6e9bb14ec893541f156b36d17bea7c9b8d8ba677bf80d7a0bb6c5853d
+size 13753034