Rearrange app

app.py

@@ -4,6 +4,7 @@ import pandas as pd
 import json
 import os
 import numpy as np
+from streamlit.elements import markdown
 from model.flax_clip_vision_bert.modeling_clip_vision_bert import FlaxCLIPVisionBertForSequenceClassification
 from utils import get_transformed_image, get_text_attributes, get_top_5_predictions, plotly_express_horizontal_bar_plot, translate_labels
 import matplotlib.pyplot as plt
@@ -15,15 +16,30 @@ from session import _get_state
 
 state = _get_state()
 
-@st.cache
+@st.cache(persist=True)
 def load_model(ckpt):
     return FlaxCLIPVisionBertForSequenceClassification.from_pretrained(ckpt)
 
+@st.cache(persist=True)
+def predict(model, transformed_image, question_inputs):
+    return np.array(model(pixel_values = transformed_image, **question_inputs)[0][0])
+    
 def softmax(logits):
     return np.exp(logits)/np.sum(np.exp(logits), axis=0)
 
+def read_markdown(path, parent="./sections/"):
+    with open(os.path.join(parent,path)) as f:
+        return f.read()
+
 checkpoints = ['./ckpt/ckpt-60k-5999'] # TODO: Maybe add more checkpoints?
 dummy_data = pd.read_csv('dummy_vqa_multilingual.tsv', sep='\t')
+code_to_name = {
+    "en": "English",
+    "fr": "French",
+    "de": "German",
+    "es": "Spanish",
+}
+
 with open('answer_reverse_mapping.json') as f:
     answer_reverse_mapping = json.load(f)
 
@@ -36,69 +52,41 @@ st.set_page_config(
 )
 
 st.title("Multilingual Visual Question Answering")
+st.write("[Gunjan Chhablani](https://huggingface.co/gchhablani), [Bhavitvya Malik](https://huggingface.co/bhavitvyamalik)")
 
 
+st.sidebar.write(read_markdown("about.md"))
+st.sidebar.write(read_markdown("caveats.md"))
+st.sidebar.write(read_markdown("challenges.md"))
+st.sidebar.write(read_markdown("social_impact.md"))
+st.sidebar.write(read_markdown("checkpoints.md"))
+st.sidebar.write(read_markdown("acknowledgements.md"))
 
-with st.beta_expander("About"):
-    st.write("This project is focused on Mutilingual Visual Question Answering. Most of the existing datasets and models on this task work with English-only image-text pairs. Our intention here is to provide a Proof-of-Concept with our simple ViT+BERT model which can be trained on multilingual text checkpoints with pre-trained image encoders and well enough. Due to lack of good-quality multilingual data, we translate subsets of the Conceptual 12M dataset into English (already in English), French, German and Spanish using the Marian models.  We achieved 0.49 accuracy on the multilingual validation set we created. With better captions, and hyperparameter-tuning, we expect to see higher performance.")
-with st.beta_expander("Method"):
-    col1, col2 = st.beta_columns([5,4])
-    col1.image("./misc/Multilingual-VQA.png")
-    col2.markdown("""
-## Pretraining
-We follow an approach similar to [VisualBERT](https://arxiv.org/abs/1908.03557). Instead of using a FasterRCNN to get image features, we use a ViT encoder.
-The task is text-only MLM (Masked Language Modeling). We mask only the text tokens and try to predict the masked tokens. The VisualBERT authors also use a sentence-image matching task where two captions are matched against an image, but we skip this for the sake of simplicity.
-### Dataset
-The dataset we use for pre-training is a cleaned version of [Conceptual 12M](https://github.com/google-research-datasets/conceptual-12m). The dataset is downloaded and then broken images are removed which gives us about 10M images. Then we use the MBart50 `
-mbart-large-50-one-to-many-mmt` checkpoint to translate the dataset into four different languages - English, French, German, and Spanish, keeping 2.5 million examples of each language.
-""")
-
-    st.markdown("""
-### Model
-The model is shown in the image above.We create a custom model in Flax which integerates the ViT model inside BERT embeddings. We also use custom configs and modules in order to accomodate for these changes, and allow loading from BERT and ViT checkpoints. The image is fed to the ViT encoder and the text is fed to the word-embedding layers of BERT model. We use the `bert-base-multilingual-uncased` and `openai/clip-vit-base-patch32` checkpoints for BERT and ViT (actually CLIPVision) models, respectively. All our code is available on [GitHub](https://github.com/gchhablani/multilingual-vqa). 
-## Fine-tuning
-
-### Dataset
-For fine-tuning, we use the [VQA 2.0](https://visualqa.org/) dataset - particularly, the `train` and `validation` sets. We translate all the questions into the four languages specified above using language-specific MarianMT models. This is because MarianMT models return better labels and are faster, hence, are better for fine-tuning. We get 4x the number of examples in each subset.
-### Model
-We use the `SequenceClassification` model as reference to create our own sequence classification model. 3129 answer labels are chosen, as is the convention for the English VQA task, which can be found [here](https://github.com/gchhablani/multilingual-vqa/blob/main/answer_mapping.json). These are the same labels used in fine-tuning of the VisualBERT models. The outputs shown here have been translated using the [`mtranslate`](https://github.com/mouuff/mtranslate) Google Translate API library. Then we use various pre-trained checkpoints and train the sequence classification model for various steps.
-
-Checkpoints:
-- Pre-trained checkpoint: [multilingual-vqa](https://huggingface.co/flax-community/multilingual-vqa)
-- Fine-tuned on 45k pretrained checkpoint: [multilingual-vqa-pt-45k-ft](https://huggingface.co/flax-community/multilingual-vqa-pt-45k-ft)
-- Fine-tuned on 45k pretrained checkpoint with AdaFactor (others use AdamW): [multilingual-vqa-pt-45k-ft-adf](https://huggingface.co/flax-community/multilingual-vqa-pt-45k-ft-adf)
-- Fine-tuned on 60k pretrained checkpoint: [multilingual-vqa-pt-60k-ft](https://huggingface.co/flax-community/multilingual-vqa-pt-60k-ft)
-- Fine-tuned on 70k pretrained checkpoint: [multilingual-vqa-pt-60k-ft](https://huggingface.co/flax-community/multilingual-vqa-pt-70k-ft)
-- From scratch (without pre-training) model: [multilingual-vqa-ft](https://huggingface.co/flax-community/multilingual-vqa-ft)
-
-**Caveat**: The best fine-tuned model only achieves 0.49 accuracy on the multilingual validation data that we create. This could be because of not-so-great quality translations, sub-optimal hyperparameters and lack of ample training. In future, we hope to improve this model by addressing such concerns.
-""")
-
-with st.beta_expander("Cherry-Picked Results"):
-    pass
-
-with st.beta_expander("Conclusion"):
-    pass
 
 with st.beta_expander("Usage"):
-    pass
+    st.markdown(read_markdown("usage.md"))
+
+with st.beta_expander("Method"):
+    st.image("./misc/Multilingual-VQA.png")
+    st.markdown(read_markdown("pretraining.md"))
+    st.markdown(read_markdown("finetuning.md"))
+
 
+
+first_index = 20
 # Init Session State
 if state.image_file is None:
-    state.image_file = dummy_data.loc[0,'image_file']
-    state.question = dummy_data.loc[0,'question'].strip('- ')
-    state.answer_label = dummy_data.loc[0,'answer_label']
-    state.question_lang_id = dummy_data.loc[0, 'lang_id']
-    state.answer_lang_id = dummy_data.loc[0, 'lang_id']
+    state.image_file = dummy_data.loc[first_index,'image_file']
+    state.question = dummy_data.loc[first_index,'question'].strip('- ')
+    state.answer_label = dummy_data.loc[first_index,'answer_label']
+    state.question_lang_id = dummy_data.loc[first_index, 'lang_id']
+    state.answer_lang_id = dummy_data.loc[first_index, 'lang_id']
     
     image_path = os.path.join('images',state.image_file)
     image = plt.imread(image_path)
     state.image = image
 
-col1, col2 = st.beta_columns([5,5])
-
-# Display Image
-col1.image(state.image, use_column_width='always')
+col1, col2 = st.beta_columns([6,4])
 
 if col2.button('Get a random example'):
     sample = dummy_data.sample(1).reset_index()
@@ -112,7 +100,7 @@ if col2.button('Get a random example'):
     image = plt.imread(image_path)
     state.image = image
 
-st.write("OR")
+col2.write("OR")
 
 uploaded_file = col2.file_uploader('Upload your image', type=['png','jpg','jpeg'])
 if uploaded_file is not None:
@@ -122,20 +110,22 @@ if uploaded_file is not None:
 
 transformed_image = get_transformed_image(state.image)
 
+# Display Image
+col1.image(state.image, use_column_width='always')
+
 # Display Question
-question = st.text_input(label="Question", value=state.question)
-st.markdown(f"""**English Translation**: {question if state.question_lang_id == "en" else translate(question, 'en')}""")
+question = col2.text_input(label="Question", value=state.question)
+col2.markdown(f"""**English Translation**: {question if state.question_lang_id == "en" else translate(question, 'en')}""")
 question_inputs = get_text_attributes(question)
 
 # Select Language
 options = ['en', 'de', 'es', 'fr']
-state.answer_lang_id = st.selectbox('Answer Language', index=options.index(state.answer_lang_id), options=options)
+state.answer_lang_id = col2.selectbox('Answer Language', index=options.index(state.answer_lang_id), options=options, format_func = lambda x: code_to_name[x])
 # Display Top-5 Predictions
 with st.spinner('Loading model...'):
     model = load_model(checkpoints[0])
 with st.spinner('Predicting...'):
-    predictions = model(pixel_values = transformed_image, **question_inputs)
-logits = np.array(predictions[0][0])
+    logits = predict(model, transformed_image, dict(question_inputs))
 logits = softmax(logits)
 labels, values = get_top_5_predictions(logits, answer_reverse_mapping)
 translated_labels = translate_labels(labels, state.answer_lang_id)

sections/about.md

@@ -0,0 +1,2 @@
+# About 
+This project is focused on Mutilingual Visual Question Answering. Most of the existing datasets and models on this task work with English-only image-text pairs. Our intention here is to provide a Proof-of-Concept with our simple ViT+BERT model which can be trained on multilingual text checkpoints with pre-trained image encoders and made to perform well enough. Due to lack of good-quality multilingual data, we translate subsets of the Conceptual 12M dataset into English (already in English), French, German and Spanish using the Marian models.  We achieved 0.49 accuracy on the multilingual validation set we created. With better captions, and hyperparameter-tuning, we expect to see higher performance.

sections/acknowledgements.md

@@ -0,0 +1,2 @@
+# Acknowledgements
+We thank [Nilakshan Kunananthaseelan](https://huggingface.co/knilakshan20) for helping us whenever he could get a chance. We also thank [Abheesht Sharma](https://huggingface.co/abheesht) for helping in the discussions in the initial phases. Lastly, [Luke Melas](https://github.com/lukemelas) helped us get the CC-12M data on our TPU-VMs and we are very grateful to him.

sections/caveats.md

@@ -0,0 +1 @@
+**Caveat**: The best fine-tuned model only achieves 0.49 accuracy on the multilingual validation data that we create. This could be because of not-so-great quality translations, sub-optimal hyperparameters and lack of ample training. In future, we hope to improve this model by addressing such concerns.

sections/challenges.md

@@ -0,0 +1,14 @@
+# Challenges and Technical Difficulties
+We faced challenges at every step of the way, despite having some example scripts and models ready by the 🤗 team in Flax.
+
+- The dataset we used - Conceptual 12M took 2-3 days to translate using MBart (since we didn't have Marian at the time). The major bottleneck was implementing the translation efficiently. We tried using `mtranslate` first but it turned out to be too slow, even with multiprocessing.
+
+- The translations with deep learning models aren't as "perfect" as translation APIs like Google and Yandex. This could lead to poor performance.
+
+- We prepared the model and config classes for our model from scratch, basing it on `ViT` and `BERT` implementations in Flax. The ViT embeddings should be used inside the BERT embeddings class, which was the major challenge here.
+
+- We prepared a training script for image-text text-only MLM and sequence classification, which we based on hybrid clip, masked LM and the text classification examples.
+
+- We were only able to get around 1.5 days of training time on TPUs due to above mentioned challenges. We were unable to perform hyperparameter tuning. Our loss curves on the pre-training model show that the training hasn't converged, and we could see further improvement in the MLM accuracy.
+
+- The VQA dataset, despite having many examples, and after translating into 4x the number of examples, is small and the model overfits. In order to address this, we need more multilingual data, and lighter models, which are both a major challenge right now.

sections/checkpoints.md

@@ -0,0 +1,7 @@
+**Checkpoints**:
+- Pre-trained checkpoint: [multilingual-vqa](https://huggingface.co/flax-community/multilingual-vqa)
+- Fine-tuned on 45k pretrained checkpoint: [multilingual-vqa-pt-45k-ft](https://huggingface.co/flax-community/multilingual-vqa-pt-45k-ft)
+- Fine-tuned on 45k pretrained checkpoint with AdaFactor (others use AdamW): [multilingual-vqa-pt-45k-ft-adf](https://huggingface.co/flax-community/multilingual-vqa-pt-45k-ft-adf)
+- Fine-tuned on 60k pretrained checkpoint: [multilingual-vqa-pt-60k-ft](https://huggingface.co/flax-community/multilingual-vqa-pt-60k-ft)
+- Fine-tuned on 70k pretrained checkpoint: [multilingual-vqa-pt-60k-ft](https://huggingface.co/flax-community/multilingual-vqa-pt-70k-ft)
+- From scratch (without pre-training) model: [multilingual-vqa-ft](https://huggingface.co/flax-community/multilingual-vqa-ft)

sections/finetuning.md

@@ -0,0 +1,5 @@
+## Fine-tuning
+### Dataset
+For fine-tuning, we use the [VQA 2.0](https://visualqa.org/) dataset - particularly, the `train` and `validation` sets. We translate all the questions into the four languages specified above using language-specific MarianMT models. This is because MarianMT models return better labels and are faster, hence, are better for fine-tuning. We get 4x the number of examples in each subset.
+### Model
+We use the `SequenceClassification` model as reference to create our own sequence classification model. In this, a classification layer is attached on top of the pre-trained BERT model in order to performance multi-class classification. 3129 answer labels are chosen, as is the convention for the English VQA task, which can be found [here](https://github.com/gchhablani/multilingual-vqa/blob/main/answer_mapping.json). These are the same labels used in fine-tuning of the VisualBERT models. The outputs shown here have been translated using the [`mtranslate`](https://github.com/mouuff/mtranslate) Google Translate API library. Then we use various pre-trained checkpoints and train the sequence classification model for various steps.

sections/pretraining.md

@@ -0,0 +1,6 @@
+## Pretraining
+We follow an approach similar to [VisualBERT](https://arxiv.org/abs/1908.03557). Instead of using a FasterRCNN to get image features, we use a ViT encoder. The pre-training task is text-only MLM (Masked Language Modeling). We mask only the text tokens and try to predict the masked tokens. The VisualBERT authors also use a sentence-image matching task where two captions are matched against an image, but we skip this for the sake of simplicity.
+### Dataset
+The dataset we use for pre-training is a cleaned version of [Conceptual 12M](https://github.com/google-research-datasets/conceptual-12m). The dataset is downloaded and then broken images are removed which gives us about 10M images. Then we use the MBart50 `mbart-large-50-one-to-many-mmt` checkpoint to translate the dataset into four different languages - English, French, German, and Spanish, keeping 2.5 million examples of each language.
+### Model
+The model is shown in the image above. The `Dummy MLM Head` is actually combined with the MLM head but it never contributes to the MLM loss, hence the name (the predictions on these tokens are ignored). We create a custom model in Flax which integerates the ViT model inside BERT embeddings. We also use custom configs and modules in order to accomodate for these changes, and allow loading from BERT and ViT checkpoints. The image is fed to the ViT encoder and the text is fed to the word-embedding layers of BERT model. We use the `bert-base-multilingual-uncased` and `openai/clip-vit-base-patch32` checkpoints for BERT and ViT (actually CLIPVision) models, respectively. All our code is available on [GitHub](https://github.com/gchhablani/multilingual-vqa).

sections/references.md

@@ -0,0 +1 @@
+# References

sections/social_impact.md

@@ -0,0 +1,2 @@
+# Social Impact
+Multilingual Visual Question Answering has not received a lot of attention. There are very few multilingual VQA datasets, and that is what we wanted to address here. Our initial plan was to include 4 high-resource and 4 low-resource languages in our training data. However, the existing translations do not perform as well and we would have received poor labels, along with longer training time needed. We hope to improve this in the future by using better translators (for e.g. Google Translate API) to get more multilingual data, especially in low-resource languages. Regardless, our aim with this project was to provide with a pipeline approach to deal with Multilingual visuo-linguistic pretraining and  perform Multilingual Visual Question Answering.

sections/usage.md

@@ -0,0 +1,13 @@
+- This demo loads the `FlaxCLIPVisionBertForSequenceClassificationModel` present in the `model` directory of this repository. The checkpoint is loaded from `ckpt/ckpt-60k-5999` which is pre-trained checkpoint with 60k steps and 5999 fine-tuning steps. 100 random examples are present in the `dummy_vqa_multilingual.tsv` which respective images in the `images/val2014` directory.
+
+- You can also upload your image using the `Upload your image` file uplaoder and type in a question of your choosing.
+
+- We provide `English Translation` of the question for users who are not acquainted with the other languages. This is done using `mtranslate` to keep things flexible enough and needs internet connection as it uses the Google Translate API.
+
+- The model predicts the answers from a list of 3129 answers which have their labels present in `answer_reverse_mapping.json`.
+
+- Lastly, once can choose the `Answer Language` which is also a saved dictionary created using `mtranslate` library for the 3129 answer options.
+
+- The top-5 predictions are displayed below and their respective confidence scores are shown in form of a bar plot.
+
+