Update ToC

apps/article.py

@@ -5,29 +5,46 @@ from .utils import Toc
 def app(state):
     toc = Toc()
     st.info("Welcome to our Multilingual-VQA demo. Please use the navigation sidebar to move to our demo, or scroll below to read all about our project. 🤗")
+    
     st.header("Table of contents")
     toc.placeholder()
+    
     toc.header("Introduction and Motivation")
-    st.write(read_markdown("intro.md"))
+    st.write(read_markdown("intro/intro.md"))
     toc.subheader("Novel Contributions")
-    st.write(read_markdown("contributions.md"))
+    st.write(read_markdown("intro/contributions.md"))
+    
     toc.header("Methodology")
+
     toc.subheader("Pre-training")
-    st.write(read_markdown("pretraining.md"))
+    st.write(read_markdown("pretraining/intro.md"))
+    # col1, col2 = st.beta_columns([5,5])
     st.image(
         "./misc/article/Multilingual-VQA.png",
         caption="Masked LM model for Image-text Pre-training.",
     )
-    st.write("**Training Logs**")
+    toc.subsubheader("Dataset")
+    st.write(read_markdown("pretraining/data.md"))
+    toc.subsubheader("Model")
+    st.write(read_markdown("pretraining/model.md"))
+    toc.subsubheader("Training Logs")
     st_tensorboard(logdir='./logs/pretrain_logs', port=6006)
+    
+    
     toc.subheader("Finetuning")
-    st.write(read_markdown("finetuning.md"))
-    st.write("**Training Logs**")
+    toc.subsubheader("Dataset")
+    st.write(read_markdown("finetuning/data.md"))
+    toc.subsubheader("Model")
+    st.write(read_markdown("finetuning/model.md"))
+    toc.subsubheader("Training Logs")
     st_tensorboard(logdir='./logs/finetune_logs', port=6007)
+    
     toc.header("Challenges and Technical Difficulties")
     st.write(read_markdown("challenges.md"))
+    
     toc.header("Limitations")
     st.write(read_markdown("limitations.md"))
+    
     toc.header("Conclusion, Future Work, and Social Impact")
     toc.subheader("Conclusion")
     st.write(read_markdown("conclusion.md"))
@@ -35,11 +52,15 @@ def app(state):
     st.write(read_markdown("future_work.md"))
     toc.subheader("Social Impact")
     st.write(read_markdown("social_impact.md"))
+    
     toc.header("References")
     st.write(read_markdown("references.md"))
+
     toc.header("Checkpoints")
     st.write(read_markdown("checkpoints.md"))
     toc.subheader("Other Checkpoints")
     st.write(read_markdown("other_checkpoints.md"))
+    
     toc.header("Acknowledgements")
-    st.write(read_markdown("acknowledgements.md"))
+    st.write(read_markdown("acknowledgements.md"))
+    toc.generate()

apps/utils.py

@@ -23,6 +23,9 @@ class Toc:
 
     def subheader(self, text):
         self._markdown(text, "h3", " " * 4)
+    
+    def subsubheader(self, text):
+        self._markdown(text, "h4", " " * 8)
 
     def placeholder(self, sidebar=False):
         self._placeholder = st.sidebar.empty() if sidebar else st.empty()

sections/finetuning/data.md

@@ -0,0 +1,3 @@
+**Dataset**
+
+For fine-tuning, we use the [VQA 2.0](https://visualqa.org/) dataset - particularly, the `train` and `validation` sets. We translate all the questions into the four languages specified above using language-specific MarianMT models. This is because MarianMT models return better labels and are faster, hence, are better for fine-tuning. We get 4x the number of examples in each subset.

sections/{finetuning.md → finetuning/model.md}

@@ -1,7 +1,3 @@
-**Dataset**
-
-For fine-tuning, we use the [VQA 2.0](https://visualqa.org/) dataset - particularly, the `train` and `validation` sets. We translate all the questions into the four languages specified above using language-specific MarianMT models. This is because MarianMT models return better labels and are faster, hence, are better for fine-tuning. We get 4x the number of examples in each subset.
-
 **Model**
 
 We use the `SequenceClassification` model as reference to create our own sequence classification model. In this, a classification layer is attached on top of the pre-trained BERT model in order to performance multi-class classification. 3129 answer labels are chosen, as is the convention for the English VQA task, which can be found [here](https://github.com/gchhablani/multilingual-vqa/blob/main/answer_mapping.json). These are the same labels used in fine-tuning of the VisualBERT models. The outputs shown here have been translated using the [`mtranslate`](https://github.com/mouuff/mtranslate) Google Translate API library. Then we use various pre-trained checkpoints and train the sequence classification model for various steps.

sections/pretraining/data.md

@@ -0,0 +1,3 @@
+**Dataset**
+
+The dataset we use for pre-training is a cleaned version of [Conceptual 12M](https://github.com/google-research-datasets/conceptual-12m). The dataset is downloaded and then broken images are removed which gives us about 10M images. Then we use the MBart50 `mbart-large-50-one-to-many-mmt` checkpoint to translate the dataset into four different languages - English, French, German, and Spanish, keeping 2.5 million examples of each language.

sections/pretraining/intro.md

@@ -0,0 +1 @@
+We follow an approach similar to [VisualBERT](https://arxiv.org/abs/1908.03557). Instead of using a FasterRCNN to get image features, we use a CLIP Vision (ViT transformer) encoder. The pre-training task is text-only MLM (Masked Language Modeling). We mask only the text tokens and try to predict the masked tokens. The VisualBERT authors also use a sentence-image matching task where two captions are matched against an image, but we skip this for the sake of simplicity.

sections/{pretraining.md → pretraining/model.md}

@@ -1,9 +1,3 @@
-We follow an approach similar to [VisualBERT](https://arxiv.org/abs/1908.03557). Instead of using a FasterRCNN to get image features, we use a CLIP Vision (ViT transformer) encoder. The pre-training task is text-only MLM (Masked Language Modeling). We mask only the text tokens and try to predict the masked tokens. The VisualBERT authors also use a sentence-image matching task where two captions are matched against an image, but we skip this for the sake of simplicity.
-
-**Dataset**
-
-The dataset we use for pre-training is a cleaned version of [Conceptual 12M](https://github.com/google-research-datasets/conceptual-12m). The dataset is downloaded and then broken images are removed which gives us about 10M images. Then we use the MBart50 `mbart-large-50-one-to-many-mmt` checkpoint to translate the dataset into four different languages - English, French, German, and Spanish, keeping 2.5 million examples of each language.
-
 **Model**
 
-The model is shown in the figure below. The `Dummy MLM Head` is actually combined with the MLM head but it never contributes to the MLM loss, hence the name (the predictions on these tokens are ignored). We create a custom model in Flax which integerates the CLIP Vision model inside BERT embeddings. We also use custom configs and modules in order to accomodate for these changes, and allow loading from BERT and CLIP Vision checkpoints. The image is fed to the CLIP Vision encoder and the text is fed to the word-embedding layers of BERT model. We use the `bert-base-multilingual-uncased` and `openai/clip-vit-base-patch32` checkpoints for BERT and CLIP Vision models, respectively. All our code and hyperparameters are available on [GitHub](https://github.com/gchhablani/multilingual-vqa).
+The model is shown in the figure below. The `Dummy MLM Head` is actually combined with the MLM head but it never contributes to the MLM loss, hence the name (the predictions on these tokens are ignored). We create a custom model in Flax which integerates the CLIP Vision model inside BERT embeddings. We also use custom configs and modules in order to accomodate for these changes, and allow loading from BERT and CLIP Vision checkpoints. The image is fed to the CLIP Vision encoder and the text is fed to the word-embedding layers of BERT model. We use the `bert-base-multilingual-uncased` and `openai/clip-vit-base-patch32` checkpoints for BERT and CLIP Vision models, respectively. All our code and hyperparameters are available on [GitHub](https://github.com/gchhablani/multilingual-vqa).