Add pretraining logs

apps/article.py

@@ -1,15 +1,19 @@
 import streamlit as st
 from apps.utils import read_markdown
+from streamlit_tensorboard import st_tensorboard
 
 def app(state):
     st.write(read_markdown("intro.md"))
     st.write("## Methodology")
-    st.markdown(read_markdown("pretraining.md"))
+    st.write(read_markdown("pretraining.md"))
     st.image(
         "./misc/article/Multilingual-VQA.png",
         caption="Masked LM model for Image-text Pretraining.",
     )
-    st.markdown(read_markdown("finetuning.md"))
+    st.write("**Training Logs**")
+    st_tensorboard(logdir='./logs/pretrain_logs', port=6006)
+
+    st.write(read_markdown("finetuning.md"))
     st.write(read_markdown("challenges.md"))
     st.write(read_markdown("limitations.md"))
     st.write(read_markdown("social_impact.md"))

requirements.txt

@@ -5,4 +5,5 @@ torchvision==0.10.0
 mtranslate==1.8
 black==21.7b0
 flax==0.3.4
-torch==1.9.0
+torch==1.9.0
+streamlit-tensorboard==0.0.2

sections/challenges.md

@@ -9,6 +9,6 @@ We faced challenges at every step of the way, despite having some example script
 
 - We prepared a training script for image-text text-only MLM and sequence classification, which we based on hybrid clip, masked LM and the text classification examples.
 
-- We were only able to get around 1.5 days of training time on TPUs due to above mentioned challenges. We were unable to perform hyperparameter tuning. Our [loss curves on the pre-training model](https://huggingface.co/flax-community/multilingual-vqa/tensorboard) show that the training hasn't converged, and we could see further improvement in the MLM accuracy.
+- We were only able to get around 1.5 days of training time on TPUs due to above mentioned challenges. We were unable to perform hyperparameter tuning. Our [loss curves on the pre-training](https://huggingface.co/flax-community/multilingual-vqa/tensorboard) show that the training hasn't converged, and we could see further improvement in the MLM accuracy.
 
 - The VQA dataset, despite having many examples, and after translating into 4x the number of examples, is small and the model overfits. In order to address this, we need more multilingual data, and lighter models, which are both a major challenge right now.

sections/pretraining.md

@@ -7,4 +7,4 @@ The dataset we use for pre-training is a cleaned version of [Conceptual 12M](htt
 
 **Model**
 
-The model is shown in the image below. The `Dummy MLM Head` is actually combined with the MLM head but it never contributes to the MLM loss, hence the name (the predictions on these tokens are ignored). We create a custom model in Flax which integerates the CLIP Vision model inside BERT embeddings. We also use custom configs and modules in order to accomodate for these changes, and allow loading from BERT and CLIP Vision checkpoints. The image is fed to the CLIP Vision encoder and the text is fed to the word-embedding layers of BERT model. We use the `bert-base-multilingual-uncased` and `openai/clip-vit-base-patch32` checkpoints for BERT and CLIP Vision models, respectively. All our code is available on [GitHub](https://github.com/gchhablani/multilingual-vqa).
+The model is shown in the image below. The `Dummy MLM Head` is actually combined with the MLM head but it never contributes to the MLM loss, hence the name (the predictions on these tokens are ignored). We create a custom model in Flax which integerates the CLIP Vision model inside BERT embeddings. We also use custom configs and modules in order to accomodate for these changes, and allow loading from BERT and CLIP Vision checkpoints. The image is fed to the CLIP Vision encoder and the text is fed to the word-embedding layers of BERT model. We use the `bert-base-multilingual-uncased` and `openai/clip-vit-base-patch32` checkpoints for BERT and CLIP Vision models, respectively. All our code and hyperparameters are available on [GitHub](https://github.com/gchhablani/multilingual-vqa).