Update README.md

README.md

@@ -8,7 +8,33 @@ language:
 - ar
 - cz
 inference: false
-license: unknown
+license: mit
 metrics:
 - wer
----
+---
+
+
+# 🚀 ViSPer
+
+ViSPer is a model for audio visual speech recognition (VSR/AVSR). Trained on 5500 hours of labelled video data. 
+
+# Training details:
+
+We use our proposed dataset to train a encoder-decoder model in a fully-supervised manner under a multi-lingual setting. While the encoder size is 12 layers, the decoder size is 6 layers. The hidden size, MLP and number of heads are set to 768, 3072 and 12, respectively. The unigram tokenizers are learned for all languages combined and have a vocabulary size of 21k.
+The models are trained for 150 epochs on 64 Nvidia A100 GPUs (40GB) using AdamW optimizer with max LR of 1e-3 and a weight decay of 0.03. A cosine scheduler with a warm-up of 5 epochs is used for training. The maximum batch size per GPU is set to 2400 video frames.
+
+# Performance:
+
+We provide the results of the model on our proposed benchmarks in this table:
+
+| Language | VSR (WER/CER) | AVSR (WER/CER) |
+|----------|---------------|----------------|
+| French   | 29.8          | 5.7            |
+| Spanish  | 39.4          | 4.4            |
+| Arabic   | 47.8          | 8.4            |
+| Chinese  | 51.3 (CER)    | 15.4 (CER)     |
+| English  | 49.1          | 8.1            |
+
+# Broader impact:
+
+In essence, while we hope that ViSPer will open the doors for new research questions and opportunities, and should only be used for this purpose. There are also potential dual use concerns that come with releasing ViSPer (dataset and models), trained on a substantial corpus of multilingual video data. While the technology behind ViSPer offers significant advances in multiomodal speech recognition, its deployment raises important considerations in terms of societal impacts and potential misuse.