Visual Speech Recognition for Multiple Languages

<p align="center"><img width="160" src="doc/lip_white.png" alt="logo"></p>
<h1 align="center">Visual Speech Recognition for Multiple Languages</h1>

<div align="center">

[📘Introduction](#Introduction) |
[🛠️Preparation](#Preparation) |
[📊Benchmark](#Benchmark-evaluation) |
[🔮Inference](#Speech-prediction) |
[🐯Model zoo](#Model-Zoo) |
[📝License](#License)
</div>

## Authors

[Pingchuan Ma](https://mpc001.github.io/), [Alexandros Haliassos](https://dblp.org/pid/257/3052.html), [Adriana Fernandez-Lopez](https://scholar.google.com/citations?user=DiVeQHkAAAAJ), [Honglie Chen](https://scholar.google.com/citations?user=HPwdvwEAAAAJ), [Stavros Petridis](https://ibug.doc.ic.ac.uk/people/spetridis), [Maja Pantic](https://ibug.doc.ic.ac.uk/people/mpantic).

## Update

`2023-03-27`: We have released our AutoAVSR models for LRS3, see [here](#autoavsr-models).

## Introduction

This is the repository of [Auto-AVSR: Audio-Visual Speech Recognition with Automatic Labels](https://arxiv.org/abs/2303.14307) and [Visual Speech Recognition for Multiple Languages](https://arxiv.org/abs/2202.13084), which is the successor of [End-to-End Audio-Visual Speech Recognition with Conformers](https://arxiv.org/abs/2102.06657). By using this repository, you can achieve the performance of 19.1%, 1.0% and 0.9% WER for automatic, visual, and audio-visual speech recognition (ASR, VSR, and AV-ASR) on LRS3.

## Tutorial

We provide a tutorial [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1jfb6e4xxhXHbmQf-nncdLno1u0b4j614) to show how to use our Auto-AVSR models to perform speech recognition (ASR, VSR, and AV-ASR), crop mouth ROIs or extract visual speech features.

## Demo

English -> Mandarin -> Spanish   |    French -> Portuguese -> Italian  |
:-------------------------------:|:------------------------------------:
<img src='doc/vsr_1.gif' title='vsr1' style='max-width:320px'></img>  |  <img src='doc/vsr_2.gif' title='vsr2' style='max-width:320px'></img>  |

<div align="center">

[Youtube](https://youtu.be/FIau-6JA9Po) |
[Bilibili](https://www.bilibili.com/video/BV1Wu411D7oP)
</div>

## Preparation
1. Clone the repository and enter it locally:

```Shell
git clone https://github.com/mpc001/Visual_Speech_Recognition_for_Multiple_Languages
cd Visual_Speech_Recognition_for_Multiple_Languages
```

2. Setup the environment.
```Shell
conda create -y -n autoavsr python=3.8
conda activate autoavsr
```

3. Install pytorch, torchvision, and torchaudio by following instructions [here](https://pytorch.org/get-started/), and install all packages:

```Shell
pip install -r requirements.txt
conda install -c conda-forge ffmpeg
```

4. Download and extract a pre-trained model and/or language model from [model zoo](#Model-Zoo) to:

- `./benchmarks/${dataset}/models`

- `./benchmarks/${dataset}/language_models`

5. [For VSR and AV-ASR] Install [RetinaFace](./tools) or [MediaPipe](https://pypi.org/project/mediapipe/) tracker.

### Benchmark evaluation

```Shell
python eval.py config_filename=[config_filename] \
               labels_filename=[labels_filename] \
               data_dir=[data_dir] \
               landmarks_dir=[landmarks_dir]
```

- `[config_filename]` is the model configuration path, located in `./configs`.

- `[labels_filename]` is the labels path, located in `${lipreading_root}/benchmarks/${dataset}/labels`.

- `[data_dir]` and `[landmarks_dir]` are the directories for original dataset and corresponding landmarks.

- `gpu_idx=-1` can be added to switch from `cuda:0` to `cpu`.

### Speech prediction

```Shell
python infer.py config_filename=[config_filename] data_filename=[data_filename]
```

- `data_filename` is the path to the audio/video file.

- `detector=mediapipe` can be added to switch from RetinaFace to MediaPipe tracker.

### Mouth ROIs cropping

```Shell
python crop_mouth.py data_filename=[data_filename] dst_filename=[dst_filename]
```

- `dst_filename` is the path where the cropped mouth will be saved.

## Model zoo

### Overview

We support a number of datasets for speech recognition:
- [x] [Lip Reading Sentences 2 (LRS2)](https://www.robots.ox.ac.uk/~vgg/data/lip_reading/lrs2.html)
- [x] [Lip Reading Sentences 3 (LRS3)](https://www.robots.ox.ac.uk/~vgg/data/lip_reading/lrs3.html)
- [x] [Chinese Mandarin Lip Reading (CMLR)](https://www.vipazoo.cn/CMLR.html)
- [x] [CMU Multimodal Opinion Sentiment, Emotions and Attributes (CMU-MOSEAS)](http://immortal.multicomp.cs.cmu.edu/cache/multilingual)
- [x] [GRID](http://spandh.dcs.shef.ac.uk/gridcorpus)
- [x] [Lombard GRID](http://spandh.dcs.shef.ac.uk/avlombard)
- [x] [TCD-TIMIT](https://sigmedia.tcd.ie)

### AutoAVSR models

<details open>

<summary>Lip Reading Sentences 3 (LRS3)</summary>

<p> </p>

|     Components        |  WER |                                             url                                         |  size (MB)  |
|:----------------------|:----:|:---------------------------------------------------------------------------------------:|:-----------:|
|   **Visual-only**     |
| -                     | 19.1 |[GoogleDrive](http://bit.ly/40EAtyX) or [BaiduDrive](https://bit.ly/3ZjbrV5)(key: dqsy)  |     891     |
|   **Audio-only**      |
| -                     | 1.0  |[GoogleDrive](http://bit.ly/3ZSdh0l) or [BaiduDrive](http://bit.ly/3Z1TlGU)(key: dvf2)   |     860     |
|   **Audio-visual**    |
| -                     | 0.9  |[GoogleDrive](http://bit.ly/3yRSXAn) or [BaiduDrive](http://bit.ly/3LAxcMY)(key: sai5)   |     1540    |
| **Language models**   |
| -                     |   -  |[GoogleDrive](http://bit.ly/3FE4XsV) or [BaiduDrive](http://bit.ly/3yRI5SY)(key: t9ep)   |     191     |
| **Landmarks**         |
| -                     |   -  |[GoogleDrive](https://bit.ly/33rEsax) or [BaiduDrive](https://bit.ly/3rwQSph)(key: mi3c) |     18577   |

</details>

### VSR for multiple languages models

<details open>

<summary>Lip Reading Sentences 2 (LRS2)</summary>

<p> </p>

|     Components        |  WER |                                             url                                         |  size (MB)  |
|:----------------------|:----:|:---------------------------------------------------------------------------------------:|:-----------:|
|   **Visual-only**     |
| -                     | 26.1 |[GoogleDrive](https://bit.ly/3I25zrH) or [BaiduDrive](https://bit.ly/3BAHBkH)(key: 48l1) |     186     |
| **Language models**   |
| -                     |   -  |[GoogleDrive](https://bit.ly/3qzWKit) or [BaiduDrive](https://bit.ly/3KgAL7T)(key: 59u2) |     180     |
| **Landmarks**         |
| -                     |   -  |[GoogleDrive](https://bit.ly/3jSMMoz) or [BaiduDrive](https://bit.ly/3BuIwBB)(key: 53rc) |     9358    |

</details>


<details open>

<summary>Lip Reading Sentences 3 (LRS3)</summary>

<p> </p>

|     Components        |  WER |                                             url                                         |  size (MB)  |
|:----------------------|:----:|:---------------------------------------------------------------------------------------:|:-----------:|
|   **Visual-only**     |
| -                     | 32.3 |[GoogleDrive](https://bit.ly/3Bp4gjV) or [BaiduDrive](https://bit.ly/3rIzLCn)(key: 1b1s) |     186     |
| **Language models**   |
| -                     |   -  |[GoogleDrive](https://bit.ly/3qzWKit) or [BaiduDrive](https://bit.ly/3KgAL7T)(key: 59u2) |     180     |
| **Landmarks**         |
| -                     |   -  |[GoogleDrive](https://bit.ly/33rEsax) or [BaiduDrive](https://bit.ly/3rwQSph)(key: mi3c) |     18577   |

</details>


<details open>

<summary>Chinese Mandarin Lip Reading (CMLR)</summary>

<p> </p>

|     Components        |  CER |                                             url                                         |  size (MB)  |
|:----------------------|:----:|:---------------------------------------------------------------------------------------:|:-----------:|
|   **Visual-only**     |
| -                     |  8.0 |[GoogleDrive](https://bit.ly/3fR8RkU) or [BaiduDrive](https://bit.ly/3IyACLB)(key: 7eq1) |     195     |
| **Language models**   |
| -                     |   -  |[GoogleDrive](https://bit.ly/3fPxXAJ) or [BaiduDrive](https://bit.ly/3rEcErr)(key: k8iv) |     187     |
| **Landmarks**         |
| -                     |   -  |[GoogleDrive](https://bit.ly/3bvetPL) or [BaiduDrive](https://bit.ly/3o2u53d)(key: 1ret) |     3721    |

</details>


<details open>

<summary>CMU Multimodal Opinion Sentiment, Emotions and Attributes (CMU-MOSEAS)</summary>

<p> </p>

|     Components        |  WER |                                             url                                         |  size (MB)  |
|:----------------------|:----:|:---------------------------------------------------------------------------------------:|:-----------:|
|   **Visual-only**     |
| Spanish               | 44.5 |[GoogleDrive](https://bit.ly/34MjWBW) or [BaiduDrive](https://bit.ly/33rMq3a)(key: m35h) |     186     |
| Portuguese            | 51.4 |[GoogleDrive](https://bit.ly/3HjXCgo) or [BaiduDrive](https://bit.ly/3IqbbMg)(key: wk2h) |     186     |
| French                | 58.6 |[GoogleDrive](https://bit.ly/3Ik6owb) or [BaiduDrive](https://bit.ly/35msiQG)(key: t1hf) |     186     |
| **Language models**   |
| Spanish               |   -  |[GoogleDrive](https://bit.ly/3rppyJN) or [BaiduDrive](https://bit.ly/3nA3wCN)(key: 0mii) |     180     |
| Portuguese            |   -  |[GoogleDrive](https://bit.ly/3gPvneF) or [BaiduDrive](https://bit.ly/33vL8Es)(key: l6ag) |     179     |
| French                |   -  |[GoogleDrive](https://bit.ly/3LDChSn) or [BaiduDrive](https://bit.ly/3sNnNql)(key: 6tan) |     179     |
| **Landmarks**         |
| -                     |   -  |[GoogleDrive](https://bit.ly/34Cf6ak) or [BaiduDrive](https://bit.ly/3BiFG4c)(key: vsic) |     3040    |


</details>


<details open>

<summary>GRID</summary>

<p> </p>

|     Components        |  WER |                                             url                                         |  size (MB)  |
|:----------------------|:----:|:---------------------------------------------------------------------------------------:|:-----------:|
|   **Visual-only**     |
| Overlapped            |  1.2 |[GoogleDrive](https://bit.ly/3Aa6PWn) or [BaiduDrive](https://bit.ly/3IdamGh)(key: d8d2) |     186     |
| Unseen                |  4.8 |[GoogleDrive](https://bit.ly/3patMVh) or [BaiduDrive](https://bit.ly/3t6459A)(key: ttsh) |     186     |
| **Landmarks**         |
| -                     |   -  |[GoogleDrive](https://bit.ly/2Yzu1PF) or [BaiduDrive](https://bit.ly/30fucjG)(key: 16l9) |     1141    |

You can include `data_ext=.mpg` in your command line to match the video file extension in the GRID dataset.

</details>


<details open>

<summary>Lombard GRID</summary>

<p> </p>

|     Components        |  WER |                                             url                                         |  size (MB)  |
|:----------------------|:----:|:---------------------------------------------------------------------------------------:|:-----------:|
|   **Visual-only**     |
| Unseen (Front Plain)  |  4.9 |[GoogleDrive](https://bit.ly/3H5zkGQ) or [BaiduDrive](https://bit.ly/3LE1xI6)(key: 38ds) |     186     |
| Unseen (Side Plain)   |  8.0 |[GoogleDrive](https://bit.ly/3BsGOSO) or [BaiduDrive](https://bit.ly/3sRZYNY)(key: k6m0) |     186     |
| **Landmarks**         |
| -                     |   -  |[GoogleDrive](https://bit.ly/354YOH0) or [BaiduDrive](https://bit.ly/3oWUCA4)(key: cusv) |     309     |

You can include `data_ext=.mov` in your command line to match the video file extension in the Lombard GRID dataset.

</details>


<details open>

<summary>TCD-TIMIT</summary>

<p> </p>

|     Components        |  WER |                                             url                                         |  size (MB)  |
|:----------------------|:----:|:---------------------------------------------------------------------------------------:|:-----------:|
|   **Visual-only**     |
| Overlapped            | 16.9 |[GoogleDrive](https://bit.ly/3Fv7u61) or [BaiduDrive](https://bit.ly/33rPlZN)(key: jh65) |     186     |
| Unseen                | 21.8 |[GoogleDrive](https://bit.ly/3530d0N) or [BaiduDrive](https://bit.ly/3nxZjzC)(key: n2gr) |     186     |
| **Language models**   |
| -                     |   -  |[GoogleDrive](https://bit.ly/3qzWKit) or [BaiduDrive](https://bit.ly/3KgAL7T)(key: 59u2) |     180     |
| **Landmarks**         |
| -                     |   -  |[GoogleDrive](https://bit.ly/3HYmifr) or [BaiduDrive](https://bit.ly/3JFJ6RH)(key: bnm8) |     930     |

</details>


## Citation

If you use the AutoAVSR models, please consider citing the following paper:

```bibtex
@inproceedings{ma2023auto,
  author={Ma, Pingchuan and Haliassos, Alexandros and Fernandez-Lopez, Adriana and Chen, Honglie and Petridis, Stavros and Pantic, Maja},
  booktitle={IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)},
  title={Auto-AVSR: Audio-Visual Speech Recognition with Automatic Labels}, 
  year={2023},
}
```

If you use the VSR models for multiple languages please consider citing the following paper:

```bibtex
@article{ma2022visual,
  title={{Visual Speech Recognition for Multiple Languages in the Wild}},
  author={Ma, Pingchuan and Petridis, Stavros and Pantic, Maja},
  journal={{Nature Machine Intelligence}},
  volume={4},
  pages={930--939},
  year={2022}
  url={https://doi.org/10.1038/s42256-022-00550-z},
  doi={10.1038/s42256-022-00550-z}
}
```

## License

It is noted that the code can only be used for comparative or benchmarking purposes. Users can only use code supplied under a [License](./LICENSE) for non-commercial purposes.

## Contact

```
[Pingchuan Ma](pingchuan.ma16[at]imperial.ac.uk)
```