# Evaluating SeamlessM4T models

Refer to the [SeamlessM4T README](../../../../../docs/m4t) for an overview of the M4T models.

Refer to the [inference README](../predict/README.md) for how to run inference with SeamlessM4T models.

## Quick start:
We use SACREBLEU library for computing BLEU scores and [JiWER library](https://github.com/jitsi/jiwer) is used to compute these CER and WER scores. 

Evaluation can be run with the CLI, from the root directory of the repository.

The model can be specified with `--model_name`: `seamlessM4T_v2_large` or `seamlessM4T_large` or `seamlessM4T_medium` 

```bash
m4t_evaluate --data_file <path_to_data_tsv_file> --task <task_name> --tgt_lang <tgt_lang> --output_path <path_to_save_evaluation_output> --ref_field <ref_field_name> --audio_root_dir <path_to_audio_root_directory>
```
## Note
1. We use raw (unnormalized) references to compute BLEU scores for S2TT, T2TT tasks.
2. For ASR task, src_lang needs to be passed as <tgt_lang> 
3. `--src_lang` arg needs to be specified to run evaluation for T2TT task