# Evaluating SeamlessM4T models | |
Refer to the [SeamlessM4T README](../../../../../docs/m4t) for an overview of the M4T models. | |
Refer to the [inference README](../predict/README.md) for how to run inference with SeamlessM4T models. | |
## Quick start: | |
We use SACREBLEU library for computing BLEU scores and [JiWER library](https://github.com/jitsi/jiwer) is used to compute these CER and WER scores. | |
Evaluation can be run with the CLI, from the root directory of the repository. | |
The model can be specified with `--model_name`: `seamlessM4T_v2_large` or `seamlessM4T_large` or `seamlessM4T_medium` | |
```bash | |
m4t_evaluate --data_file <path_to_data_tsv_file> --task <task_name> --tgt_lang <tgt_lang> --output_path <path_to_save_evaluation_output> --ref_field <ref_field_name> --audio_root_dir <path_to_audio_root_directory> | |
``` | |
## Note | |
1. We use raw (unnormalized) references to compute BLEU scores for S2TT, T2TT tasks. | |
2. For ASR task, src_lang needs to be passed as <tgt_lang> | |
3. `--src_lang` arg needs to be specified to run evaluation for T2TT task | |