|
--- |
|
language: "en" |
|
thumbnail: |
|
tags: |
|
- speechbrain |
|
- embeddings |
|
- Speaker |
|
- Verification |
|
- Identification |
|
- pytorch |
|
- ECAPA-TDNN |
|
license: "apache-2.0" |
|
datasets: |
|
- voxceleb |
|
metrics: |
|
- EER |
|
- Accuracy |
|
inference: true |
|
widget: |
|
- example_title: VoxCeleb Speaker id10003 |
|
src: https://cdn-media.huggingface.co/speech_samples/VoxCeleb1_00003.wav |
|
- example_title: VoxCeleb Speaker id10004 |
|
src: https://cdn-media.huggingface.co/speech_samples/VoxCeleb_00004.wav |
|
--- |
|
|
|
# Speaker Identification with ECAPA-TDNN embeddings on Voxceleb |
|
|
|
This repository provides a pretrained TDNN model (x-vector) using SpeechBrain. The system can be used to extract speaker embeddings as well. Since we can't find any resource that has SpeechBrain or HuggingFace compatible checkpoints that has only been trained on VoxCeleb2 development data, so we decide to pre-train an TDNN system from scratch. |
|
|
|
# Pipeline description |
|
|
|
This system is composed of an TDNN model (x-vector). It is a combination of convolutional and residual blocks. The embeddings are extracted using temporal statistical pooling. The system is trained with Additive Margin Softmax Loss. |
|
|
|
We use FBank (16kHz, 25ms frame length, 10ms hop length, 80 filter-bank channels) as the input features. It was trained using initial learning rate of 0.001 and batch size of 512 with linear scheduler for 30 epochs on 4 A100 GPUs. We employ additive noises and reverberation from [MUSAN](http://www.openslr.org/17/) and [RIR](http://www.openslr.org/28/) datasets to enrich the supervised information. The pre-training progress takes approximately seven days for the TDNN model. |
|
|
|
# Performance |
|
|
|
**VoxCeleb1-O** is the original verification test set from VoxCeleb1 consisting of 40 speakers. All speakers with names starting with "E" are reserved for testing. **VoxCeleb1-E** uses the entire VoxCeleb1 dataset, covering 1251 speakers. **VoxCeleb1-H** is a hard version of evaluation set consisting of 552536 pairs with 1190 speakers with the same nationality and gender. There are 18 nationality-gender combinations each with at least 5 individuals. |
|
|
|
| Splits | Backend | S-norm | EER(%) | minDCF(0.01) | |
|
|:-------------:|:--------------:|:--------------:|:--------------:|:--------------:| |
|
| VoxCeleb1-O | cosine | no | TBD | TBD | |
|
| VoxCeleb1-O | cosine | yes | TBD | TBD | |
|
| VoxCeleb1-E | cosine | no | TBD | TBD | |
|
| VoxCeleb1-E | cosine | yes | TBD | TBD | |
|
| VoxCeleb1-H | cosine | no | TBD | TBD | |
|
| VoxCeleb1-H | cosine | yes | TBD | TBD | |
|
|
|
- VoxCeleb1-O: includes 37611 test pairs with 40 speakers. |
|
- VoxCeleb1-E: includes 579818 test pairs with 1251 speakers. |
|
- VoxCeleb1-H: includes 550894 test pairs with 1190 speakers. |
|
|
|
# Compute the speaker embeddings |
|
|
|
The system is trained with recordings sampled at 16kHz (single channel). |
|
|
|
```python |
|
import torch |
|
import torchaudio |
|
from speechbrain.pretrained.interfaces import Pretrained |
|
from speechbrain.pretrained import EncoderClassifier |
|
|
|
|
|
class Encoder(Pretrained): |
|
|
|
MODULES_NEEDED = [ |
|
"compute_features", |
|
"mean_var_norm", |
|
"embedding_model" |
|
] |
|
|
|
def __init__(self, *args, **kwargs): |
|
super().__init__(*args, **kwargs) |
|
|
|
def encode_batch(self, wavs, wav_lens=None, normalize=False): |
|
# Manage single waveforms in input |
|
if len(wavs.shape) == 1: |
|
wavs = wavs.unsqueeze(0) |
|
|
|
# Assign full length if wav_lens is not assigned |
|
if wav_lens is None: |
|
wav_lens = torch.ones(wavs.shape[0], device=self.device) |
|
|
|
# Storing waveform in the specified device |
|
wavs, wav_lens = wavs.to(self.device), wav_lens.to(self.device) |
|
wavs = wavs.float() |
|
|
|
# Computing features and embeddings |
|
feats = self.mods.compute_features(wavs) |
|
feats = self.mods.mean_var_norm(feats, wav_lens) |
|
embeddings = self.mods.embedding_model(feats, wav_lens) |
|
if normalize: |
|
embeddings = self.hparams.mean_var_norm_emb( |
|
embeddings, |
|
torch.ones(embeddings.shape[0], device=self.device) |
|
) |
|
return embeddings |
|
|
|
|
|
classifier = Encoder.from_hparams( |
|
source="yangwang825/tdnn-vox2" |
|
) |
|
signal, fs = torchaudio.load('spk1_snt1.wav') |
|
embeddings = classifier.encode_batch(signal) |
|
>>> torch.Size([1, 1, 192]) |
|
``` |
|
|
|
We will release our training results (models, logs, etc) shortly. |
