metadata

license: apache-2.0
language: en
tags:
  - generated_from_trainer
datasets:
  - speech_commands
metrics:
  - accuracy
model-index:
  - name: wav2vec2-conformer-rel-pos-large-finetuned-speech-commands
    results:
      - task:
          type: audio-classification
          name: audio classification
        dataset:
          type: speech_commands
          name: speech_commands
          split: v0.02
        metrics:
          - type: accuracy
            value: 0.9724
            name: accuracy

wav2vec2-conformer-rel-pos-large-finetuned-speech-commands

Model description

This model is a fine-tuned version of facebook/wav2vec2-conformer-rel-pos-large on the speech_commands dataset.

It achieves the following results on the evaluation set:

Loss: 0.5245
Accuracy: 0.9724

Intended uses & limitations

The model can spot one of the following keywords: "Yes", "No", "Up", "Down", "Left", "Right", "On", "Off", "Stop", "Go", "Zero", "One", "Two", "Three", "Four", "Five", "Six", "Seven", "Eight", "Nine", "Bed", "Bird", "Cat", "Dog", "Happy", "House", "Marvin", "Sheila", "Tree", "Wow", "Backward", "Forward", "Follow", "Learn", "Visual".

The repository includes sample files that I recorded (WAV, 16Khz sampling rate, mono). The simplest way to use the model is with the pipeline API:

>>> from transformers import pipeline
>>> p = pipeline("audio-classification", model="juliensimon/wav2vec2-conformer-rel-pos-large-finetuned-speech-commands")
>>> p("up16k.wav")
[{'score': 0.7008192539215088, 'label': 'up'}, {'score': 0.04346614331007004, 'label': 'off'}, {'score': 0.029526518657803535, 'label': 'left'}, {'score': 0.02905120886862278, 'label': 'stop'}, {'score': 0.027142534032464027, 'label': 'on'}]
>>> p("stop16k.wav")
[{'score': 0.6969656944274902, 'label': 'stop'}, {'score': 0.03391443192958832, 'label': 'up'}, {'score': 0.027382319793105125, 'label': 'seven'}, {'score': 0.020835857838392258, 'label': 'five'}, {'score': 0.018051736056804657, 'label': 'down'}]
>>> p("marvin16k.wav")
[{'score': 0.5276530981063843, 'label': 'marvin'}, {'score': 0.04645705968141556, 'label': 'down'}, {'score': 0.038583893328905106, 'label': 'backward'}, {'score': 0.03578080236911774, 'label': 'wow'}, {'score': 0.03178196772933006, 'label': 'bird'}]

You can also use with the AutoAPI:

>>> import torch, librosa
>>> from transformers import AutoModelForAudioClassification, Wav2Vec2FeatureExtractor
>>> feature_extractor = Wav2Vec2FeatureExtractor()
>>> model = AutoModelForAudioClassification.from_pretrained("juliensimon/wav2vec2-conformer-rel-pos-large-finetuned-speech-commands")
>>> audio, rate = librosa.load("up16k.wav", sr = 16000)
>>> inputs = feature_extractor(audio, sampling_rate=16000, return_tensors = "pt")
>>> logits = model(inputs['input_values'])
>>> logits
SequenceClassifierOutput(loss=None, logits=tensor([[-0.4635, -1.0112,  4.7935,  0.8528,  1.6265,  0.6456,  1.5423,  2.0132,
          1.6103,  0.5847, -2.2526,  0.8839,  0.8163, -1.5655, -1.4160, -0.4196,
         -0.1097, -1.8827,  0.6609, -0.2022,  0.0971, -0.6205,  0.4492,  0.0926,
         -2.4848,  0.2630, -0.4584, -2.4327, -1.1654,  0.3897, -0.3374, -1.2418,
         -0.1045,  0.2827, -1.5667, -0.0963]], grad_fn=<AddmmBackward0>), hidden_states=None, attentions=None)
>>> classes = torch.softmax(logits.logits, dim = -1)
>>> torch.set_printoptions(precision=3, sci_mode=False)
>>> classes
tensor([[    0.004,     0.002,     0.701,     0.014,     0.030,     0.011,
             0.027,     0.043,     0.029,     0.010,     0.001,     0.014,
             0.013,     0.001,     0.001,     0.004,     0.005,     0.001,
             0.011,     0.005,     0.006,     0.003,     0.009,     0.006,
             0.000,     0.008,     0.004,     0.001,     0.002,     0.009,
             0.004,     0.002,     0.005,     0.008,     0.001,     0.005]],
       grad_fn=<SoftmaxBackward0>)
>>> top_class = torch.argmax(logits.logits, dim = -1)
>>> top_class
tensor([2])
>>> model.config.id2label[top_class.numpy()[0]]
'up'

Training and evaluation data

subset: v0.02
full training set
full validation set

Training procedure

The model was fine-tuned on Amazon SageMaker, using an ml.p3dn.24xlarge instance (8 NVIDIA V100 GPUs). Total training time for 10 epochs was 4.5 hours.

Training hyperparameters

The following hyperparameters were used during training:

learning_rate: 3e-05
train_batch_size: 256
eval_batch_size: 256
seed: 42
gradient_accumulation_steps: 4
total_train_batch_size: 1024
optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
lr_scheduler_type: linear
lr_scheduler_warmup_ratio: 0.1
num_epochs: 10

Training results

Training Loss	Epoch	Step	Validation Loss	Accuracy
2.2901	1.0	83	2.0542	0.8875
1.8375	2.0	166	1.5610	0.9316
1.4957	3.0	249	1.1850	0.9558
1.1917	4.0	332	0.9159	0.9695
1.0449	5.0	415	0.7624	0.9687
0.9319	6.0	498	0.6444	0.9715
0.8559	7.0	581	0.5806	0.9711
0.8199	8.0	664	0.5394	0.9721
0.7949	9.0	747	0.5245	0.9724
0.7975	10.0	830	0.5256	0.9721

Framework versions

Transformers 4.20.1
Pytorch 1.11.0+cu102
Datasets 2.3.2
Tokenizers 0.12.1