wav2vec2-conformer-rel-pos-large-finetuned-speech-commands

Model description

This model is a fine-tuned version of facebook/wav2vec2-conformer-rel-pos-large on the speech_commands dataset.

It achieves the following results on the evaluation set:

• Loss: 0.5245
• Accuracy: 0.9724

Intended uses & limitations

The model can spot one of the following keywords: "Yes", "No", "Up", "Down", "Left", "Right", "On", "Off", "Stop", "Go", "Zero", "One", "Two", "Three", "Four", "Five", "Six", "Seven", "Eight", "Nine", "Bed", "Bird", "Cat", "Dog", "Happy", "House", "Marvin", "Sheila", "Tree", "Wow", "Backward", "Forward", "Follow", "Learn", "Visual".

The repository includes sample files that I recorded (WAV, 16Khz sampling rate, mono). The simplest way to use the model is with the pipeline API:

>>> from transformers import pipeline
>>> p = pipeline("audio-classification", model="juliensimon/wav2vec2-conformer-rel-pos-large-finetuned-speech-commands")
>>> p("up16k.wav")
[{'score': 0.7008192539215088, 'label': 'up'}, {'score': 0.04346614331007004, 'label': 'off'}, {'score': 0.029526518657803535, 'label': 'left'}, {'score': 0.02905120886862278, 'label': 'stop'}, {'score': 0.027142534032464027, 'label': 'on'}]
>>> p("stop16k.wav")
[{'score': 0.6969656944274902, 'label': 'stop'}, {'score': 0.03391443192958832, 'label': 'up'}, {'score': 0.027382319793105125, 'label': 'seven'}, {'score': 0.020835857838392258, 'label': 'five'}, {'score': 0.018051736056804657, 'label': 'down'}]
>>> p("marvin16k.wav")
[{'score': 0.5276530981063843, 'label': 'marvin'}, {'score': 0.04645705968141556, 'label': 'down'}, {'score': 0.038583893328905106, 'label': 'backward'}, {'score': 0.03578080236911774, 'label': 'wow'}, {'score': 0.03178196772933006, 'label': 'bird'}]

You can also use them with the AutoAPI:

>>> import torch, librosa
>>> from transformers import AutoModelForAudioClassification, Wav2Vec2FeatureExtractor
>>> feature_extractor = Wav2Vec2FeatureExtractor()
>>> model = AutoModelForAudioClassification.from_pretrained("juliensimon/wav2vec2-conformer-rel-pos-large-finetuned-speech-commands")
>>> audio, rate = librosa.load("up16k.wav", sr = 16000)
>>> inputs = feature_extractor(audio, sampling_rate=16000, return_tensors = "pt")
>>> logits = model(inputs['input_values'])
>>> logits
SequenceClassifierOutput(loss=None, logits=tensor([[-0.4635, -1.0112,  4.7935,  0.8528,  1.6265,  0.6456,  1.5423,  2.0132,
          1.6103,  0.5847, -2.2526,  0.8839,  0.8163, -1.5655, -1.4160, -0.4196,
         -0.1097, -1.8827,  0.6609, -0.2022,  0.0971, -0.6205,  0.4492,  0.0926,
         -2.4848,  0.2630, -0.4584, -2.4327, -1.1654,  0.3897, -0.3374, -1.2418,
         -0.1045,  0.2827, -1.5667, -0.0963]], grad_fn=<AddmmBackward0>), hidden_states=None, attentions=None)
>>> classes = torch.softmax(logits.logits, dim=1)
>>> torch.set_printoptions(precision=3, sci_mode=False)
>>> classes
tensor([[    0.004,     0.002,     0.701,     0.014,     0.030,     0.011,
             0.027,     0.043,     0.029,     0.010,     0.001,     0.014,
             0.013,     0.001,     0.001,     0.004,     0.005,     0.001,
             0.011,     0.005,     0.006,     0.003,     0.009,     0.006,
             0.000,     0.008,     0.004,     0.001,     0.002,     0.009,
             0.004,     0.002,     0.005,     0.008,     0.001,     0.005]],
       grad_fn=<SoftmaxBackward0>)
>>> top_class = torch.argmax(logits.logits, dim=1)
>>> top_class
tensor([2])
>>> model.config.id2label[top_class.numpy()[0]]
'up'

Training and evaluation data

• subset: v0.02
• full training set
• full validation set

Training procedure

The model was fine-tuned on Amazon SageMaker, using an ml.p3dn.24xlarge instance (8 NVIDIA V100 GPUs). Total training time for 10 epochs was 4.5 hours.

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 3e-05
• train_batch_size: 256
• eval_batch_size: 256
• seed: 42
• gradient_accumulation_steps: 4
• total_train_batch_size: 1024
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• lr_scheduler_warmup_ratio: 0.1
• num_epochs: 10

Training results

Training Loss	Epoch	Step	Validation Loss	Accuracy
2.2901	1.0	83	2.0542	0.8875
1.8375	2.0	166	1.5610	0.9316
1.4957	3.0	249	1.1850	0.9558
1.1917	4.0	332	0.9159	0.9695
1.0449	5.0	415	0.7624	0.9687
0.9319	6.0	498	0.6444	0.9715
0.8559	7.0	581	0.5806	0.9711
0.8199	8.0	664	0.5394	0.9721
0.7949	9.0	747	0.5245	0.9724
0.7975	10.0	830	0.5256	0.9721

Framework versions

• Transformers 4.20.1
• Pytorch 1.11.0+cu102
• Datasets 2.3.2
• Tokenizers 0.12.1