speech-test commited on
Commit
60a812d
1 Parent(s): a2ec440

Update info

Browse files
Files changed (1) hide show
  1. README.md +74 -11
README.md CHANGED
@@ -4,27 +4,90 @@ datasets:
4
  - superb
5
  tags:
6
  - speech
 
7
  - hubert
8
- - s3prl
9
  license: apache-2.0
10
  ---
11
 
12
  # Hubert-Base for Emotion Recognition
13
 
14
- [S3PRL speech toolkit](https://github.com/s3prl/s3prl)
15
 
16
- [Facebook's Hubert](https://ai.facebook.com/blog/hubert-self-supervised-representation-learning-for-speech-recognition-generation-and-compression)
 
17
 
18
- The base model is pretrained on 16kHz sampled speech audio. When using the model make sure that your speech input is also sampled at 16Khz.
19
- The classification head is trained using the Emotion Recognition part of the SUPERB dataset.
20
 
21
- [Paper](https://arxiv.org/abs/2105.01051)
22
 
23
- Authors: Shu-wen Yang, Po-Han Chi, Yung-Sung Chuang, Cheng-I Jeff Lai, Kushal Lakhotia, Yist Y. Lin, Andy T. Liu, Jiatong Shi, Xuankai Chang, Guan-Ting Lin, Tzu-Hsien Huang, Wei-Cheng Tseng, Ko-tik Lee, Da-Rong Liu, Zili Huang, Shuyan Dong, Shang-Wen Li, Shinji Watanabe, Abdelrahman Mohamed, Hung-yi Lee
24
 
25
- **Abstract**
26
- Self-supervised learning (SSL) has proven vital for advancing research in natural language processing (NLP) and computer vision (CV). The paradigm pretrains a shared model on large volumes of unlabeled data and achieves state-of-the-art (SOTA) for various tasks with minimal adaptation. However, the speech processing community lacks a similar setup to systematically explore the paradigm. To bridge this gap, we introduce Speech processing Universal PERformance Benchmark (SUPERB). SUPERB is a leaderboard to benchmark the performance of a shared model across a wide range of speech processing tasks with minimal architecture changes and labeled data. Among multiple usages of the shared model, we especially focus on extracting the representation learned from SSL due to its preferable re-usability. We present a simple framework to solve SUPERB tasks by learning task-specialized lightweight prediction heads on top of the frozen shared model. Our results demonstrate that the framework is promising as SSL representations show competitive generalizability and accessibility across SUPERB tasks. We release SUPERB as a challenge with a leaderboard and a benchmark toolkit to fuel the research in representation learning and general speech processing.
 
 
27
 
28
- The original model can be found under https://github.com/s3prl/s3prl/tree/master/s3prl/downstream/emotion
 
29
 
30
- # Usage
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
4
  - superb
5
  tags:
6
  - speech
7
+ - audio
8
  - hubert
9
+ - audio-classification
10
  license: apache-2.0
11
  ---
12
 
13
  # Hubert-Base for Emotion Recognition
14
 
15
+ ## Model description
16
 
17
+ This is a ported version of
18
+ [S3PRL's Hubert for the SUPERB Emotion Recognition task](https://github.com/s3prl/s3prl/tree/master/s3prl/downstream/emotion).
19
 
20
+ The base model is [hubert-base-ls960](https://huggingface.co/facebook/hubert-base-ls960), which is pretrained on 16kHz
21
+ sampled speech audio. When using the model make sure that your speech input is also sampled at 16Khz.
22
 
23
+ For more information refer to [SUPERB: Speech processing Universal PERformance Benchmark](https://arxiv.org/abs/2105.01051)
24
 
25
+ ## Task and dataset description
26
 
27
+ Emotion Recognition (ER) predicts an emotion class for each utterance. The most widely used ER dataset
28
+ [IEMOCAP](https://sail.usc.edu/iemocap/) is adopted, and we follow the conventional evaluation protocol:
29
+ we drop the unbalanced emotion classes to leave the final four classes with a similar amount of data points and
30
+ cross-validate on five folds of the standard splits.
31
 
32
+ For the original model's training and evaluation instructions refer to the
33
+ [S3PRL downstream task README](https://github.com/s3prl/s3prl/tree/master/s3prl/downstream#er-emotion-recognition).
34
 
35
+
36
+ ## Usage examples
37
+
38
+ You can use the model via the Audio Classification pipeline:
39
+ ```python
40
+ from datasets import load_dataset
41
+ from transformers import pipeline
42
+
43
+ dataset = load_dataset("anton-l/superb_demo", "er", split="session1")
44
+
45
+ classifier = pipeline("audio-classification", model="superb/hubert-base-superb-er")
46
+ labels = classifier(dataset[0]["file"], top_k=5)
47
+ ```
48
+
49
+ Or use the model directly:
50
+ ```python
51
+ import torch
52
+ import librosa
53
+ from datasets import load_dataset
54
+ from transformers import HubertForSequenceClassification, Wav2Vec2FeatureExtractor
55
+
56
+ def map_to_array(example):
57
+ speech, _ = librosa.load(example["file"], sr=16000, mono=True)
58
+ example["speech"] = speech
59
+ return example
60
+
61
+ # load a demo dataset and read audio files
62
+ dataset = load_dataset("anton-l/superb_demo", "er", split="session1")
63
+ dataset = dataset.map(map_to_array)
64
+
65
+ model = HubertForSequenceClassification.from_pretrained("superb/hubert-base-superb-er")
66
+ feature_extractor = Wav2Vec2FeatureExtractor.from_pretrained("superb/hubert-base-superb-er")
67
+
68
+ # compute attention masks and normalize the waveform if needed
69
+ inputs = feature_extractor(dataset[:4]["speech"], sampling_rate=16000, padding=True, return_tensors="pt")
70
+
71
+ logits = model(**inputs).logits
72
+ predicted_ids = torch.argmax(logits, dim=-1)
73
+ labels = [model.config.id2label[_id] for _id in predicted_ids.tolist()]
74
+ ```
75
+
76
+ ## Eval results
77
+
78
+ The evaluation metric is accuracy.
79
+
80
+ | | **s3prl** | **transformers** |
81
+ |--------|-----------|------------------|
82
+ |**session1**| `0.6492` | `0.6359` |
83
+
84
+ ### BibTeX entry and citation info
85
+
86
+ ```bibtex
87
+ @article{yang2021superb,
88
+ title={SUPERB: Speech processing Universal PERformance Benchmark},
89
+ author={Yang, Shu-wen and Chi, Po-Han and Chuang, Yung-Sung and Lai, Cheng-I Jeff and Lakhotia, Kushal and Lin, Yist Y and Liu, Andy T and Shi, Jiatong and Chang, Xuankai and Lin, Guan-Ting and others},
90
+ journal={arXiv preprint arXiv:2105.01051},
91
+ year={2021}
92
+ }
93
+ ```