EMOTION2VEC

emotion2vec: universal speech emotion representation model
emotion2vec: Self-Supervised Pre-Training for Speech Emotion Representation

Guides

emotion2vec is the first universal speech emotion representation model. Through self-supervised pre-training, emotion2vec has the ability to extract emotion representation across different tasks, languages, and scenarios.

The version is an pre-trained representation model without fine-tuning, which can be used for feature extraction.

Model Card

GitHub Repo: emotion2vec

Model	⭐Model Scope	🤗Hugging Face	Fine-tuning Data (Hours)
emotion2vec	Link	Link	/
emotion2vec+ seed	Link	Link	201
emotion2vec+ base	Link	Link	4788
emotion2vec+ large	Link	Link	42526

Installation

pip install -U funasr modelscope

Usage

input: 16k Hz speech recording

granularity:

"utterance": Extract features from the entire utterance
"frame": Extract frame-level features (50 Hz)

extract_embedding: Whether to extract features

Inference based on ModelScope

from modelscope.pipelines import pipeline
from modelscope.utils.constant import Tasks

inference_pipeline = pipeline(
    task=Tasks.emotion_recognition,
    model="iic/emotion2vec_base")

rec_result = inference_pipeline('https://isv-data.oss-cn-hangzhou.aliyuncs.com/ics/MaaS/ASR/test_audio/asr_example_zh.wav', output_dir="./outputs", granularity="utterance", extract_embedding=True)
print(rec_result)

Inference based on FunASR

from funasr import AutoModel

model = AutoModel(model="iic/emotion2vec_base")

res = model(input='https://isv-data.oss-cn-hangzhou.aliyuncs.com/ics/MaaS/ASR/test_audio/asr_example_zh.wav', output_dir="./outputs", granularity="utterance", extract_embedding=True)
print(res)

Note: The model will automatically download.

Supports input file list, wav.scp (Kaldi style):

wav_name1 wav_path1.wav
wav_name2 wav_path2.wav
...

Outputs are emotion representation, saved in the output_dir in numpy format (can be loaded with np.load())

Note

This repository is the Huggingface version of emotion2vec, with identical model parameters as the original model and Model Scope version.

Original repository: https://github.com/ddlBoJack/emotion2vec

Model Scope repository: https://www.modelscope.cn/models/iic/emotion2vec_plus_large/summary

Hugging Face repository: https://huggingface.co/emotion2vec

FunASR repository: https://github.com/alibaba-damo-academy/FunASR

Citation

@article{ma2023emotion2vec,
  title={emotion2vec: Self-Supervised Pre-Training for Speech Emotion Representation},
  author={Ma, Ziyang and Zheng, Zhisheng and Ye, Jiaxin and Li, Jinchao and Gao, Zhifu and Zhang, Shiliang and Chen, Xie},
  journal={arXiv preprint arXiv:2312.15185},
  year={2023}
}