游雁 commited on
Commit
c0321ff
1 Parent(s): 157e795
Files changed (1) hide show
  1. README.md +175 -0
README.md CHANGED
@@ -3,3 +3,178 @@ license: other
3
  license_name: model-license
4
  license_link: https://github.com/alibaba-damo-academy/FunASR
5
  ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
3
  license_name: model-license
4
  license_link: https://github.com/alibaba-damo-academy/FunASR
5
  ---
6
+
7
+
8
+
9
+ # FunASR: A Fundamental End-to-End Speech Recognition Toolkit
10
+
11
+
12
+ [![PyPI](https://img.shields.io/pypi/v/funasr)](https://pypi.org/project/funasr/)
13
+
14
+
15
+ <strong>FunASR</strong> hopes to build a bridge between academic research and industrial applications on speech recognition. By supporting the training & finetuning of the industrial-grade speech recognition model, researchers and developers can conduct research and production of speech recognition models more conveniently, and promote the development of speech recognition ecology. ASR for Fun!
16
+
17
+ [**Highlights**](#highlights)
18
+ | [**News**](https://github.com/alibaba-damo-academy/FunASR#whats-new)
19
+ | [**Installation**](#installation)
20
+ | [**Quick Start**](#quick-start)
21
+ | [**Runtime**](./runtime/readme.md)
22
+ | [**Model Zoo**](#model-zoo)
23
+ | [**Contact**](#contact)
24
+
25
+
26
+ <a name="highlights"></a>
27
+ ## Highlights
28
+ - FunASR is a fundamental speech recognition toolkit that offers a variety of features, including speech recognition (ASR), Voice Activity Detection (VAD), Punctuation Restoration, Language Models, Speaker Verification, Speaker Diarization and multi-talker ASR. FunASR provides convenient scripts and tutorials, supporting inference and fine-tuning of pre-trained models.
29
+ - We have released a vast collection of academic and industrial pretrained models on the [ModelScope](https://www.modelscope.cn/models?page=1&tasks=auto-speech-recognition) and [huggingface](https://huggingface.co/FunASR), which can be accessed through our [Model Zoo](https://github.com/alibaba-damo-academy/FunASR/blob/main/docs/model_zoo/modelscope_models.md). The representative [Paraformer-large](https://www.modelscope.cn/models/damo/speech_paraformer-large_asr_nat-zh-cn-16k-common-vocab8404-pytorch/summary), a non-autoregressive end-to-end speech recognition model, has the advantages of high accuracy, high efficiency, and convenient deployment, supporting the rapid construction of speech recognition services. For more details on service deployment, please refer to the [service deployment document](runtime/readme_cn.md).
30
+
31
+
32
+ <a name="Installation"></a>
33
+ ## Installation
34
+
35
+ ```shell
36
+ pip3 install -U funasr
37
+ ```
38
+ Or install from source code
39
+ ``` sh
40
+ git clone https://github.com/alibaba/FunASR.git && cd FunASR
41
+ pip3 install -e ./
42
+ ```
43
+ Install modelscope for the pretrained models (Optional)
44
+
45
+ ```shell
46
+ pip3 install -U modelscope
47
+ ```
48
+
49
+ ## Model Zoo
50
+ FunASR has open-sourced a large number of pre-trained models on industrial data. You are free to use, copy, modify, and share FunASR models under the [Model License Agreement](./MODEL_LICENSE). Below are some representative models, for more models please refer to the [Model Zoo]().
51
+
52
+ (Note: 🤗 represents the Huggingface model zoo link, ⭐ represents the ModelScope model zoo link)
53
+
54
+
55
+ | Model Name | Task Details | Training Data | Parameters |
56
+ |:------------------------------------------------------------------------------------------------------------------------------------------------------------------:|:--------------------------------------------------:|:--------------------------------:|:----------:|
57
+ | paraformer-zh <br> ([⭐](https://www.modelscope.cn/models/damo/speech_paraformer-large-vad-punc_asr_nat-zh-cn-16k-common-vocab8404-pytorch/summary) [🤗]() ) | speech recognition, with timestamps, non-streaming | 60000 hours, Mandarin | 220M |
58
+ | <nobr>paraformer-zh-streaming <br> ( [⭐](https://modelscope.cn/models/damo/speech_paraformer-large_asr_nat-zh-cn-16k-common-vocab8404-online/summary) [🤗]() )</nobr> | speech recognition, streaming | 60000 hours, Mandarin | 220M |
59
+ | paraformer-en <br> ( [⭐](https://www.modelscope.cn/models/damo/speech_paraformer-large-vad-punc_asr_nat-en-16k-common-vocab10020/summary) [🤗]() ) | speech recognition, with timestamps, non-streaming | 50000 hours, English | 220M |
60
+ | conformer-en <br> ( [⭐](https://modelscope.cn/models/damo/speech_conformer_asr-en-16k-vocab4199-pytorch/summary) [🤗]() ) | speech recognition, non-streaming | 50000 hours, English | 220M |
61
+ | ct-punc <br> ( [⭐](https://modelscope.cn/models/damo/punc_ct-transformer_cn-en-common-vocab471067-large/summary) [🤗]() ) | punctuation restoration | 100M, Mandarin and English | 1.1G |
62
+ | fsmn-vad <br> ( [⭐](https://modelscope.cn/models/damo/speech_fsmn_vad_zh-cn-16k-common-pytorch/summary) [🤗]() ) | voice activity detection | 5000 hours, Mandarin and English | 0.4M |
63
+ | fa-zh <br> ( [⭐](https://modelscope.cn/models/damo/speech_timestamp_prediction-v1-16k-offline/summary) [🤗]() ) | timestamp prediction | 5000 hours, Mandarin | 38M |
64
+ | cam++ <br> ( [⭐](https://modelscope.cn/models/iic/speech_campplus_sv_zh-cn_16k-common/summary) [🤗]() ) | speaker verification/diarization | 5000 hours | 7.2M |
65
+
66
+
67
+
68
+
69
+ [//]: # ()
70
+ [//]: # (FunASR supports pre-trained or further fine-tuned models for deployment as a service. The CPU version of the Chinese offline file conversion service has been released, details can be found in [docs]&#40;funasr/runtime/docs/SDK_tutorial.md&#41;. More detailed information about service deployment can be found in the [deployment roadmap]&#40;funasr/runtime/readme_cn.md&#41;.)
71
+
72
+
73
+ <a name="quick-start"></a>
74
+ ## Quick Start
75
+
76
+ Below is a quick start tutorial. Test audio files ([Mandarin](https://isv-data.oss-cn-hangzhou.aliyuncs.com/ics/MaaS/ASR/test_audio/vad_example.wav), [English]()).
77
+
78
+ ### Command-line usage
79
+
80
+ ```shell
81
+ funasr +model=paraformer-zh +vad_model="fsmn-vad" +punc_model="ct-punc" +input=asr_example_zh.wav
82
+ ```
83
+
84
+ Notes: Support recognition of single audio file, as well as file list in Kaldi-style wav.scp format: `wav_id wav_pat`
85
+
86
+ ### Speech Recognition (Non-streaming)
87
+ ```python
88
+ from funasr import AutoModel
89
+ # paraformer-zh is a multi-functional asr model
90
+ # use vad, punc, spk or not as you need
91
+ model = AutoModel(model="paraformer-zh", model_revision="v2.0.4",
92
+ vad_model="fsmn-vad", vad_model_revision="v2.0.4",
93
+ punc_model="ct-punc-c", punc_model_revision="v2.0.4",
94
+ # spk_model="cam++", spk_model_revision="v2.0.2",
95
+ )
96
+ res = model.generate(input=f"{model.model_path}/example/asr_example.wav",
97
+ batch_size_s=300,
98
+ hotword='魔搭')
99
+ print(res)
100
+ ```
101
+ Note: `model_hub`: represents the model repository, `ms` stands for selecting ModelScope download, `hf` stands for selecting Huggingface download.
102
+
103
+ ### Speech Recognition (Streaming)
104
+ ```python
105
+ from funasr import AutoModel
106
+
107
+ chunk_size = [0, 10, 5] #[0, 10, 5] 600ms, [0, 8, 4] 480ms
108
+ encoder_chunk_look_back = 4 #number of chunks to lookback for encoder self-attention
109
+ decoder_chunk_look_back = 1 #number of encoder chunks to lookback for decoder cross-attention
110
+
111
+ model = AutoModel(model="paraformer-zh-streaming", model_revision="v2.0.4")
112
+
113
+ import soundfile
114
+ import os
115
+
116
+ wav_file = os.path.join(model.model_path, "example/asr_example.wav")
117
+ speech, sample_rate = soundfile.read(wav_file)
118
+ chunk_stride = chunk_size[1] * 960 # 600ms
119
+
120
+ cache = {}
121
+ total_chunk_num = int(len((speech)-1)/chunk_stride+1)
122
+ for i in range(total_chunk_num):
123
+ speech_chunk = speech[i*chunk_stride:(i+1)*chunk_stride]
124
+ is_final = i == total_chunk_num - 1
125
+ res = model.generate(input=speech_chunk, cache=cache, is_final=is_final, chunk_size=chunk_size, encoder_chunk_look_back=encoder_chunk_look_back, decoder_chunk_look_back=decoder_chunk_look_back)
126
+ print(res)
127
+ ```
128
+ Note: `chunk_size` is the configuration for streaming latency.` [0,10,5]` indicates that the real-time display granularity is `10*60=600ms`, and the lookahead information is `5*60=300ms`. Each inference input is `600ms` (sample points are `16000*0.6=960`), and the output is the corresponding text. For the last speech segment input, `is_final=True` needs to be set to force the output of the last word.
129
+
130
+ ### Voice Activity Detection (Non-Streaming)
131
+ ```python
132
+ from funasr import AutoModel
133
+
134
+ model = AutoModel(model="fsmn-vad", model_revision="v2.0.4")
135
+ wav_file = f"{model.model_path}/example/asr_example.wav"
136
+ res = model.generate(input=wav_file)
137
+ print(res)
138
+ ```
139
+ ### Voice Activity Detection (Streaming)
140
+ ```python
141
+ from funasr import AutoModel
142
+
143
+ chunk_size = 200 # ms
144
+ model = AutoModel(model="fsmn-vad", model_revision="v2.0.4")
145
+
146
+ import soundfile
147
+
148
+ wav_file = f"{model.model_path}/example/vad_example.wav"
149
+ speech, sample_rate = soundfile.read(wav_file)
150
+ chunk_stride = int(chunk_size * sample_rate / 1000)
151
+
152
+ cache = {}
153
+ total_chunk_num = int(len((speech)-1)/chunk_stride+1)
154
+ for i in range(total_chunk_num):
155
+ speech_chunk = speech[i*chunk_stride:(i+1)*chunk_stride]
156
+ is_final = i == total_chunk_num - 1
157
+ res = model.generate(input=speech_chunk, cache=cache, is_final=is_final, chunk_size=chunk_size)
158
+ if len(res[0]["value"]):
159
+ print(res)
160
+ ```
161
+ ### Punctuation Restoration
162
+ ```python
163
+ from funasr import AutoModel
164
+
165
+ model = AutoModel(model="ct-punc", model_revision="v2.0.4")
166
+ res = model.generate(input="那今天的会就到这里吧 happy new year 明年见")
167
+ print(res)
168
+ ```
169
+ ### Timestamp Prediction
170
+ ```python
171
+ from funasr import AutoModel
172
+
173
+ model = AutoModel(model="fa-zh", model_revision="v2.0.4")
174
+ wav_file = f"{model.model_path}/example/asr_example.wav"
175
+ text_file = f"{model.model_path}/example/text.txt"
176
+ res = model.generate(input=(wav_file, text_file), data_type=("sound", "text"))
177
+ print(res)
178
+ ```
179
+
180
+ More examples ref to [docs](https://github.com/alibaba-damo-academy/FunASR/tree/main/examples/industrial_data_pretraining)