reach-vb HF staff sanchit-gandhi HF staff commited on
Commit
a1e44be
1 Parent(s): a27233a

Update README (#3)

Browse files

- Update weights and README.md (dbc578f57f4f260f7e386a8e8d9e7b4e1d282a4a)


Co-authored-by: Sanchit Gandhi <sanchit-gandhi@users.noreply.huggingface.co>

Files changed (2) hide show
  1. .gitattributes +1 -0
  2. README.md +11 -12
.gitattributes CHANGED
@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
 
 
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
36
+ *.ptl filter=lfs diff=lfs merge=lfs -text
README.md CHANGED
@@ -15,7 +15,8 @@ SeamlessM4T covers:
15
  - 🗣️ 35 languages for speech output.
16
 
17
  Apart from [SeamlessM4T-LARGE (2.3B)](https://huggingface.co/facebook/seamless-m4t-large) and [SeamlessM4T-MEDIUM (1.2B)](https://huggingface.co/facebook/seamless-m4t-medium) models, we are also developing a small model (281M) targeting for on-device inference.
18
- [This folder](https://huggingface.co/facebook/seamless-m4t-unity-small) contains an example to run an exported small model covering most tasks (ASR/S2TT/S2ST). The model could be executed on popular mobile devices with Pytorch Mobile (https://pytorch.org/mobile/home/).
 
19
 
20
  ## Overview
21
  | Model | Checkpoint | Num Params | Disk Size | Supported Tasks | Supported Languages|
@@ -25,30 +26,28 @@ Apart from [SeamlessM4T-LARGE (2.3B)](https://huggingface.co/facebook/seamless-m
25
 
26
  UnitY-Small-S2T is a pruned version of UnitY-Small without 2nd pass unit decoding.
27
 
28
- Note: If using pytorch runtime in python, only **pytorch<=1.11.0** is supported for **UnitY-Small(281M)**. We tested UnitY-Small-S2T(235M), it works with later versions.
29
-
30
  ## Inference
31
  To use exported model, users don't need seamless_communication or fairseq2 dependency.
32
 
33
  ```python
34
  import torchaudio
35
  import torch
36
- audio_input, _ = torchaudio.load(TEST_AUDIO_PATH) # Load waveform using torchaudio
37
 
38
- s2t_model = torch.jit.load("unity_on_device_s2t.ptl") # Load exported S2T model
39
- text = s2t_model(audio_input, tgt_lang=TGT_LANG) # Forward call with tgt_lang specified for ASR or S2TT
40
- print(f"{lang}:{text}")
41
 
42
  s2st_model = torch.jit.load("unity_on_device.ptl")
43
- text, units, waveform = s2st_model(audio_input, tgt_lang=TGT_LANG) # S2ST model also returns waveform
44
- print(f"{lang}:{text}")
45
- torchaudio.save(f"{OUTPUT_FOLDER}/{lang}.wav", waveform.unsqueeze(0), sample_rate=16000) # Save output waveform to local file
 
 
 
46
  ```
47
 
48
  Also running the exported model doesn't need python runtime. For example, you could load this model in C++ following [this tutorial](https://pytorch.org/tutorials/advanced/cpp_export.html), or building your own on-device applications similar to [this example](https://github.com/pytorch/ios-demo-app/tree/master/SpeechRecognition)
49
 
50
  # Citation
51
- If you use SeamlessM4T in your work or any models/datasets/artifacts published in SeamlessM4T, please cite :
52
 
53
  ```bibtex
54
  @article{seamlessm4t2023,
@@ -60,4 +59,4 @@ If you use SeamlessM4T in your work or any models/datasets/artifacts published i
60
  ```
61
  # License
62
 
63
- seamless_communication is CC-BY-NC 4.0 licensed
 
15
  - 🗣️ 35 languages for speech output.
16
 
17
  Apart from [SeamlessM4T-LARGE (2.3B)](https://huggingface.co/facebook/seamless-m4t-large) and [SeamlessM4T-MEDIUM (1.2B)](https://huggingface.co/facebook/seamless-m4t-medium) models, we are also developing a small model (281M) targeting for on-device inference.
18
+
19
+ This README contains an example to run an exported small model covering most tasks (ASR/S2TT/S2ST). The model could be executed on popular mobile devices with Pytorch Mobile (https://pytorch.org/mobile/home/).
20
 
21
  ## Overview
22
  | Model | Checkpoint | Num Params | Disk Size | Supported Tasks | Supported Languages|
 
26
 
27
  UnitY-Small-S2T is a pruned version of UnitY-Small without 2nd pass unit decoding.
28
 
 
 
29
  ## Inference
30
  To use exported model, users don't need seamless_communication or fairseq2 dependency.
31
 
32
  ```python
33
  import torchaudio
34
  import torch
 
35
 
36
+ audio_input, _ = torchaudio.load(TEST_AUDIO_PATH) # Load waveform using torchaudio
 
 
37
 
38
  s2st_model = torch.jit.load("unity_on_device.ptl")
39
+
40
+ with torch.no_grad():
41
+ text, units, waveform = s2st_model(audio_input, tgt_lang=TGT_LANG) # S2ST model also returns waveform
42
+
43
+ print(text)
44
+ torchaudio.save(f"{OUTPUT_FOLDER}/result.wav", waveform.unsqueeze(0), sample_rate=16000) # Save output waveform to local file
45
  ```
46
 
47
  Also running the exported model doesn't need python runtime. For example, you could load this model in C++ following [this tutorial](https://pytorch.org/tutorials/advanced/cpp_export.html), or building your own on-device applications similar to [this example](https://github.com/pytorch/ios-demo-app/tree/master/SpeechRecognition)
48
 
49
  # Citation
50
+ If you use SeamlessM4T in your work or any models/datasets/artifacts published in SeamlessM4T, please cite:
51
 
52
  ```bibtex
53
  @article{seamlessm4t2023,
 
59
  ```
60
  # License
61
 
62
+ seamless_communication is CC-BY-NC 4.0 licensed