Update with HF Implementation
#4
by
sanchit-gandhi
HF staff
- opened
README.md
CHANGED
@@ -6,9 +6,9 @@ license: cc-by-nc-4.0
|
|
6 |
|
7 |
# MusicGen - Large - 3.3B
|
8 |
|
9 |
-
|
10 |
-
|
11 |
-
Unlike existing methods like MusicLM, MusicGen doesn't
|
12 |
By introducing a small delay between the codebooks, we show we can predict them in parallel, thus having only 50 auto-regressive steps per second of audio.
|
13 |
|
14 |
MusicGen was published in [Simple and Controllable Music Generation](https://arxiv.org/abs/2306.05284) by *Jade Copet, Felix Kreuk, Itai Gat, Tal Remez, David Kant, Gabriel Synnaeve, Yossi Adi, Alexandre Défossez*.
|
@@ -23,15 +23,75 @@ Four checkpoints are released:
|
|
23 |
|
24 |
Try out MusicGen yourself!
|
25 |
|
26 |
-
|
|
|
|
|
27 |
<img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/>
|
28 |
</a>
|
29 |
|
30 |
-
|
31 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
32 |
</a>
|
33 |
|
34 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
35 |
|
36 |
1. First install the [`audiocraft` library](https://github.com/facebookresearch/audiocraft)
|
37 |
```
|
@@ -46,16 +106,15 @@ apt get install ffmpeg
|
|
46 |
3. Run the following Python code:
|
47 |
|
48 |
```py
|
49 |
-
import torchaudio
|
50 |
from audiocraft.models import MusicGen
|
51 |
from audiocraft.data.audio import audio_write
|
52 |
|
53 |
-
model = MusicGen.get_pretrained(
|
54 |
model.set_generation_params(duration=8) # generate 8 seconds.
|
55 |
|
56 |
-
descriptions = [
|
57 |
|
58 |
-
wav = model.generate(descriptions) # generates
|
59 |
|
60 |
for idx, one_wav in enumerate(wav):
|
61 |
# Will save under {idx}.wav, with loudness normalization at -14 db LUFS.
|
|
|
6 |
|
7 |
# MusicGen - Large - 3.3B
|
8 |
|
9 |
+
MusicGen is a text-to-music model capable of genreating high-quality music samples conditioned on text descriptions or audio prompts.
|
10 |
+
It is a single stage auto-regressive Transformer model trained over a 32kHz EnCodec tokenizer with 4 codebooks sampled at 50 Hz.
|
11 |
+
Unlike existing methods, like MusicLM, MusicGen doesn't require a self-supervised semantic representation, and it generates all 4 codebooks in one pass.
|
12 |
By introducing a small delay between the codebooks, we show we can predict them in parallel, thus having only 50 auto-regressive steps per second of audio.
|
13 |
|
14 |
MusicGen was published in [Simple and Controllable Music Generation](https://arxiv.org/abs/2306.05284) by *Jade Copet, Felix Kreuk, Itai Gat, Tal Remez, David Kant, Gabriel Synnaeve, Yossi Adi, Alexandre Défossez*.
|
|
|
23 |
|
24 |
Try out MusicGen yourself!
|
25 |
|
26 |
+
* Audiocraft Colab:
|
27 |
+
|
28 |
+
<a target="_blank" href="https://colab.research.google.com/drive/1fxGqfg96RBUvGxZ1XXN07s3DthrKUl4-?usp=sharing">
|
29 |
<img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/>
|
30 |
</a>
|
31 |
|
32 |
+
* Hugging Face Colab:
|
33 |
+
|
34 |
+
<a target="_blank" href="https://colab.research.google.com/github/sanchit-gandhi/notebooks/blob/main/MusicGen.ipynb">
|
35 |
+
<img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/>
|
36 |
+
</a>
|
37 |
+
|
38 |
+
* Hugging Face Demo:
|
39 |
+
|
40 |
+
<a target="_blank" href="https://huggingface.co/spaces/facebook/MusicGen">
|
41 |
+
<img src="https://huggingface.co/datasets/huggingface/badges/raw/main/open-in-hf-spaces-sm.svg" alt="Open in HuggingFace"/>
|
42 |
</a>
|
43 |
|
44 |
+
## 🤗 Transformers Usage
|
45 |
+
|
46 |
+
You can run MusicGen locally with the 🤗 Transformers library from version 4.31.0 onwards.
|
47 |
+
|
48 |
+
1. First install the 🤗 [Transformers library](https://github.com/huggingface/transformers) from main:
|
49 |
+
|
50 |
+
```
|
51 |
+
pip install git+https://github.com/huggingface/transformers.git
|
52 |
+
```
|
53 |
+
|
54 |
+
2. Run the following Python code to generate text-conditional audio samples:
|
55 |
+
|
56 |
+
```py
|
57 |
+
from transformers import AutoProcessor, MusicgenForConditionalGeneration
|
58 |
+
|
59 |
+
|
60 |
+
processor = AutoProcessor.from_pretrained("facebook/musicgen-large")
|
61 |
+
model = MusicgenForConditionalGeneration.from_pretrained("facebook/musicgen-large")
|
62 |
+
|
63 |
+
inputs = processor(
|
64 |
+
text=["80s pop track with bassy drums and synth", "90s rock song with loud guitars and heavy drums"],
|
65 |
+
padding=True,
|
66 |
+
return_tensors="pt",
|
67 |
+
)
|
68 |
+
|
69 |
+
audio_values = model.generate(**inputs, do_sample=True, guidance_scale=3, max_new_tokens=256)
|
70 |
+
```
|
71 |
+
|
72 |
+
3. Listen to the audio samples either in an ipynb notebook:
|
73 |
+
|
74 |
+
```py
|
75 |
+
from IPython.display import Audio
|
76 |
+
|
77 |
+
sampling_rate = model.config.audio_encoder.sampling_rate
|
78 |
+
Audio(audio_values[0].numpy(), rate=sampling_rate)
|
79 |
+
```
|
80 |
+
|
81 |
+
Or save them as a `.wav` file using a third-party library, e.g. `scipy`:
|
82 |
+
|
83 |
+
```py
|
84 |
+
import scipy
|
85 |
+
|
86 |
+
sampling_rate = model.config.audio_encoder.sampling_rate
|
87 |
+
scipy.io.wavfile.write("musicgen_out.wav", rate=sampling_rate, data=audio_values[0, 0].numpy())
|
88 |
+
```
|
89 |
+
|
90 |
+
For more details on using the MusicGen model for inference using the 🤗 Transformers library, refer to the [MusicGen docs](https://huggingface.co/docs/transformers/model_doc/musicgen).
|
91 |
+
|
92 |
+
## Audiocraft Usage
|
93 |
+
|
94 |
+
You can also run MusicGen locally through the original [Audiocraft library]((https://github.com/facebookresearch/audiocraft):
|
95 |
|
96 |
1. First install the [`audiocraft` library](https://github.com/facebookresearch/audiocraft)
|
97 |
```
|
|
|
106 |
3. Run the following Python code:
|
107 |
|
108 |
```py
|
|
|
109 |
from audiocraft.models import MusicGen
|
110 |
from audiocraft.data.audio import audio_write
|
111 |
|
112 |
+
model = MusicGen.get_pretrained("large")
|
113 |
model.set_generation_params(duration=8) # generate 8 seconds.
|
114 |
|
115 |
+
descriptions = ["happy rock", "energetic EDM"]
|
116 |
|
117 |
+
wav = model.generate(descriptions) # generates 2 samples.
|
118 |
|
119 |
for idx, one_wav in enumerate(wav):
|
120 |
# Will save under {idx}.wav, with loudness normalization at -14 db LUFS.
|