reach-vb HF staff JadeCopet commited on
Commit
324dd73
1 Parent(s): a026621

Update README.md (#10)

Browse files

- Update README.md (3062c5a1aabde0523eeed36002a92629e60f0268)
- Update README.md (2234698ffbce2e3270ce8d9453920306c9091b9f)


Co-authored-by: Jade Copet <JadeCopet@users.noreply.huggingface.co>

Files changed (1) hide show
  1. README.md +17 -8
README.md CHANGED
@@ -132,9 +132,9 @@ for idx, one_wav in enumerate(wav):
132
 
133
  **Model type:** MusicGen consists of an EnCodec model for audio tokenization, an auto-regressive language model based on the transformer architecture for music modeling. The model comes in different sizes: 300M, 1.5B and 3.3B parameters ; and two variants: a model trained for text-to-music generation task and a model trained for melody-guided music generation.
134
 
135
- **Paper or resources for more information:** More information can be found in the paper [Simple and Controllable Music Generation][https://arxiv.org/abs/2306.05284].
136
 
137
- **Citation details**:
138
  ```
139
  @misc{copet2023simple,
140
  title={Simple and Controllable Music Generation},
@@ -146,7 +146,7 @@ for idx, one_wav in enumerate(wav):
146
  }
147
  ```
148
 
149
- **License** Code is released under MIT, model weights are released under CC-BY-NC 4.0.
150
 
151
  **Where to send questions or comments about the model:** Questions and comments about MusicGen can be sent via the [Github repository](https://github.com/facebookresearch/audiocraft) of the project, or by opening an issue.
152
 
@@ -158,7 +158,7 @@ for idx, one_wav in enumerate(wav):
158
 
159
  **Primary intended users:** The primary intended users of the model are researchers in audio, machine learning and artificial intelligence, as well as amateur seeking to better understand those models.
160
 
161
- **Out-of-scope use cases** The model should not be used on downstream applications without further risk evaluation and mitigation. The model should not be used to intentionally create or disseminate music pieces that create hostile or alienating environments for people. This includes generating music that people would foreseeably find disturbing, distressing, or offensive; or content that propagates historical or current stereotypes.
162
 
163
  ## Metrics
164
 
@@ -184,17 +184,26 @@ The model was evaluated on the [MusicCaps benchmark](https://www.kaggle.com/data
184
 
185
  ## Training datasets
186
 
187
- The model was trained using the following sources: the [Meta Music Initiative Sound Collection](https://www.fb.com/sound), [Shutterstock music collection](https://www.shutterstock.com/music) and the [Pond5 music collection](https://www.pond5.com/). See the paper for more details about the training set and corresponding preprocessing.
188
 
189
- ## Quantitative analysis
190
 
191
- More information can be found in the paper [Simple and Controllable Music Generation][arxiv], in the Experimental Setup section.
 
 
 
 
 
 
 
 
 
192
 
193
  ## Limitations and biases
194
 
195
  **Data:** The data sources used to train the model are created by music professionals and covered by legal agreements with the right holders. The model is trained on 20K hours of data, we believe that scaling the model on larger datasets can further improve the performance of the model.
196
 
197
- **Mitigations:** All vocals have been removed from the data source using a state-of-the-art music source separation method, namely using the open source [Hybrid Transformer for Music Source Separation](https://github.com/facebookresearch/demucs) (HT-Demucs). The model is therefore not able to produce vocals.
198
 
199
  **Limitations:**
200
 
 
132
 
133
  **Model type:** MusicGen consists of an EnCodec model for audio tokenization, an auto-regressive language model based on the transformer architecture for music modeling. The model comes in different sizes: 300M, 1.5B and 3.3B parameters ; and two variants: a model trained for text-to-music generation task and a model trained for melody-guided music generation.
134
 
135
+ **Paper or resources for more information:** More information can be found in the paper [Simple and Controllable Music Generation](https://arxiv.org/abs/2306.05284).
136
 
137
+ **Citation details:**
138
  ```
139
  @misc{copet2023simple,
140
  title={Simple and Controllable Music Generation},
 
146
  }
147
  ```
148
 
149
+ **License:** Code is released under MIT, model weights are released under CC-BY-NC 4.0.
150
 
151
  **Where to send questions or comments about the model:** Questions and comments about MusicGen can be sent via the [Github repository](https://github.com/facebookresearch/audiocraft) of the project, or by opening an issue.
152
 
 
158
 
159
  **Primary intended users:** The primary intended users of the model are researchers in audio, machine learning and artificial intelligence, as well as amateur seeking to better understand those models.
160
 
161
+ **Out-of-scope use cases:** The model should not be used on downstream applications without further risk evaluation and mitigation. The model should not be used to intentionally create or disseminate music pieces that create hostile or alienating environments for people. This includes generating music that people would foreseeably find disturbing, distressing, or offensive; or content that propagates historical or current stereotypes.
162
 
163
  ## Metrics
164
 
 
184
 
185
  ## Training datasets
186
 
187
+ The model was trained on licensed data using the following sources: the [Meta Music Initiative Sound Collection](https://www.fb.com/sound), [Shutterstock music collection](https://www.shutterstock.com/music) and the [Pond5 music collection](https://www.pond5.com/). See the paper for more details about the training set and corresponding preprocessing.
188
 
189
+ ## Evaluation results
190
 
191
+ Below are the objective metrics obtained on MusicCaps with the released model. Note that for the publicly released models, we had all the datasets go through a state-of-the-art music source separation method, namely using the open source [Hybrid Transformer for Music Source Separation](https://github.com/facebookresearch/demucs) (HT-Demucs), in order to keep only the instrumental part. This explains the difference in objective metrics with the models used in the paper.
192
+
193
+ | Model | Frechet Audio Distance | KLD | Text Consistency | Chroma Cosine Similarity |
194
+ |---|---|---|---|---|
195
+ | facebook/musicgen-small | 4.88 | 1.42 | 0.27 | - |
196
+ | facebook/musicgen-medium | 5.14 | 1.38 | 0.28 | - |
197
+ | **facebook/musicgen-large** | 5.48 | 1.37 | 0.28 | - |
198
+ | facebook/musicgen-melody | 4.93 | 1.41 | 0.27 | 0.44 |
199
+
200
+ More information can be found in the paper [Simple and Controllable Music Generation](https://arxiv.org/abs/2306.05284), in the Results section.
201
 
202
  ## Limitations and biases
203
 
204
  **Data:** The data sources used to train the model are created by music professionals and covered by legal agreements with the right holders. The model is trained on 20K hours of data, we believe that scaling the model on larger datasets can further improve the performance of the model.
205
 
206
+ **Mitigations:** Vocals have been removed from the data source using corresponding tags, and then using a state-of-the-art music source separation method, namely using the open source [Hybrid Transformer for Music Source Separation](https://github.com/facebookresearch/demucs) (HT-Demucs).
207
 
208
  **Limitations:**
209