upload files

README.md

@@ -1,3 +1,112 @@
 ---
 license: apache-2.0
+datasets:
+- amaai-lab/MusicBench
+tags:
+- music
 ---
+
+<div align="center">
+
+# Mustango: Toward Controllable Text-to-Music Generation
+
+[Demo](https://replicate.com/declare-lab/mustango) | [Model](https://huggingface.co/declare-lab/mustango) | [Website and Examples](https://amaai-lab.github.io/mustango/) | [Paper](https://arxiv.org/abs/2311.08355) | [Dataset](https://huggingface.co/datasets/amaai-lab/MusicBench)
+
+[![Hugging Face Spaces](https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-blue)](https://huggingface.co/spaces/declare-lab/mustango)
+</div>
+
+Meet Mustango, an exciting addition to the vibrant landscape of Multimodal Large Language Models designed for controlled music generation. Mustango leverages Latent Diffusion Model (LDM), Flan-T5, and musical features to do the magic!
+
+🔥 Live demo available on [Replicate](https://replicate.com/declare-lab/mustango) and [HuggingFace](https://huggingface.co/spaces/declare-lab/mustango).
+
+<div align="center">
+  <img src="mustango.jpg" width="500"/>
+</div>
+
+
+## Quickstart Guide
+
+Generate music from a text prompt:
+
+```python
+import IPython
+import soundfile as sf
+from mustango import Mustango
+
+model = Mustango("declare-lab/mustango")
+
+prompt = "This is a new age piece. There is a flute playing the main melody with a lot of staccato notes. The rhythmic background consists of a medium tempo electronic drum beat with percussive elements all over the spectrum. There is a playful atmosphere to the piece. This piece can be used in the soundtrack of a children's TV show or an advertisement jingle."
+
+music = model.generate(prompt)
+sf.write(f"{prompt}.wav", audio, samplerate=16000)
+IPython.display.Audio(data=audio, rate=16000)
+```
+
+## Installation
+
+```bash
+git clone https://github.com/AMAAI-Lab/mustango
+cd mustango
+pip install -r requirements.txt
+cd diffusers
+pip install -e .
+```
+
+## Datasets
+
+The [MusicBench](https://huggingface.co/datasets/amaai-lab/MusicBench) dataset contains 52k music fragments with a rich music-specific text caption. 
+## Subjective Evaluation by Expert Listeners
+
+| **Model** | **Dataset** | **Pre-trained** | **Overall Match** ↑ | **Chord Match** ↑ | **Tempo Match** ↑ | **Audio Quality** ↑ | **Musicality** ↑ | **Rhythmic Presence and Stability** ↑ | **Harmony and Consonance** ↑ |
+|-----------|-------------|:-----------------:|:-----------:|:-----------:|:-----------:|:----------:|:----------:|:----------:|:----------:|
+| Tango     | MusicCaps   | ✓               | 4.35      | 2.75      | 3.88      | 3.35     | 2.83     | 3.95     | 3.84     |
+| Tango     | MusicBench  | ✓               | 4.91      | 3.61      | 3.86      | 3.88     | 3.54     | 4.01     | 4.34     |
+| Mustango  | MusicBench  | ✓               | 5.49      | 5.76      | 4.98      | 4.30     | 4.28     | 4.65     | 5.18     |
+| Mustango  | MusicBench  | ✗               | 5.75      | 6.06      | 5.11      | 4.80     | 4.80     | 4.75     | 5.59     |
+
+
+
+
+## Training
+
+We use the `accelerate` package from Hugging Face for multi-gpu training. Run `accelerate config` from terminal and set up your run configuration by the answering the questions asked.
+
+You can now train **Mustango** on the MusicBench dataset using:
+
+```bash
+accelerate launch train.py \
+--text_encoder_name="google/flan-t5-large" \
+--scheduler_name="stabilityai/stable-diffusion-2-1" \
+--unet_model_config="configs/diffusion_model_config_munet.json" \
+--model_type Mustango --freeze_text_encoder --uncondition_all --uncondition_single \
+--drop_sentences --random_pick_text_column --snr_gamma 5 \
+```
+
+The `--model_type` flag allows to choose either Mustango, or Tango to be trained with the same code. However, do note that you also need to change `--unet_model_config` to the relevant config: diffusion_model_config_munet for Mustango; diffusion_model_config for Tango.
+
+The arguments `--uncondition_all`, `--uncondition_single`, `--drop_sentences` control the dropout functions as per Section 5.2 in our paper. The argument of `--random_pick_text_column` allows to randomly pick between two input text prompts - in the case of MusicBench, we pick between ChatGPT rephrased captions and original enhanced MusicCaps prompts, as depicted in Figure 1 in our paper.
+
+Recommended training time from scratch on MusicBench is at least 40 epochs.
+
+
+## Model Zoo
+
+We have released the following models:
+
+Mustango Pretrained: https://huggingface.co/declare-lab/mustango-pretrained
+
+
+Mustango: https://huggingface.co/declare-lab/mustango
+
+
+## Citation
+Please consider citing the following article if you found our work useful:
+```
+@misc{melechovsky2023mustango,
+      title={Mustango: Toward Controllable Text-to-Music Generation}, 
+      author={Jan Melechovsky and Zixun Guo and Deepanway Ghosal and Navonil Majumder and Dorien Herremans and Soujanya Poria},
+      year={2023},
+      eprint={2311.08355},
+      archivePrefix={arXiv},
+}
+```

beats/microsoft-deberta-v3-large.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c1498d38c9cff8fb278b0d2e7a40a7b01c90e78caf9edba137258a35987849e9
+size 1744651736

chords/flan-t5-large.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6d73492576022d13ac89e2c20c2e08c87a87f0edc164d5c883c3c7e32024a8e8
+size 3132793669

config.json

@@ -0,0 +1 @@
+{"text_encoder_name": "google/flan-t5-large", "scheduler_name": "stabilityai/stable-diffusion-2-1", "unet_model_name": null, "unet_model_config_path": "configs/music_diffusion_model_config.json", "snr_gamma": 5.0}

configs/main_config.json

@@ -0,0 +1 @@
+{"text_encoder_name": "google/flan-t5-large", "scheduler_name": "stabilityai/stable-diffusion-2-1", "unet_model_name": null, "unet_model_config_path": "configs/music_diffusion_model_config.json", "snr_gamma": 5.0}

configs/music_diffusion_model_config.json

@@ -0,0 +1,47 @@
+{
+  "_class_name": "UNet2DConditionModel",
+  "_diffusers_version": "0.10.0.dev0",
+  "act_fn": "silu",
+  "attention_head_dim": [
+    5,
+    10,
+    20,
+    20
+  ],
+  "block_out_channels": [
+    320,
+    640,
+    1280,
+    1280
+  ],
+  "center_input_sample": false,
+  "cross_attention_dim": 1024,
+  "down_block_types": [
+    "CrossAttnDownBlock2DMusic",
+    "CrossAttnDownBlock2DMusic",
+    "CrossAttnDownBlock2DMusic",
+    "DownBlock2D"
+  ],
+  "downsample_padding": 1,
+  "dual_cross_attention": false,
+  "flip_sin_to_cos": true,
+  "freq_shift": 0,
+  "in_channels": 8,
+  "layers_per_block": 2,
+  "mid_block_type": "UNetMidBlock2DCrossAttnMusic",
+  "mid_block_scale_factor": 1,
+  "norm_eps": 1e-05,
+  "norm_num_groups": 32,
+  "num_class_embeds": null,
+  "only_cross_attention": false,
+  "out_channels": 8,
+  "sample_size": [32, 2],
+  "up_block_types": [
+    "UpBlock2D",
+    "CrossAttnUpBlock2DMusic",
+    "CrossAttnUpBlock2DMusic",
+    "CrossAttnUpBlock2DMusic"
+  ],
+  "use_linear_projection": true,
+  "upcast_attention": true
+}

configs/stft_config.json

@@ -0,0 +1 @@
+{"filter_length": 1024, "hop_length": 160, "win_length": 1024, "n_mel_channels": 64, "sampling_rate": 16000, "mel_fmin": 0, "mel_fmax": 8000}

configs/vae_config.json

@@ -0,0 +1 @@
+{"image_key": "fbank", "subband": 1, "embed_dim": 8, "time_shuffle": 1, "ddconfig": {"double_z": true, "z_channels": 8, "resolution": 256, "downsample_time": false, "in_channels": 1, "out_ch": 1, "ch": 128, "ch_mult": [1, 2, 4], "num_res_blocks": 2, "attn_resolutions": [], "dropout": 0.0}, "scale_factor": 0.9227914214134216}

ldm/pytorch_model_ldm.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c7cad0ff3dd6b346898b12b2e3627e8262684646ff82f59d0a69891e42fcaa66
+size 7051656406

stft/pytorch_model_stft.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8674a4cc9755fafa48350bfa3412cf9b9a0d357d18289dbfd86f0fb34e1ca4db
+size 8537803

vae/pytorch_model_vae.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7d49e1881f38bd4f4fcaaf1c56686c02fb15f75e80dec5f773ae235b2cf1b61b
+size 442713669