Spaces:
Running
Running
| # Audiocraft | |
|  | |
|  | |
|  | |
| Audiocraft is a PyTorch library for deep learning research on audio generation. At the moment, it contains the code for MusicGen, a state-of-the-art controllable text-to-music model. | |
| ## MusicGen | |
| Audiocraft provides the code and models for MusicGen, [a simple and controllable model for music generation][arxiv]. MusicGen is a single stage auto-regressive | |
| Transformer model trained over a 32kHz <a href="https://github.com/facebookresearch/encodec">EnCodec tokenizer</a> with 4 codebooks sampled at 50 Hz. Unlike existing methods like [MusicLM](https://arxiv.org/abs/2301.11325), MusicGen doesn't require a self-supervised semantic representation, and it generates | |
| all 4 codebooks in one pass. By introducing a small delay between the codebooks, we show we can predict | |
| them in parallel, thus having only 50 auto-regressive steps per second of audio. | |
| Check out our [sample page][musicgen_samples] or test the available demo! | |
| <a target="_blank" href="https://colab.research.google.com/drive/1-Xe9NCdIs2sCUbiSmwHXozK6AAhMm7_i?usp=sharing"> | |
| <img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/> | |
| </a> | |
| <a target="_blank" href="https://huggingface.co/spaces/facebook/MusicGen"> | |
| <img src="https://huggingface.co/datasets/huggingface/badges/raw/main/open-in-hf-spaces-sm.svg" alt="Open in HugginFace"/> | |
| </a> | |
| <br> | |
| We use 20K hours of licensed music to train MusicGen. Specifically, we rely on an internal dataset of 10K high-quality music tracks, and on the ShutterStock and Pond5 music data. | |
| ## Installation | |
| Audiocraft requires Python 3.9, PyTorch 2.0.0, and a GPU with at least 16 GB of memory (for the medium-sized model). To install Audiocraft, you can run the following: | |
| ```shell | |
| # Best to make sure you have torch installed first, in particular before installing xformers. | |
| # Don't run this if you already have PyTorch installed. | |
| pip install 'torch>=2.0' | |
| # Then proceed to one of the following | |
| pip install -U audiocraft # stable release | |
| pip install -U git+https://git@github.com/facebookresearch/audiocraft#egg=audiocraft # bleeding edge | |
| pip install -e . # or if you cloned the repo locally | |
| ``` | |
| ## Usage | |
| We offer a number of way to interact with MusicGen: | |
| 1. You can play with MusicGen by running the jupyter notebook at [`demo.ipynb`](./demo.ipynb) locally, or use the provided [colab notebook](https://colab.research.google.com/drive/1fxGqfg96RBUvGxZ1XXN07s3DthrKUl4-?usp=sharing). | |
| 2. You can use the gradio demo locally by running `python app.py`. | |
| 3. A demo is also available on the [`facebook/MusicGen` HuggingFace Space](https://huggingface.co/spaces/facebook/MusicGen) (huge thanks to all the HF team for their support). | |
| 4. Finally, you can run the [Gradio demo with a Colab GPU](https://colab.research.google.com/drive/1-Xe9NCdIs2sCUbiSmwHXozK6AAhMm7_i?usp=sharing), | |
| as adapted from [@camenduru Colab](https://github.com/camenduru/MusicGen-colab). | |
| ### More info about Top-k, Top-p, Temperature and Classifier Free Guidance from ChatGPT | |
| Top-k: Top-k is a parameter used in text generation models, including music generation models. It determines the number of most likely next tokens to consider at each step of the generation process. The model ranks all possible tokens based on their predicted probabilities, and then selects the top-k tokens from the ranked list. The model then samples from this reduced set of tokens to determine the next token in the generated sequence. A smaller value of k results in a more focused and deterministic output, while a larger value of k allows for more diversity in the generated music. | |
| Top-p (or nucleus sampling): Top-p, also known as nucleus sampling or probabilistic sampling, is another method used for token selection during text generation. Instead of specifying a fixed number like top-k, top-p considers the cumulative probability distribution of the ranked tokens. It selects the smallest possible set of tokens whose cumulative probability exceeds a certain threshold (usually denoted as p). The model then samples from this set to choose the next token. This approach ensures that the generated output maintains a balance between diversity and coherence, as it allows for a varying number of tokens to be considered based on their probabilities. | |
| Temperature: Temperature is a parameter that controls the randomness of the generated output. It is applied during the sampling process, where a higher temperature value results in more random and diverse outputs, while a lower temperature value leads to more deterministic and focused outputs. In the context of music generation, a higher temperature can introduce more variability and creativity into the generated music, but it may also lead to less coherent or structured compositions. On the other hand, a lower temperature can produce more repetitive and predictable music. | |
| Classifier-Free Guidance: Classifier-Free Guidance refers to a technique used in some music generation models where a separate classifier network is trained to provide guidance or control over the generated music. This classifier is trained on labeled data to recognize specific musical characteristics or styles. During the generation process, the output of the generator model is evaluated by the classifier, and the generator is encouraged to produce music that aligns with the desired characteristics or style. This approach allows for more fine-grained control over the generated music, enabling users to specify certain attributes they want the model to capture. | |
| These parameters, such as top-k, top-p, temperature, and classifier-free guidance, provide different ways to influence the output of a music generation model and strike a balance between creativity, diversity, coherence, and control. The specific values for these parameters can be tuned based on the desired outcome and user preferences. | |
| ## API | |
| We provide a simple API and 4 pre-trained models. The pre trained models are: | |
| - `small`: 300M model, text to music only - [🤗 Hub](https://huggingface.co/facebook/musicgen-small) | |
| - `medium`: 1.5B model, text to music only - [🤗 Hub](https://huggingface.co/facebook/musicgen-medium) | |
| - `melody`: 1.5B model, text to music and text+melody to music - [🤗 Hub](https://huggingface.co/facebook/musicgen-melody) | |
| - `large`: 3.3B model, text to music only - [🤗 Hub](https://huggingface.co/facebook/musicgen-large) | |
| We observe the best trade-off between quality and compute with the `medium` or `melody` model. | |
| In order to use MusicGen locally **you must have a GPU**. We recommend 16GB of memory, but smaller | |
| GPUs will be able to generate short sequences, or longer sequences with the `small` model. | |
| **Note**: Please make sure to have [ffmpeg](https://ffmpeg.org/download.html) installed when using newer version of `torchaudio`. | |
| You can install it with: | |
| ``` | |
| apt-get install ffmpeg | |
| ``` | |
| See after a quick example for using the API. | |
| ```python | |
| import torchaudio | |
| from audiocraft.models import MusicGen | |
| from audiocraft.data.audio import audio_write | |
| model = MusicGen.get_pretrained('melody') | |
| model.set_generation_params(duration=8) # generate 8 seconds. | |
| wav = model.generate_unconditional(4) # generates 4 unconditional audio samples | |
| descriptions = ['happy rock', 'energetic EDM', 'sad jazz'] | |
| wav = model.generate(descriptions) # generates 3 samples. | |
| melody, sr = torchaudio.load('./assets/bach.mp3') | |
| # generates using the melody from the given audio and the provided descriptions. | |
| wav = model.generate_with_chroma(descriptions, melody[None].expand(3, -1, -1), sr) | |
| for idx, one_wav in enumerate(wav): | |
| # Will save under {idx}.wav, with loudness normalization at -14 db LUFS. | |
| audio_write(f'{idx}', one_wav.cpu(), model.sample_rate, strategy="loudness", loudness_compressor=True) | |
| ``` | |
| ## Model Card | |
| See [the model card page](./MODEL_CARD.md). | |
| ## FAQ | |
| #### Will the training code be released? | |
| Yes. We will soon release the training code for MusicGen and EnCodec. | |
| #### I need help on Windows | |
| @FurkanGozukara made a complete tutorial for [Audiocraft/MusicGen on Windows](https://youtu.be/v-YpvPkhdO4) | |
| ## Citation | |
| ``` | |
| @article{copet2023simple, | |
| title={Simple and Controllable Music Generation}, | |
| author={Jade Copet and Felix Kreuk and Itai Gat and Tal Remez and David Kant and Gabriel Synnaeve and Yossi Adi and Alexandre Défossez}, | |
| year={2023}, | |
| journal={arXiv preprint arXiv:2306.05284}, | |
| } | |
| ``` | |
| ## License | |
| * The code in this repository is released under the MIT license as found in the [LICENSE file](LICENSE). | |
| * The weights in this repository are released under the CC-BY-NC 4.0 license as found in the [LICENSE_weights file](LICENSE_weights). | |
| [arxiv]: https://arxiv.org/abs/2306.05284 | |
| [musicgen_samples]: https://ai.honu.io/papers/musicgen/ | |