Riffusion-Currulao (WIP) is a fine-tuned version of the Riffusion model, which is based on the Stable Diffusion 1.5 framework. This model is specifically designed to generate spectrograms for Currulao music, a traditional Afro-Colombian genre characterized by the use of the Marimba de Chonta, a unique wooden percussion instrument.
Currulao is an essential part of the Afro-Colombian cultural heritage and represents the rich history, traditions, and artistic expressions of the Afro-Colombian community. By leveraging the power of the Riffusion model and the additional fine-tuning through Keras using the (DreamBooth)[https://dreambooth.github.io/] technique, Riffusion-Currulao is capable of generating spectrograms that capture the essence of Currulao music.
The model was further enhanced using DreamBooth, which allowed the integration of the concepts of Currulao and Marimba de Chonta, ensuring that the generated spectrograms somehow represent the musical characteristics of this genre.
Riffusion-Currulao can be used for various applications, including music synthesis, audio analysis, and research on Afro-Colombian music and culture. It can also serve as a creative tool for artists and musicians who wish to explore Currulao music and incorporate its elements into their compositions.
The model is designed to be accessible and user-friendly, making it suitable for both professionals and enthusiasts interested in generating spectrograms of Currulao music for their projects or studies.
Prompt: A $currulao song, lo-fi, nostalgic.
You need to load the Riffusion weights to use this model. Example for Colab:
!git lfs install !git clone https://huggingface.co/juancopi81/dreambooth_riffusion_model_currulao_v1 !pip install git+https://github.com/costiash/stable-diffusion-tensorflow --upgrade --quiet !pip install tensorflow tensorflow_addons ftfy --upgrade --quiet !pip install pytorch-lightning !pip install pydub !pip install huggingface-hub !pip install -q -U keras_cv !pip install -q -U tensorflow import tensorflow as tf from keras_cv.models.stable_diffusion.clip_tokenizer import SimpleTokenizer from keras_cv.models.stable_diffusion.diffusion_model import DiffusionModel from keras_cv.models.stable_diffusion.stable_diffusion import StableDiffusion as StableDiffusionKeras from keras_cv.models.stable_diffusion.image_encoder import ImageEncoder from keras_cv.models.stable_diffusion.noise_scheduler import NoiseScheduler from keras_cv.models.stable_diffusion.text_encoder import TextEncoder import keras from stable_diffusion_tf.stable_diffusion import StableDiffusion as StableDiffusionPy resolution=512 sd_dreambooth_model_1=StableDiffusionPy(resolution, resolution, download_weights=False, jit_compile=True) diffusion_model_pytorch_weights = keras.utils.get_file( origin="https://huggingface.co/riffusion/riffusion-model-v1/resolve/main/riffusion-model-v1.ckpt", file_hash="99a6eb51c18e16a6121180f3daa69344e571618b195533f67ae94be4eb135a57", ) sd_dreambooth_model_1.load_weights_from_pytorch_ckpt(diffusion_model_pytorch_weights) sd_dreambooth_model_1.diffusion_model.load_weights("/content/dreambooth_riffusion_model_currulao_v1") img = sd_dreambooth_model_1.generate( "A $currulao song, EDM", num_steps=100, unconditional_guidance_scale=12, temperature=1.2, batch_size=1, ) pil_img = Image.fromarray(img)
Riffusion-Currulao (work in progress) will be intended for:
- Music synthesis: Generate new Currulao music pieces or augment existing compositions with Marimba de Chonta sounds.
- Audio analysis: Study the characteristics of Currulao music and its elements, such as rhythm, melody, and timbre, through the generated spectrograms.
- Research: Explore the Afro-Colombian cultural heritage and its musical expressions by analyzing the generated Currulao music.
- Creative inspiration: Use the model as a tool to inspire new musical ideas and compositions that incorporate the unique sounds and rhythms of Currulao music.
Always ensure that the use of generated spectrograms aligns with the intended purpose and respects the cultural significance of the Currulao music and Afro-Colombian heritage.
This model was fine-tuned using the DreamBooth technique. The spectrograms for fine-tuning the model were generated from the song: "MUSICA PARA RELAJAR DEL PACÍFICO". The spectrograms were generated using the "Riffusion Manipulation Tools" repo.
- Downloads last month
Unable to determine this model’s pipeline type. Check the docs .