license: cc-by-nc-sa-4.0
datasets:
- bjoernp/AudioCaps
language:
- en
TANGO: Text to Audio using iNstruction-Guided diffusiOn
TANGO is a latent diffusion model for text to audio generation. TANGO can generate realistic audios including human sounds, animal sounds, natural and artificial sounds and sound effects from textual prompts. We use the frozen instruction-tuned LLM Flan-T5 as the text encoder and train a UNet based diffusion model for audio generation. We outperform current state-of-the-art models for audio generation across both objective and subjective metrics. We release our model, training, inference code and pre-trained checkpoints for the research community.
Code
Our code is released here: https://github.com/declare-lab/tango
We uploaded several TANGO generated samples here: https://tango-web.github.io/
Please follow the instructions in the repository for installation, usage and experiments.
Quickstart Guide
Download the TANGO model and generate audio from a text prompt:
import IPython
import soundfile as sf
from tango import Tango
tango = Tango("declare-lab/tango")
prompt = "An audience cheering and clapping"
audio = tango.generate(prompt)
sf.write(f"{prompt}.wav", audio, samplerate=16000)
IPython.display.Audio(data=audio, rate=16000)
An audience cheering and clapping.webm
The model will be automatically downloaded and saved in cache. Subsequent runs will load the model directly from cache.
The generate function uses 100 steps by default to sample from the latent diffusion model. We recommend using 200 steps for generating better quality audios. This comes at the cost of increased run-time.
prompt = "Rolling thunder with lightning strikes"
audio = tango.generate(prompt, steps=200)
IPython.display.Audio(data=audio, rate=16000)
Rolling thunder with lightning strikes.webm
Use the generate_for_batch function to generate multiple audio samples for a batch of text prompts:
prompts = [
"A car engine revving",
"A dog barks and rustles with some clicking",
"Water flowing and trickling"
]
audios = tango.generate_for_batch(prompts, samples=2)
This will generate two samples for each of the three text prompts.
Limitations
TANGO is not always able to finely control its generations over textual control prompts as it is trained only on the small AudioCaps dataset. For example, the generations from TANGO for prompts Chopping tomatoes on a wooden table and Chopping potatoes on a metal table are very similar. Chopping vegetables on a table also produces similar audio samples. Training text-to-audio generation models on larger datasets is thus required for the model to learn the composition of textual concepts and varied text-audio mappings. In the future, we plan to improve TANGO by training it on larger datasets and enhancing its compositional and controllable generation ability.