Create an audio dataset
You can share a dataset with your team or with anyone in the community by creating a dataset repository on the Hugging Face Hub:
from datasets import load_dataset
dataset = load_dataset("<username>/my_dataset")
There are several methods for creating and sharing an audio dataset:
Create an audio dataset from local files in python with Dataset.push_to_hub(). This is an easy way that requires only a few steps in python.
Create an audio dataset repository with the
AudioFolder
builder. This is a no-code solution for quickly creating an audio dataset with several thousand audio files.
You can control access to your dataset by requiring users to share their contact information first. Check out the Gated datasets guide for more information about how to enable this feature on the Hub.
Local files
You can load your own dataset using the paths to your audio files. Use the cast_column() function to take a column of audio file paths, and cast it to the Audio feature:
>>> audio_dataset = Dataset.from_dict({"audio": ["path/to/audio_1", "path/to/audio_2", ..., "path/to/audio_n"]}).cast_column("audio", Audio())
>>> audio_dataset[0]["audio"]
{'array': array([ 0. , 0.00024414, -0.00024414, ..., -0.00024414,
0. , 0. ], dtype=float32),
'path': 'path/to/audio_1',
'sampling_rate': 16000}
Then upload the dataset to the Hugging Face Hub using Dataset.push_to_hub():
audio_dataset.push_to_hub("<username>/my_dataset")
This will create a dataset repository containing your audio dataset:
my_dataset/
├── README.md
└── data/
└── train-00000-of-00001.parquet
AudioFolder
The AudioFolder
is a dataset builder designed to quickly load an audio dataset with several thousand audio files without requiring you to write any code.
Any additional information about your dataset - such as transcription, speaker accent, or speaker intent - is automatically loaded by AudioFolder
as long as you include this information in a metadata file (metadata.csv
/metadata.jsonl
).
💡 Take a look at the Split pattern hierarchy to learn more about how AudioFolder
creates dataset splits based on your dataset repository structure.
Create a dataset repository on the Hugging Face Hub and upload your dataset directory following the AudioFolder
structure:
my_dataset/
├── README.md
├── metadata.csv
└── data/
The data
folder can be any name you want.
It can be helpful to store your metadata as a jsonl
file if the data columns contain a more complex format (like a list of floats) to avoid parsing errors or reading complex values as strings.
The metadata file should include a file_name
column to link an audio file to it’s metadata:
file_name,transcription
data/first_audio_file.mp3,znowu się duch z ciałem zrośnie w młodocianej wstaniesz wiosnie i możesz skutkiem tych leków umierać wstawać wiek wieków dalej tam były przestrogi jak siekać głowę jak nogi
data/second_audio_file.mp3,już u źwierzyńca podwojów król zasiada przy nim książęta i panowie rada a gdzie wzniosły krążył ganek rycerze obok kochanek król skinął palcem zaczęto igrzysko
data/third_audio_file.mp3,pewnie kędyś w obłędzie ubite minęły szlaki zaczekajmy dzień jaki poślemy szukać wszędzie dziś jutro pewnie będzie posłali wszędzie sługi czekali dzień i drugi gdy nic nie doczekali z płaczem chcą jechać dali
Then you can store your dataset in a directory structure like this:
metadata.csv
data/first_audio_file.mp3
data/second_audio_file.mp3
data/third_audio_file.mp3
Users can now load your dataset and the associated metadata by specifying audiofolder
in load_dataset() and the dataset directory in data_dir
:
>>> from datasets import load_dataset
>>> dataset = load_dataset("audiofolder", data_dir="/path/to/data")
>>> dataset["train"][0]
{'audio':
{'path': '/path/to/extracted/audio/first_audio_file.mp3',
'array': array([ 0.00088501, 0.0012207 , 0.00131226, ..., -0.00045776, -0.00054932, -0.00054932], dtype=float32),
'sampling_rate': 16000},
'transcription': 'znowu się duch z ciałem zrośnie w młodocianej wstaniesz wiosnie i możesz skutkiem tych leków umierać wstawać wiek wieków dalej tam były przestrogi jak siekać głowę jak nogi'
}
You can also use audiofolder
to load datasets involving multiple splits. To do so, your dataset directory might have the following structure:
data/train/first_train_audio_file.mp3
data/train/second_train_audio_file.mp3
data/test/first_test_audio_file.mp3
data/test/second_test_audio_file.mp3
Note that if audio files are located not right next to a metadata file, file_name
column should be a full relative path to an audio file, not just its filename.
For audio datasets that don’t have any associated metadata, AudioFolder
automatically infers the class labels of the dataset based on the directory name. It might be useful for audio classification tasks. Your dataset directory might look like:
data/train/electronic/01.mp3
data/train/punk/01.mp3
data/test/electronic/09.mp3
data/test/punk/09.mp3
Load the dataset with AudioFolder
, and it will create a label
column from the directory name (language id):
>>> from datasets import load_dataset
>>> dataset = load_dataset("audiofolder", data_dir="/path/to/data")
>>> dataset["train"][0]
{'audio':
{'path': '/path/to/electronic/01.mp3',
'array': array([ 3.9714024e-07, 7.3031038e-07, 7.5640685e-07, ...,
-1.1963668e-01, -1.1681189e-01, -1.1244172e-01], dtype=float32),
'sampling_rate': 44100},
'label': 0 # "electronic"
}
>>> dataset["train"][-1]
{'audio':
{'path': '/path/to/punk/01.mp3',
'array': array([0.15237972, 0.13222949, 0.10627693, ..., 0.41940814, 0.37578005,
0.33717662], dtype=float32),
'sampling_rate': 44100},
'label': 1 # "punk"
}
If all audio files are contained in a single directory or if they are not on the same level of directory structure, label
column won’t be added automatically. If you need it, set drop_labels=False
explicitly.
Some audio datasets, like those found in Kaggle competitions, have separate metadata files for each split. Provided the metadata features are the same for each split, audiofolder
can be used to load all splits at once. If the metadata features differ across each split, you should load them with separate load_dataset()
calls.
(Legacy) Loading script
Write a dataset loading script to manually create a dataset. It defines a dataset’s splits and configurations, and handles downloading and generating the dataset examples. The script should have the same name as your dataset folder or repository:
my_dataset/
├── README.md
├── my_dataset.py
└── data/
The data
folder can be any name you want, it doesn’t have to be data
. This folder is optional, unless you’re hosting your dataset on the Hub.
This directory structure allows your dataset to be loaded in one line:
>>> from datasets import load_dataset
>>> dataset = load_dataset("path/to/my_dataset")
This guide will show you how to create a dataset loading script for audio datasets, which is a bit different from creating a loading script for text datasets.
Audio datasets are commonly stored in tar.gz
archives which requires a particular approach to support streaming mode. While streaming is not required, we highly encourage implementing streaming support in your audio dataset because users without a lot of disk space can use your dataset without downloading it. Learn more about streaming in the Stream guide!
Here is an example using TAR archives:
my_dataset/
├── README.md
├── my_dataset.py
└── data/
├── train.tar.gz
├── test.tar.gz
└── metadata.csv
In addition to learning how to create a streamable dataset, you’ll also learn how to:
- Create a dataset builder class.
- Create dataset configurations.
- Add dataset metadata.
- Download and define the dataset splits.
- Generate the dataset.
- Upload the dataset to the Hub.
The best way to learn is to open up an existing audio dataset loading script, like Vivos, and follow along!
This guide shows how to process audio data stored in TAR archives - the most frequent case for audio datasets. Check out minds14 dataset for an example of an audio script which uses ZIP archives.
To help you get started, we created a loading script template you can copy and use as a starting point!
Create a dataset builder class
GeneratorBasedBuilder is the base class for datasets generated from a dictionary generator. Within this class, there are three methods to help create your dataset:
_info
stores information about your dataset like its description, license, and features._split_generators
downloads the dataset and defines its splits._generate_examples
generates the dataset’s samples containing the audio data and other features specified ininfo
for each split.
Start by creating your dataset class as a subclass of GeneratorBasedBuilder and add the three methods. Don’t worry about filling in each of these methods yet, you’ll develop those over the next few sections:
class VivosDataset(datasets.GeneratorBasedBuilder):
"""VIVOS is a free Vietnamese speech corpus consisting of 15 hours of recording speech prepared for
Vietnamese Automatic Speech Recognition task."""
def _info(self):
def _split_generators(self, dl_manager):
def _generate_examples(self, prompts_path, path_to_clips, audio_files):
Multiple configurations
In some cases, a dataset may have more than one configuration. For example, LibriVox Indonesia dataset has several configurations corresponding to different languages.
To create different configurations, use the BuilderConfig class to create a subclass of your dataset. The only required parameter is the name
of the configuration, which must be passed to the configuration’s superclass __init__()
. Otherwise, you can specify any custom parameters you want in your configuration class.
class LibriVoxIndonesiaConfig(datasets.BuilderConfig):
"""BuilderConfig for LibriVoxIndonesia."""
def __init__(self, name, version, **kwargs):
self.language = kwargs.pop("language", None)
self.release_date = kwargs.pop("release_date", None)
self.num_clips = kwargs.pop("num_clips", None)
self.num_speakers = kwargs.pop("num_speakers", None)
self.validated_hr = kwargs.pop("validated_hr", None)
self.total_hr = kwargs.pop("total_hr", None)
self.size_bytes = kwargs.pop("size_bytes", None)
self.size_human = size_str(self.size_bytes)
description = (
f"LibriVox-Indonesia speech to text dataset in {self.language} released on {self.release_date}. "
f"The dataset comprises {self.validated_hr} hours of transcribed speech data"
)
super(LibriVoxIndonesiaConfig, self).__init__(
name=name,
version=datasets.Version(version),
description=description,
**kwargs,
)
Define your configurations in the BUILDER_CONFIGS
class variable inside GeneratorBasedBuilder. In this example, the author imports the languages from a separate release_stats.py
file from their repository, and then loops through each language to create a configuration:
class LibriVoxIndonesia(datasets.GeneratorBasedBuilder):
DEFAULT_CONFIG_NAME = "all"
BUILDER_CONFIGS = [
LibriVoxIndonesiaConfig(
name=lang,
version=STATS["version"],
language=LANGUAGES[lang],
release_date=STATS["date"],
num_clips=lang_stats["clips"],
num_speakers=lang_stats["users"],
total_hr=float(lang_stats["totalHrs"]) if lang_stats["totalHrs"] else None,
size_bytes=int(lang_stats["size"]) if lang_stats["size"] else None,
)
for lang, lang_stats in STATS["locales"].items()
]
Typically, users need to specify a configuration to load in load_dataset(), otherwise a ValueError
is raised. You can avoid this by setting a default dataset configuration to load in DEFAULT_CONFIG_NAME
.
Now if users want to load the Balinese (bal
) configuration, they can use the configuration name:
>>> from datasets import load_dataset
>>> dataset = load_dataset("indonesian-nlp/librivox-indonesia", "bal", split="train")
Add dataset metadata
Adding information about your dataset helps users to learn more about it. This information is stored in the DatasetInfo class which is returned by the info
method. Users can access this information by:
>>> from datasets import load_dataset_builder
>>> ds_builder = load_dataset_builder("vivos")
>>> ds_builder.info
There is a lot of information you can include about your dataset, but some important ones are:
description
provides a concise description of the dataset.features
specify the dataset column types. Since you’re creating an audio loading script, you’ll need to include the Audio feature and thesampling_rate
of the dataset.homepage
provides a link to the dataset homepage.license
specify the permissions for using a dataset as defined by the license type.citation
is a BibTeX citation of the dataset.
You’ll notice a lot of the dataset information is defined earlier in the loading script which can make it easier to read. There are also other ~Dataset.Features
you can input, so be sure to check out the full list and features guide for more details.
def _info(self):
return datasets.DatasetInfo(
description=_DESCRIPTION,
features=datasets.Features(
{
"speaker_id": datasets.Value("string"),
"path": datasets.Value("string"),
"audio": datasets.Audio(sampling_rate=16_000),
"sentence": datasets.Value("string"),
}
),
supervised_keys=None,
homepage=_HOMEPAGE,
license=_LICENSE,
citation=_CITATION,
)
Download and define the dataset splits
Now that you’ve added some information about your dataset, the next step is to download the dataset and define the splits.
Use the download() method to download metadata file at
_PROMPTS_URLS
and audio TAR archive at_DATA_URL
. This method returns the path to the local file/archive. In streaming mode, it doesn’t download the file(s) and just returns a URL to stream the data from. This method accepts:- a relative path to a file inside a Hub dataset repository (for example, in the
data/
folder) - a URL to a file hosted somewhere else
- a (nested) list or dictionary of file names or URLs
- a relative path to a file inside a Hub dataset repository (for example, in the
After you’ve downloaded the dataset, use the SplitGenerator to organize the audio files and sentence prompts in each split. Name each split with a standard name like:
Split.TRAIN
,Split.TEST
, andSPLIT.Validation
.In the
gen_kwargs
parameter, specify the file path to theprompts_path
andpath_to_clips
. Foraudio_files
, you’ll need to use iter_archive() to iterate over the audio files in the TAR archive. This enables streaming for your dataset. All of these file paths are passed onto the next step where you’ll actually generate the dataset.
def _split_generators(self, dl_manager):
"""Returns SplitGenerators."""
prompts_paths = dl_manager.download(_PROMPTS_URLS)
archive = dl_manager.download(_DATA_URL)
train_dir = "vivos/train"
test_dir = "vivos/test"
return [
datasets.SplitGenerator(
name=datasets.Split.TRAIN,
gen_kwargs={
"prompts_path": prompts_paths["train"],
"path_to_clips": train_dir + "/waves",
"audio_files": dl_manager.iter_archive(archive),
},
),
datasets.SplitGenerator(
name=datasets.Split.TEST,
gen_kwargs={
"prompts_path": prompts_paths["test"],
"path_to_clips": test_dir + "/waves",
"audio_files": dl_manager.iter_archive(archive),
},
),
]
This implementation does not extract downloaded archives. If you want to extract files after download, you need to additionally use extract(), see the (Advanced) Extract TAR archives section.
Generate the dataset
The last method in the GeneratorBasedBuilder class actually generates the samples in the dataset. It yields a dataset according to the structure specified in features
from the info
method. As you can see, generate_examples
accepts the prompts_path
, path_to_clips
, and audio_files
from the previous method as arguments.
Files inside TAR archives are accessed and yielded sequentially. This means you need to have the metadata associated with the audio files in the TAR file in hand first so you can yield it with its corresponding audio file.
examples = {}
with open(prompts_path, encoding="utf-8") as f:
for row in f:
data = row.strip().split(" ", 1)
speaker_id = data[0].split("_")[0]
audio_path = "/".join([path_to_clips, speaker_id, data[0] + ".wav"])
examples[audio_path] = {
"speaker_id": speaker_id,
"path": audio_path,
"sentence": data[1],
}
Finally, iterate over files in audio_files
and yield them along with their corresponding metadata. iter_archive() yields a tuple of (path
, f
) where path
is a relative path to a file inside TAR archive and f
is a file object itself.
inside_clips_dir = False
id_ = 0
for path, f in audio_files:
if path.startswith(path_to_clips):
inside_clips_dir = True
if path in examples:
audio = {"path": path, "bytes": f.read()}
yield id_, {**examples[path], "audio": audio}
id_ += 1
elif inside_clips_dir:
break
Put these two steps together, and the whole _generate_examples
method looks like:
def _generate_examples(self, prompts_path, path_to_clips, audio_files):
"""Yields examples as (key, example) tuples."""
examples = {}
with open(prompts_path, encoding="utf-8") as f:
for row in f:
data = row.strip().split(" ", 1)
speaker_id = data[0].split("_")[0]
audio_path = "/".join([path_to_clips, speaker_id, data[0] + ".wav"])
examples[audio_path] = {
"speaker_id": speaker_id,
"path": audio_path,
"sentence": data[1],
}
inside_clips_dir = False
id_ = 0
for path, f in audio_files:
if path.startswith(path_to_clips):
inside_clips_dir = True
if path in examples:
audio = {"path": path, "bytes": f.read()}
yield id_, {**examples[path], "audio": audio}
id_ += 1
elif inside_clips_dir:
break
Upload the dataset to the Hub
Once your script is ready, create a dataset card and upload it to the Hub.
Congratulations, you can now load your dataset from the Hub! 🥳
>>> from datasets import load_dataset
>>> load_dataset("<username>/my_dataset")
(Advanced) Extract TAR archives locally
In the example above downloaded archives are not extracted and therefore examples do not contain information about where they are stored locally. To explain how to do the extraction in a way that it also supports streaming, we will briefly go through the LibriVox Indonesia loading script.
Download and define the dataset splits
Use the download() method to download the audio data at
_AUDIO_URL
.To extract audio TAR archive locally, use the extract(). You can use this method only in non-streaming mode (when
dl_manager.is_streaming=False
). This returns a local path to the extracted archive directory:local_extracted_archive = dl_manager.extract(audio_path) if not dl_manager.is_streaming else None
Use the iter_archive() method to iterate over the archive at
audio_path
, just like in the Vivos example above. iter_archive() doesn’t provide any information about the full paths of files from the archive, even if it has been extracted. As a result, you need to pass thelocal_extracted_archive
path to the next step ingen_kwargs
, in order to preserve information about where the archive was extracted to. This is required to construct the correct paths to the local files when you generate the examples.
The reason you need to use a combination of download() and iter_archive() is because files in TAR archives can’t be accessed directly by their paths. Instead, you’ll need to iterate over the files within the archive! You can use download_and_extract() and extract() with TAR archives only in non-streaming mode, otherwise it would throw an error.
Use the download_and_extract() method to download the metadata file specified in
_METADATA_URL
. This method returns a path to a local file in non-streaming mode. In streaming mode, it doesn’t download file locally and returns the same URL.Now use the SplitGenerator to organize the audio files and metadata in each split. Name each split with a standard name like:
Split.TRAIN
,Split.TEST
, andSPLIT.Validation
.In the
gen_kwargs
parameter, specify the file paths tolocal_extracted_archive
,audio_files
,metadata_path
, andpath_to_clips
. Remember, foraudio_files
, you need to use iter_archive() to iterate over the audio files in the TAR archives. This enables streaming for your dataset! All of these file paths are passed onto the next step where the dataset samples are generated.
def _split_generators(self, dl_manager):
"""Returns SplitGenerators."""
audio_path = dl_manager.download(_AUDIO_URL)
local_extracted_archive = dl_manager.extract(audio_path) if not dl_manager.is_streaming else None
path_to_clips = "librivox-indonesia"
return [
datasets.SplitGenerator(
name=datasets.Split.TRAIN,
gen_kwargs={
"local_extracted_archive": local_extracted_archive,
"audio_files": dl_manager.iter_archive(audio_path),
"metadata_path": dl_manager.download_and_extract(_METADATA_URL + "/metadata_train.csv.gz"),
"path_to_clips": path_to_clips,
},
),
datasets.SplitGenerator(
name=datasets.Split.TEST,
gen_kwargs={
"local_extracted_archive": local_extracted_archive,
"audio_files": dl_manager.iter_archive(audio_path),
"metadata_path": dl_manager.download_and_extract(_METADATA_URL + "/metadata_test.csv.gz"),
"path_to_clips": path_to_clips,
},
),
]
Generate the dataset
Here _generate_examples
accepts local_extracted_archive
, audio_files
, metadata_path
, and path_to_clips
from the previous method as arguments.
TAR files are accessed and yielded sequentially. This means you need to have the metadata in
metadata_path
associated with the audio files in the TAR file in hand first so that you can yield it with its corresponding audio file further:with open(metadata_path, "r", encoding="utf-8") as f: reader = csv.DictReader(f) for row in reader: if self.config.name == "all" or self.config.name == row["language"]: row["path"] = os.path.join(path_to_clips, row["path"]) # if data is incomplete, fill with empty values for field in data_fields: if field not in row: row[field] = "" metadata[row["path"]] = row
Now you can yield the files in
audio_files
archive. When you use iter_archive(), it yielded a tuple of (path
,f
) wherepath
is a relative path to a file inside the archive, andf
is the file object itself. To get the full path to the locally extracted file, join the path of the directory (local_extracted_path
) where the archive is extracted to and the relative audio file path (path
):for path, f in audio_files: if path in metadata: result = dict(metadata[path]) # set the audio feature and the path to the extracted file path = os.path.join(local_extracted_archive, path) if local_extracted_archive else path result["audio"] = {"path": path, "bytes": f.read()} result["path"] = path yield id_, result id_ += 1
Put both of these steps together, and the whole _generate_examples
method should look like:
def _generate_examples(
self,
local_extracted_archive,
audio_files,
metadata_path,
path_to_clips,
):
"""Yields examples."""
data_fields = list(self._info().features.keys())
metadata = {}
with open(metadata_path, "r", encoding="utf-8") as f:
reader = csv.DictReader(f)
for row in reader:
if self.config.name == "all" or self.config.name == row["language"]:
row["path"] = os.path.join(path_to_clips, row["path"])
# if data is incomplete, fill with empty values
for field in data_fields:
if field not in row:
row[field] = ""
metadata[row["path"]] = row
id_ = 0
for path, f in audio_files:
if path in metadata:
result = dict(metadata[path])
# set the audio feature and the path to the extracted file
path = os.path.join(local_extracted_archive, path) if local_extracted_archive else path
result["audio"] = {"path": path, "bytes": f.read()}
result["path"] = path
yield id_, result
id_ += 1