Cloud storage

🤗 Datasets supports access to cloud storage providers through a S3 filesystem implementation: filesystems.S3FileSystem. You can save and load datasets from your Amazon S3 bucket in a Pythonic way. Take a look at the following table for other supported cloud storage providers:

Storage provider	Filesystem implementation
Amazon S3	s3fs
Google Cloud Storage	gcsfs
Azure Blob/DataLake	adlfs
Dropbox	dropboxdrivefs
Google Drive	gdrivefs

This guide will show you how to save and load datasets with s3fs to a S3 bucket, but other filesystem implementations can be used similarly. An example is shown also for Google Cloud Storage and Azure Blob Storage.

Amazon S3

Listing datasets

Install the S3 dependency with 🤗 Datasets:

>>> pip install datasets[s3]

List files from a public S3 bucket with s3.ls:

>>> import datasets
>>> s3 = datasets.filesystems.S3FileSystem(anon=True)  
>>> s3.ls('public-datasets/imdb/train')
['dataset_info.json.json','dataset.arrow','state.json']

Access a private S3 bucket by entering your aws_access_key_id and aws_secret_access_key:

>>> import datasets
>>> s3 = datasets.filesystems.S3FileSystem(key=aws_access_key_id, secret=aws_secret_access_key)  
>>> s3.ls('my-private-datasets/imdb/train')  
['dataset_info.json.json','dataset.arrow','state.json']

Saving datasets

After you have processed your dataset, you can save it to S3 with Dataset.save_to_disk():

>>> from datasets.filesystems import S3FileSystem

# create S3FileSystem instance
>>> s3 = S3FileSystem(anon=True)  

# saves encoded_dataset to your s3 bucket
>>> encoded_dataset.save_to_disk('s3://my-private-datasets/imdb/train', fs=s3)

Remember to include your aws_access_key_id and aws_secret_access_key whenever you are interacting with a private S3 bucket.

Save your dataset with botocore.session.Session and a custom AWS profile:

>>> import botocore
>>> from datasets.filesystems import S3FileSystem

# creates a botocore session with the provided AWS profile
>>> s3_session = botocore.session.Session(profile='my_profile_name')

# create S3FileSystem instance with s3_session
>>> s3 = S3FileSystem(session=s3_session)  

# saves encoded_dataset to your s3 bucket
>>> encoded_dataset.save_to_disk('s3://my-private-datasets/imdb/train',fs=s3)

Loading datasets

When you are ready to use your dataset again, reload it with Dataset.load_from_disk():

>>> from datasets import load_from_disk
>>> from datasets.filesystems import S3FileSystem

# create S3FileSystem without credentials
>>> s3 = S3FileSystem(anon=True)  

# load encoded_dataset to from s3 bucket
>>> dataset = load_from_disk('s3://a-public-datasets/imdb/train',fs=s3)  

>>> print(len(dataset))
>>> # 25000

Load with botocore.session.Session and custom AWS profile:

>>> import botocore
>>> from datasets.filesystems import S3FileSystem

# create S3FileSystem instance with aws_access_key_id and aws_secret_access_key
>>> s3_session = botocore.session.Session(profile='my_profile_name')

# create S3FileSystem instance with s3_session
>>> s3 = S3FileSystem(session=s3_session)

# load encoded_dataset to from s3 bucket
>>> dataset = load_from_disk('s3://my-private-datasets/imdb/train',fs=s3)  

>>> print(len(dataset))
>>> # 25000

Google Cloud Storage

Install the Google Cloud Storage implementation:

>>> conda install -c conda-forge gcsfs
# or install with pip
>>> pip install gcsfs

Save your dataset:

>>> import gcsfs

# create GCSFileSystem instance using default gcloud credentials with project
>>> gcs = gcsfs.GCSFileSystem(project='my-google-project')

# saves encoded_dataset to your gcs bucket
>>> encoded_dataset.save_to_disk('gcs://my-private-datasets/imdb/train', fs=gcs)

Load your dataset:

>>> import gcsfs
>>> from datasets import load_from_disk

# create GCSFileSystem instance using default gcloud credentials with project
>>> gcs = gcsfs.GCSFileSystem(project='my-google-project')

# loads encoded_dataset from your gcs bucket
>>> dataset = load_from_disk('gcs://my-private-datasets/imdb/train', fs=gcs)

Azure Blob Storage

Install the Azure Blob Storage implementation:

>>> conda install -c conda-forge adlfs
# or install with pip
>>> pip install adlfs

Save your dataset:

>>> import adlfs

# create AzureBlobFileSystem instance with account_name and account_key
>>> abfs = adlfs.AzureBlobFileSystem(account_name="XXXX", account_key="XXXX")

# saves encoded_dataset to your azure container
>>> encoded_dataset.save_to_disk('abfs://my-private-datasets/imdb/train', fs=abfs)

Load your dataset:

>>> import adlfs
>>> from datasets import load_from_disk

# create AzureBlobFileSystem instance with account_name and account_key
>>> abfs = adlfs.AzureBlobFileSystem(account_name="XXXX", account_key="XXXX")

# loads encoded_dataset from your azure container
>>> dataset = load_from_disk('abfs://my-private-datasets/imdb/train', fs=abfs)