Troubleshooting
This guide aims to provide you the tools and knowledge required to navigate some common issues. If the suggestions listed in this guide do not cover your such situation, please refer to the Asking for Help section to learn where to find help with your specific issue.
Issues when uploading datasets with push_to_hub
Authentication issues
If you are experiencing authentication issues when sharing a dataset on 🤗 Hub using Dataset.push_to_hub() and a Hugging Face access token:
- Make sure that the Hugging Face token you’re using to authenticate yourself is a token with write permission.
- On OSX, it may help to clean up all the huggingface.co passwords on your keychain access, as well as reconfigure
git config --global credential.helper osxkeychain
, before usinghuggingface-cli login
.
Alternatively, you can use SSH keys to authenticate yourself - read more in the 🤗 Hub documentation.
Lost connection on large dataset upload
When uploading large datasets to Hub, if the number of dataset shards is large, it can create too many commits for the Hub in a short period. This will result in a connection error. The connection error can also be caused by a HTTP 500 error returned by AWS S3 bucket that Hub uses internally. In either situation, you can re-run Dataset.push_to_hub() to proceed with the dataset upload. Hub will check the SHAs of already uploaded shards to avoid reuploading them. We are working on making upload process more robust to transient errors, so updating to the latest library version is always a good idea.
Too Many Requests
Uploading large datasets via push_to_hub()
can result in an error:
HfHubHTTPError: 429 Client Error: Too Many Requests for url: ...
You have exceeded our hourly quotas for action: commit. We invite you to retry later.
If you encounter this issue, you need to upgrade the datasets
library to the latest version (or at least 2.15.0
).
Issues when creating datasets from custom data
Loading images and audio from a folder
When creating a dataset from a folder, one of the most common issues is that the file structure does not follow the expected format, or there’s an issue with the metadata file.
Learn more about required folder structure in corresponding documentation pages:
Pickling issues
Pickling issues when using Dataset.from_generator
When creating a dataset, IterableDataset.from_generator() and Dataset.from_generator() expect a “picklable” generator function.
This is required to hash the function using pickle
to be able to cache the dataset on disk.
While generator functions are generally “picklable”, note that generator objects are not. So if you’re using a generator object,
you will encounter a TypeError
like this:
TypeError: cannot pickle 'generator' object
This error can also occur when using a generator function that uses a global object that is not “picklable”, such as a DB connection, for example. If that’s the case, you can initialize such object directly inside the generator function to avoid this error.
Pickling issues with Dataset.map
Pickling errors can also happen in the multiprocess Dataset.map() - objects are pickled to be passed to child processes.
If the objects used in the transformation are not picklable, it’s not possible to cache the result of map
, which leads to an error being raised.
Here are some ways to address this issue:
- A universal solution to pickle issues is to make sure the objects (or generator classes) are pickable manually by implementing
__getstate__
/__setstate__
/__reduce__
. - You can also provide your own unique hash in
map
with thenew_fingerprint
argument. - You can also disable caching by calling
datasets.disable_caching()
, however, this is undesirable - read more about importance of cache
Asking for help
If the above troubleshooting advice did not help you resolve your issue, reach out for help to the community and the team.
Forums
Ask for help on the Hugging Face forums - post your question in the 🤗Datasets category Make sure to write a descriptive post with relevant context about your setup and reproducible code to maximize the likelihood that your problem is solved!
Discord
Post a question on Discord, and let the team and the community help you.
Community Discussions on 🤗 Hub
If you are facing issues creating a custom dataset with a script on Hub, you can ask the Hugging Face team for help by opening a discussion in the Community tab of your dataset with this message:
# Dataset rewiew request for <Dataset name> ## Description <brief description of the dataset> ## Files to review - file1 - file2 - ... cc @lhoestq @polinaeterna @mariosasko @albertvillanova
GitHub Issues
Finally, if you suspect to have found a bug related to the library itself, create an Issue on the 🤗 Datasets GitHub repository. Include context regarding the bug: code snippet to reproduce, details about your environment and data, etc. to help us figure out what’s wrong and how we can fix it.