HuggingFace Datasets

Datasets and evaluation metrics for natural language processing

Compatible with NumPy, Pandas, PyTorch and TensorFlow

🤗datasets is a lightweight and extensible library to easily share and access datasets and evaluation metrics for Natural Language Processing (NLP).

🤗Datasets has many interesting features (beside easy sharing and accessing datasets/metrics):

Built-in interoperability with Numpy, Pandas, PyTorch and Tensorflow 2 Lightweight and fast with a transparent and pythonic API Strive on large datasets: 🤗Datasets naturally frees the user from RAM memory limitation, all datasets are memory-mapped on drive by default. Smart caching: never wait for your data to process several times 🤗Datasets currently provides access to ~100 NLP datasets and ~10 evaluation metrics and is designed to let the community easily add and share new datasets and evaluation metrics. You can browse the full set of datasets with the live 🤗Datasets viewer.

🤗Datasets originated from a fork of the awesome TensorFlow Datasets and the HuggingFace team want to deeply thank the TensorFlow Datasets team for building this amazing library. More details on the differences between 🤗Datasets and tfds can be found in the section Main differences between 🤗Datasets and tfds.

Contents

The documentation is organized in five parts:

  • GET STARTED contains a quick tour and the installation instructions.

  • USING DATASETS contains general tutorials on how to use and contribute to the datasets in the library.

  • USING METRICS contains general tutorials on how to use and contribute to the metrics in the library.

  • ADVANCED GUIDES contains more advanced guides that are more specific to a part of the library.

  • PACKAGE REFERENCE contains the documentation of each public class and function.

Get started

Using datasets

Using metrics

Adding new datasets/metrics

Advanced guides

Package reference