nlp

Datasets and evaluation metrics for natural language processing

Compatible with NumPy, Pandas, PyTorch and TensorFlow

🤗nlp is a lightweight and extensible library to easily share and access datasets and evaluation metrics for Natural Language Processing (NLP).

nlp has many interesting features (beside easy sharing and accessing datasets/metrics):

Built-in interoperability with Numpy, Pandas, PyTorch and Tensorflow 2 Lightweight and fast with a transparent and pythonic API Strive on large datasets: nlp naturally frees the user from RAM memory limitation, all datasets are memory-mapped on drive by default. Smart caching: never wait for your data to process several times nlp currently provides access to ~100 NLP datasets and ~10 evaluation metrics and is designed to let the community easily add and share new datasets and evaluation metrics. You can browse the full set of datasets with the live nlp viewer.

nlp originated from a fork of the awesome TensorFlow Datasets and the HuggingFace team want to deeply thank the TensorFlow Datasets team for building this amazing library. More details on the differences between nlp and tfds can be found in the section Main differences between nlp and tfds.

Contents

The documentation is organized in five parts:

• GET STARTED contains a quick tour and the installation instructions.

• USING DATASETS contains general tutorials on how to use and contribute to the datasets in the library.

• USING METRICS contains general tutorials on how to use and contribute to the metrics in the library.

• ADVANCED GUIDES contains more advanced guides that are more specific to a part of the library.

• PACKAGE REFERENCE contains the documentation of each public class and function.

Get started

• Quick tour
  • Loading a dataset
  • Fine-tuning a deep-learning model
• Installation
  • Installation with pip
  • Installing from source
  • Caching datasets and metrics

Using datasets

• Loading a Dataset
  • From the HuggingFace Hub
  • From local files
  • From in-memory data
  • Using a custom dataset loading script
• What’s in the Dataset object
  • Features and columns
  • Metadata
  • Cache files and memory-usage
  • Getting rows, slices, batches and columns
  • Working with NumPy, pandas, PyTorch, TensorFlow
• Processing data in a Dataset
  • Selecting, sorting, shuffling, splitting rows
  • Processing data with map
  • Augmenting the dataset
  • Processing several splits at once
  • Controling the cache behavior
• Adding a FAISS or Elastic Search index to a Dataset
  • Adding a FAISS index
  • Adding an ElasticSearch index

Creating and sharing datasets

• Writing a dataset loading script
  • Adding dataset metadata
  • Downloading data files and organizing splits
  • Generating the samples in each split
  • Specifying several dataset configurations
  • Testing the dataset loading script
• Sharing your dataset
  • Sharing a “canonical” dataset
  • Sharing a “community provided” dataset
  • Adding tests and metadata to the dataset

Using metrics

• Loading a Metric
• Using a Metric

Advanced guides

• Dataset features
• Splits and slicing
  • Slicing API
• Beam Datasets
  • Intro
  • Already processed datasets are provided
  • How to run a Beam dataset processing pipeline
  • How to create your own Beam dataset

Package reference

• Loading methods
  • Datasets
  • Metrics
• Main classes
  • DatasetInfo
  • Dataset
  • DatasetDict
  • Features
  • MetricInfo
  • Metric
• Classes used during the dataset building process