Create a dataset card

Each dataset should have a dataset card to promote responsible usage and inform users of any potential biases within the dataset. This idea was inspired by the Model Cards proposed by Mitchell, 2018. Dataset cards help users understand a dataset’s contents, the context for using the dataset, how it was created, and any other considerations a user should be aware of.

This guide shows you how to create a dataset card.

1. Create a new dataset card by copying this template to a README.md file in your repository.

2. Generate structured tags to help users discover your dataset on the Hub. Create the tags with the online Datasets Tagging app.

3. Select the appropriate tags for your dataset from the dropdown menus.

4. Copy the YAML tags under Finalized tag set and paste the tags at the top of your README.md file.

5. Fill out the dataset card sections to the best of your ability. Take a look at the Dataset Card Creation Guide for more detailed information about what to include in each section of the card. For fields you are unable to complete, you can write [More Information Needed].

6. Once you’re done filling out the dataset card, commit the changes to the README.md file and you should see the completed dataset card on your repository.

Feel free to take a look at these dataset card examples to help you get started:

• SNLI
• CNN / DailyMail
• Allociné

You can also check out the (similar) documentation about dataset cards on the Hub side.

More YAML tags

You can use the dataset_info YAML fields to define additional metadata for the dataset. Here is an example for SQuAD:

pretty_name: SQuAD
language:
- en
...
dataset_info:
  features:
  - name: id
    dtype: string
  - name: title
    dtype: string
  - name: context
    dtype: string
  - name: question
    dtype: string
  - name: answers
    sequence:
    - name: text
      dtype: string
    - name: answer_start
      dtype: int32
  splits:
  - name: train
    num_bytes: 79346360
    num_examples: 87599
  - name: validation
    num_bytes: 10473040
    num_examples: 10570
  download_size: 35142551
  dataset_size: 89819400

These metadata used to be included in the dataset_infos.json file, which is now deprecated.

Feature types

Using the features field you can explicitly define the feature types of your dataset. This is especially useful when type inference is not obvious. For example if there’s only one non-empty example in a 1TB dataset, the type inference is not able to infer the type of each column without going through the full dataset. In this case, specifying the features field makes type inference much easier.

Split sizes

Specifying the split sizes with num_examples enables TQDM bars (otherwise it doesn’t know how many examples are left). It also enables integrity verifications: if the dataset doesn’t have the right number of num_examples, an error is returned.

Additionally you can add num_bytes to specify how big each split is.

Dataset size

When load_dataset() is called, it first downloads the dataset raw data files, and then it prepares the dataset in Arrow format.

You can specify how many bytes are required to download the raw data files with dataset_size, and use dataset_size for the size of the dataset in Arrow format.

Multiple configurations

Certain datasets like glue have several configurations (cola, sst2, etc.) that can be loaded using load_dataset("glue", "cola") for example.

Each configuration can have different features, splits and sizes. You can specify those fields per configuration using a YAML list:

dataset_info:
- config_name: cola
  features:
    ...
  splits:
    ...
  download_size: ...
  dataset_size: ...
- config_name: sst2
  features:
    ...
  splits:
    ...
  download_size: ...
  dataset_size: ...