Title: Task casting for text classification & question answering
Author: SBrandeis (CONTRIBUTOR)
State: closed
Created: 2021-04-23T16:00:41Z

This PR implements task preparation for a given task, in the continuation of #2143 

Task taxonomy follows 🤗 Transformers's pipelines taxonomy: https://github.com/huggingface/transformers/tree/master/src/transformers/pipelines

Edit by @lewtun:

This PR implements support for the following tasks:

* `text-classification`
* `question-answering`

The intended usage is as follows:

```python
# Load a dataset with default column names / features
ds = load_dataset("dataset_name")
# Cast column names / features to schema. Casting is defined in the dataset's `DatasetInfo`
ds = ds.prepare_for_task(task="text-classification")
# Casting can also be realised during load
ds = load_dataset("dataset_name", task="text-classification")
# We can also combine shared tasks across dataset concatenation
ds1 = load_dataset("dataset_name_1", task="text-classification")
ds2 = load_dataset("dataset_name_2", task="text-classification")
# If the tasks have the same schema, so will `ds_concat`
ds_concat = concatenate_datasets([ds1, ds2])
```

Note that the current implementation assumes that `DatasetInfo.task_templates` has been pre-defined by the user / contributor when overriding the `MyDataset(GeneratorBasedBuilder)._info` function.

As pointed out by @SBrandeis, for evaluation we'll need a way to detect which datasets are already have a compatible schema so we don't have to edit hundreds of dataset scripts. One possibility is to check if the schema features are a subset of the dataset ones, e.g.

```python
squad = load_dataset("./datasets/squad", split="train")
qa = QuestionAnswering()
schema = Features({**qa.input_schema, **qa.label_schema})
assert all(item in squad.features.items() for item in schema.items())
```

Comments:
cc @abhi1thakur 

Looks really nice so far, thanks !
Maybe if a dataset doesn't have a template for a specific task we could try the default template of this task ?

hey @SBrandeis @lhoestq,

i now have a better idea about what you guys are trying to achieve with the task templates and have a few follow-up questions:

1. how did you envision using `DatasetInfo` for running evaluation? my understanding is that all `dataset_infos.json` files are stored in the `datasets` repo (unlike `transformers` where each model's weights etc are stored in a dedicated repo). 
this suggests the following workflow:

```
- git clone datasets
- load target dataset to evaluate
- load `dataset_infos.json` for target dataset
- run eval for each task template in `task_templates`
- store metrics as evaluation cards (similar to what is done in `autonlp`)
```
2. assuming the above workflow, i see that the current `TaskTemplate` attributes of `task`, `input_schema`, and `label_schema` still require some wrangling from `dataset_infos.json` to reproduce additional mappings like `label2id` that we'd need for e.g. text classification. an alternative would be to instantiate the task template class directly from the JSON with something like
```python
from datasets.tasks import TextClassification
from transformers import AutoModelForSequenceClassification, AutoConfig

tc = TextClassification.from_json("path/to/dataset_infos.json")
# load a model with the desired config
model_ckpt = ...
config = AutoConfig.from_pretrained(model_ckpt, label2id=tc.label2id, id2label=tc.id2label)
model = AutoModelForSequenceClassification.from_pretrained(model_ckpt, config=config)
# run eval ...
```
perhaps this is what @SBrandeis had in mind with the `TaskTemplate.from_dict` method?

3. i personally prefer using `task_templates` over `supervised_keys` because it encourages the contributor to think in terms of 1 or more tasks. my question here is do we currently use `supervised_keys` for anything important in the `datasets` library?

1. How do you envision using DatasetInfo for running evaluation?

The initial idea was to be able to do something like this:
```python
from datasets import load_dataset
dset = load_dataset("name", task="binary_classification")
# OR
dset = load_dataset("name")
dset = dset.prepare_for_task("binary_classification")
```

2. I don't think that's needed if we proceed as mentioned above

3. `supervised_keys` are mostly a legacy compatibility thing with TF datasets, not sure it's used for anything right now. I'll let @lhoestq give more details on that

[Edit 1] Typo

> The initial idea was to be able to do something like this:
> 
> ```python
> from datasets import load_dataset
> dset = load_dataset("name", task="binary_classification")
> # OR
> dset = load_dataset("name")
> dset = dset.prepare_for_task("binary_classification")
> ```

ah that's very elegant! just so i've completely understood, the result would be that the relevant column names of `dset` would be mapped to e.g. `text` and `label` and thus we'd have a uniform schema for the evaluation of all `binary_classification` tasks?

That's correct! Also, the features need to be appropriately casted
For a classification task for example, we would need to cast the datasets features to something like this:
```python
datasets.Features({
    "text": datasets.Value("string"),
    "label": datasets.ClassLabel(names=[...]),
})
```


3. We can ignore `supervised_keys` (it came from TFDS and we're not using it) and use `task_templates`

great, thanks a lot for your answers! now it's much clearer what i need to do next 😃 

hey @lhoestq @SBrandeis, 

i've made some small tweaks to @SBrandeis's code so that `Dataset.prepare_for_task` is called in `DatasetBuilder`. using the `emotion` dataset as a test case, the following now works:

 ```python
# DatasetDict with default columns
ds = load_dataset("./datasets/emotion/")
# DatasetDict({
#     train: Dataset({
#         features: ['tweet', 'emotion'],
#         num_rows: 16000
#     })
#     validation: Dataset({
#         features: ['tweet', 'emotion'],
#         num_rows: 2000
#     })
#     test: Dataset({
#         features: ['tweet', 'emotion'],
#         num_rows: 2000
#     })
# })

# DatasetDict with remapped columns
ds = load_dataset("./datasets/emotion/", task="text_classification")
DatasetDict({
#     train: Dataset({
#         features: ['text', 'label'],
#         num_rows: 16000
#     })
#     validation: Dataset({
#         features: ['text', 'label'],
#         num_rows: 2000
#     })
#     test: Dataset({
#         features: ['text', 'label'],
#         num_rows: 2000
#     })
# })

# Dataset with default columns
ds = load_dataset("./datasets/emotion/", split="train")
# Map/cast features
ds = ds.prepare_for_task("text_classification")
# Dataset({
#     features: ['text', 'label'],
#     num_rows: 16000
# })
```

i have a few follow-up questions / remarks:

1. i'm working under the assumption that contributors / users only provide a unique set of task types. in particular, the current implementation does not support something like:
```python
task_templates=[TextClassification(labels=class_names, text_column="tweet", label_column="emotion"), TextClassification(labels=class_names, text_column="some_other_column", label_column="some_other_column")]
```
since we use `TaskTemplate.task` and the filter for compatible templates in `Dataset.prepare_for_task`. should we support these scenarios? my hunch is that this is rare in practice, but please correct me if i'm wrong.

2. when we eventually run evaluation for `transformers` models, i expect we'll be using the `Trainer` for which we can pass the standard label names to `TrainingArguments.label_names`. if that's the case, it might be prudent to heed the warning from the [docs](https://huggingface.co/transformers/main_classes/trainer.html?highlight=trainer#trainer) and use `labels` instead of `label` in the schema:
> your model can accept multiple label arguments (use the label_names in your TrainingArguments to indicate their name to the Trainer) but none of them should be named "label".

3. i plan to forge ahead on the rest of the pipeline taxonomy. please let me know if you'd prefer smaller, self-contained pull requests (e.g. one per task)

hey @lhoestq @SBrandeis, i think this is ready for another review 😃 

in addition to a few comments / questions i've left in the pr, here's a few remarks:

1. after some experimentation, i decided against allowing the user to specify nested column names for question-answering. i couldn't find a simple solution with the current api and suspect that i'd have to touch many areas of `datasets` to "unflatten" columns in a generic fashion.
2. in the current implementation, the user can specify the outer column name for question-answering, but is expected to follow the inner schema for e.g. `answers.text` and `answers.answer_start`. we can decide later how much flexibility we want to give users
3. i added a few unit tests
4. as discussed, let's keep this pr focused on text classification / question answering and i'll add the other tasks in separate prs
5. i renamed the tasks e.g. `text_classification` -> `text-classification` for consistency with the `Trainer` model cards [here](https://github.com/huggingface/transformers/pull/11599#pullrequestreview-656371007).

i'm not sure why the benchmarks are getting cancelled - is this expected?

> i'm not sure why the benchmarks are getting cancelled - is this expected?

Hmm I don't know. It's certainly unrelated to this PR though. Maybe github has some issues

Something is happening with actions: https://www.githubstatus.com/

hey @lhoestq and @SBrandeis, i've: 

* extended the `prepare_for_task` API along the lines that @lhoestq suggested. i wasn't entirely sure what the `datasets` convention is for docstrings with mixed types, so please see if my proposal makes sense
* added a few new tests to check that we trigger the value errors on incorrect input

i think this is ready for another review :)

> Looks all good thank you :)
> 
> Can you also add `prepare_for_task` in the `main_classes.rst` file of the documentation ?

Done! I also remembered that I needed to do the same for `DatasetDict`, so included this as well :)