Title: Fix XTREME PAN-X loading
Author: lvwerra (MEMBER)
State: closed
Created: 2020-07-25T14:44:57Z

Hi 🤗 
In response to the discussion in #425 @lewtun and I made some fixes to the repo. In the original XTREME implementation the PAN-X dataset for named entity recognition loaded each word/tag pair as a single row and the sentence relation was lost. With the fix each row contains the list of all words in a single sentence and their NER tags. This is also in agreement with the [NER example](https://github.com/huggingface/transformers/tree/master/examples/token-classification) in the transformers repo.

With the fix the output of the dataset should look as follows:
```python
>>> dataset = load_dataset("xtreme", "PAN-X.en", data_dir='./data')
>>> dataset['train'][0]
{'words': ['R.H.', 'Saunders', '(', 'St.', 'Lawrence', 'River', ')', '(', '968', 'MW', ')'],
 'ner_tags': ['B-ORG', 'I-ORG', 'O', 'B-ORG', 'I-ORG', 'I-ORG', 'O', 'O', 'O', 'O', 'O'],
 'langs': ['en', 'en', 'en', 'en', 'en', 'en', 'en', 'en', 'en', 'en', 'en']}
```

Comments:
There is an interesting design question here (cc @lhoestq).

I guess the labels form a closed set so we could also use a [nlp.ClassLabel](https://huggingface.co/nlp/package_reference/main_classes.html#nlp.ClassLabel) instead of a string. The differences will be mainly that:
- the labels are stored as integers and thus ready for training a model
- the string to int conversion methods are handled by the `nlp.ClassLabel` feature (see the [doc](https://huggingface.co/nlp/package_reference/main_classes.html#nlp.ClassLabel)  and [here](https://huggingface.co/nlp/features.html)  and [here](https://huggingface.co/nlp/quicktour.html#fine-tuning-a-deep-learning-model)).

In my opinion, storing the labels as integers instead of strings makes it:
- slightly less readable when accessing a dataset example (e.g. with `dataset[0]`)
- force you with a specific mapping from string to integers
- more clear that there is a fixed and predefined list of labels
- easier to list all the labels (directly visible in the features).

=> overall I'm pretty neutral about using one or the other option (`nlp.string` or `nlp.ClassLabel`).

Note that we can now rather easily convert from one to the other with the map function and something like:
```python
dataset = dataset.map(lambda x: x, features=nlp.Features({'labels': nlp.ClassLabel(MY_LABELS_NAMES)}))
dataset = dataset.map(lambda x: {'labels': dataset.features['labels'].int2str(x['labels'])}, features=nlp.Features({'labels': nlp.Value('string')}))
```
^^ this could probably be made even simpler (in particular for the second case)

I see. This is an interesting question.
Maybe as the dataset doesn't provide the mapping we shouldn't force an arbitrary one, and keep them as strings ?
Moreover for NER the labels are often different from a dataset to the other so it's probably good to keep strings (there is no conventional mapping).
Also as the column is called "ner_tags" (or "langs"), you can already assume that there is a fixed and predefined list of labels.

Yes sounds good to me.
This make me wonder if we don’t want to have a default identity function in `map` so this method could also be used to easily cast features. What do you think?

Yes sounds good. I also noticed that people use map with identity to write a dataset into a specified cache file.