Title: Cannot load udpos subsets from xtreme dataset using load_dataset()
Author: adzcodez (NONE)
State: closed
Created: 2021-03-16T09:32:13Z
Labels: good first issue

Hello, 

I am trying to load the udpos English subset from xtreme dataset, but this faces an error during loading. I am using datasets v1.4.1, pip install. I have tried with other udpos languages which also fail, though loading a different subset altogether (such as XNLI) has no issue. I have also tried on Colab and faced the same error. 

Reprex is: 

`from datasets import load_dataset `
`dataset = load_dataset('xtreme', 'udpos.English')`

The error is: 
`KeyError: '_'`

The full traceback is: 
KeyError                                  Traceback (most recent call last)
<ipython-input-5-7181359ea09d> in <module>
      1 from datasets import load_dataset
----> 2 dataset = load_dataset('xtreme', 'udpos.English')

~\Anaconda3\envs\mlenv\lib\site-packages\datasets\load.py in load_dataset(path, name, data_dir, data_files, split, cache_dir, features, download_config, download_mode, ignore_verifications, keep_in_memory, save_infos, script_version, use_auth_token, **config_kwargs)
    738 
    739     # Download and prepare data
--> 740     builder_instance.download_and_prepare(
    741         download_config=download_config,
    742         download_mode=download_mode,

~\Anaconda3\envs\mlenv\lib\site-packages\datasets\builder.py in download_and_prepare(self, download_config, download_mode, ignore_verifications, try_from_hf_gcs, dl_manager, base_path, use_auth_token, **download_and_prepare_kwargs)
    576                             logger.warning("HF google storage unreachable. Downloading and preparing it from source")
    577                     if not downloaded_from_gcs:
--> 578                         self._download_and_prepare(
    579                             dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
    580                         )

~\Anaconda3\envs\mlenv\lib\site-packages\datasets\builder.py in _download_and_prepare(self, dl_manager, verify_infos, **prepare_split_kwargs)
    654             try:
    655                 # Prepare split will record examples associated to the split
--> 656                 self._prepare_split(split_generator, **prepare_split_kwargs)
    657             except OSError as e:
    658                 raise OSError(

~\Anaconda3\envs\mlenv\lib\site-packages\datasets\builder.py in _prepare_split(self, split_generator)
    977                 generator, unit=" examples", total=split_info.num_examples, leave=False, disable=not_verbose
    978             ):
--> 979                 example = self.info.features.encode_example(record)
    980                 writer.write(example)
    981         finally:

~\Anaconda3\envs\mlenv\lib\site-packages\datasets\features.py in encode_example(self, example)
    946     def encode_example(self, example):
    947         example = cast_to_python_objects(example)
--> 948         return encode_nested_example(self, example)
    949 
    950     def encode_batch(self, batch):

~\Anaconda3\envs\mlenv\lib\site-packages\datasets\features.py in encode_nested_example(schema, obj)
    840     # Nested structures: we allow dict, list/tuples, sequences
    841     if isinstance(schema, dict):
--> 842         return {
    843             k: encode_nested_example(sub_schema, sub_obj) for k, (sub_schema, sub_obj) in utils.zip_dict(schema, obj)
    844         }

~\Anaconda3\envs\mlenv\lib\site-packages\datasets\features.py in <dictcomp>(.0)
    841     if isinstance(schema, dict):
    842         return {
--> 843             k: encode_nested_example(sub_schema, sub_obj) for k, (sub_schema, sub_obj) in utils.zip_dict(schema, obj)
    844         }
    845     elif isinstance(schema, (list, tuple)):

~\Anaconda3\envs\mlenv\lib\site-packages\datasets\features.py in encode_nested_example(schema, obj)
    868     # ClassLabel will convert from string to int, TranslationVariableLanguages does some checks
    869     elif isinstance(schema, (ClassLabel, TranslationVariableLanguages, Value, _ArrayXD)):
--> 870         return schema.encode_example(obj)
    871     # Other object should be directly convertible to a native Arrow type (like Translation and Translation)
    872     return obj

~\Anaconda3\envs\mlenv\lib\site-packages\datasets\features.py in encode_example(self, example_data)
    647         # If a string is given, convert to associated integer
    648         if isinstance(example_data, str):
--> 649             example_data = self.str2int(example_data)
    650 
    651         # Allowing -1 to mean no label.

~\Anaconda3\envs\mlenv\lib\site-packages\datasets\features.py in str2int(self, values)
    605                 if value not in self._str2int:
    606                     value = value.strip()
--> 607                 output.append(self._str2int[str(value)])
    608             else:
    609                 # No names provided, try to integerize

KeyError: '_'

Comments:
@lhoestq Adding "_" to the class labels in the dataset script will fix the issue.

The bigger issue IMO is that the data files are in conll format, but the examples are tokens, not sentences.

Hi ! Thanks for reporting @adzcodez 


> @lhoestq Adding "_" to the class labels in the dataset script will fix the issue.
> 
> The bigger issue IMO is that the data files are in conll format, but the examples are tokens, not sentences.

You're right: "_" should be added to the list of labels, and the examples must be sequences of tokens, not singles tokens.


@lhoestq Can you please label this issue with the "good first issue" label? I'm not sure I'll find time to fix this.

To resolve it, the user should:
1. add `"_"` to the list of labels
2. transform the udpos subset to the conll format (I think the preprocessing logic can be borrowed from [the original repo](https://github.com/google-research/xtreme/blob/58a76a0d02458c4b3b6a742d3fd4ffaca80ff0de/utils_preprocess.py#L187-L204))
3. update the dummy data
4. update the dataset info
5. [optional] add info about the data fields structure of the udpos subset to the dataset readme

I tried fixing this issue, but its working fine in the dev version : "1.6.2.dev0"

I think somebody already fixed it. 

Hi,

after #2326, the lines with pos tags equal to `"_"` are filtered out when generating the dataset, so this fixes the KeyError described above. However, the udpos subset should be in the conll format i.e. it should yield sequences of tokens and not single tokens, so it would be great to see this fixed (feel free to borrow the logic from [here](https://github.com/google-research/xtreme/blob/58a76a0d02458c4b3b6a742d3fd4ffaca80ff0de/utils_preprocess.py#L187-L204) if you decide to work on this). 

Closed by #2466.