Title: Adding the Linguistic Code-switching Evaluation (LinCE) benchmark
Author: gaguilar (CONTRIBUTOR)
State: closed
Created: 2020-07-11T22:35:20Z

Hi,

First of all, this library is really cool! Thanks for putting all of this together! 

This PR contains the [Linguistic Code-switching Evaluation (LinCE) benchmark](https://ritual.uh.edu/lince). As described in the official website (FAQ):

> 1. Why do we need LinCE?
>LinCE brings 10 code-switching datasets together for 4 tasks and 4 language pairs with 5 leaderboards in a single evaluation platform. We examined each dataset and fixed major issues on the partitions (or even define official partitions) with a comprehensive stratification method (see our paper for more details).
>Besides, we believe that online benchmarks like LinCE bring steady research progress and allow to compare state-of-the-art models at the pace of the progress in NLP. We expect to benefit greatly the code-switching community with this benchmark.


The data comes from social media and here's the summary table of tasks per language pair:

| Language Pairs                         | LID | POS | NER | SA |
|----------------------------------------|-----|-----|-----|----|
| Spanish-English                        | ✅  | ✅  | ✅  | ✅ |
| Hindi-English                          | ✅  | ✅  | ✅  |    |
| Modern Standard Arabic-Egyptian Arabic | ✅  |     | ✅  |    |
| Nepali-English                         | ✅  |     |     |    |

The tasks are as follows:
* LID: token-level language identification
* POS: part-of-speech tagging
* NER: named entity recognition
* SA: sentiment analysis

With the exception of MSA-EA, the rest of the datasets contain token-level LID labels.

## Usage

For Spanish-English LID, we can load the data as follows:
```
import nlp

data = nlp.load_dataset('./datasets/lince/lince.py', 'lid_spaeng')

for split in data:
    print(data[split])
```

Here's the output:
```
Dataset(schema: {'idx': 'int32', 'tokens': 'list<item: string>', 'lid': 'list<item: string>'}, num_rows: 21030)
Dataset(schema: {'idx': 'int32', 'tokens': 'list<item: string>', 'lid': 'list<item: string>'}, num_rows: 3332)
Dataset(schema: {'idx': 'int32', 'tokens': 'list<item: string>', 'lid': 'list<item: string>'}, num_rows: 8289)
```

Here's the list of shortcut names for every dataset available in LinCE:
* `lid_spaeng`
* `lid_hineng`
* `lid_nepeng`
* `lid_msaea`
* `pos_spaeng`
* `pos_hineng`
* `ner_spaeng`
* `ner_hineng`
* `ner_msaea`
* `sa_spaeng`


All the numbers match with Table 3 in the LinCE [paper](https://www.aclweb.org/anthology/2020.lrec-1.223.pdf). Also, note that the MSA-EA datasets use the Persian script while the other datasets use the Roman script.


## Features

Here is how the features look in the case of language identification (LID) tasks:

| LID Feature          | Type          | Description                               |
|----------------------|---------------|-------------------------------------------|
| `idx`                | `int`       | Dataset index of current sentence         |
| `tokens`             | `list<str>` | List of tokens (string) of a sentence     |
| `lid`                | `list<str>` | List of LID labels (string) of a sentence |

For part-of-speech (POS) tagging:

| POS Feature          | Type          | Description                               |
|----------------------|---------------|-------------------------------------------|
| `idx`                | `int`       | Dataset index of current sentence         |
| `tokens`             | `list<str>` | List of tokens (string) of a sentence     |
| `lid`                | `list<str>` | List of LID labels (string) of a sentence |
| `pos`                | `list<str>` | List of POS tags (string) of a sentence   |

For named entity recognition (NER):

| NER Feature          | Type          | Description                               |
|----------------------|---------------|-------------------------------------------|
| `idx`                | `int`       | Dataset index of current sentence         |
| `tokens`             | `list<str>` | List of tokens (string) of a sentence     |
| `lid`                | `list<str>` | List of LID labels (string) of a sentence |
| `ner`                | `list<str>` | List of NER labels (string) of a sentence |

**NOTE**: the MSA-EA NER dataset does not contain the `lid` feature.

For sentiment analysis (SA):

| SA Feature          | Type        | Description                               |
|---------------------|-------------|-------------------------------------------|
| `idx`               | `int`       | Dataset index of current sentence         |
| `tokens`            | `list<str>` | List of tokens (string) of a sentence     |
| `lid`               | `list<str>` | List of LID labels (string) of a sentence |
| `sa`                | `str`       | Sentiment label (string) of a sentence    |

Comments:
I am checking the details of the CI log for the failed test, but I don't see how the error relates to the code I added; the error is coming from a config builder different than the `LinceConfig`, and it crashes when `self.config.data_files` because is self.config is None. I would appreciate if someone could help me find out where I could have messed things up :)

Also, the real and dummy data tests passed before committing and pushing my changes.

Thanks a lot in advance!

```
=================================== FAILURES ===================================
____________________ AWSDatasetTest.test_load_dataset_text _____________________

self = <tests.test_dataset_common.AWSDatasetTest testMethod=test_load_dataset_text>
dataset_name = 'text'

    def test_load_dataset(self, dataset_name):
        configs = self.dataset_tester.load_all_configs(dataset_name)[:1]
>       self.dataset_tester.check_load_dataset(dataset_name, configs)

tests/test_dataset_common.py:243: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tests/test_dataset_common.py:137: in check_load_dataset
    try_from_hf_gcs=False,
../.local/lib/python3.6/site-packages/nlp/builder.py:432: in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
../.local/lib/python3.6/site-packages/nlp/builder.py:466: in _download_and_prepare
    split_generators = self._split_generators(dl_manager, **split_generators_kwargs)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <nlp.datasets.text.bf5568367c6707640e5601a44ed0af98f40a8483db81a7db99b85fab6606fc8b.text.Text object at 0x7efa744ffb70>
dl_manager = <nlp.utils.mock_download_manager.MockDownloadManager object at 0x7efb304c52b0>

    def _split_generators(self, dl_manager):
        """ The `datafiles` kwarg in load_dataset() can be a str, List[str], Dict[str,str], or Dict[str,List[str]].
    
            If str or List[str], then the dataset returns only the 'train' split.
            If dict, then keys should be from the `nlp.Split` enum.
        """
        if isinstance(self.config.data_files, (str, list, tuple)):
            # Handle case with only one split
            files = self.config.data_files
            if isinstance(files, str):
                files = [files]
            return [nlp.SplitGenerator(name=nlp.Split.TRAIN, gen_kwargs={"files": files})]
        else:
            # Handle case with several splits and a dict mapping
            splits = []
            for split_name in [nlp.Split.TRAIN, nlp.Split.VALIDATION, nlp.Split.TEST]:
>               if split_name in self.config.data_files:
E               TypeError: argument of type 'NoneType' is not iterable

../.local/lib/python3.6/site-packages/nlp/datasets/text/bf5568367c6707640e5601a44ed0af98f40a8483db81a7db99b85fab6606fc8b/text.py:24: TypeError
=============================== warnings summary ===============================
... 
=========================== short test summary info ============================
FAILED tests/test_dataset_common.py::AWSDatasetTest::test_load_dataset_text
====== 1 failed, 963 passed, 532 skipped, 5 warnings in 166.33s (0:02:46) ======

Exited with code exit status 1
```

@lhoestq Hi Quentin, I was wondering if you could give some feedback on this error from the `run_dataset_script_tests` script. It seems that's coming from a different config builder than the one I added, so I am not sure why this error would occur. Thanks in advance!

Awesome! Thank you for all your comments! 👌  I will update the PR in a bit with all the required changes 🙂 

Let me just provide a bit of context for my changes:

I was referring to the GLUE, XTREME and WNUT_17 dataset scripts to build mine (not sure if the new documentation was available last week). This is where I took the naming convention for the citation and description variables. Also, these scripts didn't have the `BUILDER_CONFIG_CLASS = LinceConfig` line so I commented this out thinking I didn't need that; I tried this line in my attempts to make the real and dummy data tests pass but it was not helping. 

The problem I was facing was that the tests were passing a default `BuilderConfig` (i.e., `self.config.name` property was set to `'default'` and my custom properties were not available). This means, for example, that within the `def _info(...)`  method, I was not able to access the specific fields of my `LinceConfig` class (which is why I have now a global variable `_LINCE_CITATIONS`, to detach the individual citations from the corresponding LinceConfig objects, as well as I am constructing manually the feature infos). This default `BuilderConfig` is why I added the `if not isinstance(self.config, LinceConfig): return []` statement. Otherwise, accessing custom properties like `self.config.colnames` was failing the test because such properties did not exist in the default config (i.e., it was not a `LinceConfig`).

I will update the PR and see if these problems happen in the CI tests.

Thanks again for the follow-up! @lhoestq 

Ok I see !

To give you more details: the line `BUILDER_CONFIG_CLASS = LinceConfig` tells the tests how to instantiate a config for this dataset. Therefore if you have this line you should have all the fields of your config available.

To fix the errors you get you'll have to, first, have the `BUILDER_CONFIG_CLASS = LinceConfig` line, and second, add default values for the parameters of your config (or the tests functions will be unable to instantiate it by calling `LinceConfig()`.

An example of dataset with a custom config with additional filed like this one is [biomrc](https://github.com/huggingface/nlp/blob/master/datasets/biomrc/biomrc.py).
Feel free to give a look at it if you want.

Thanks for the reference!

I just updated the PR with the suggested changes. It seems the CI failed on the same test you said we could ignore, so I guess it's okay :) 

Please let me know if there is something else I may need to change.