t20062443/AT1ds huggingface dataset:
TheophileBlard (CONTRIBUTOR) commented on 2020-06-09 09:46:49:
@mariamabarham  @lbourdois, Yea I don't think we can had the parsing datasets without asking the authors permission first. I also hope they'll change their policy.

Regarding `vsd` and `nsd`, if I understand well the task, the labels are "word senses" and the goal is to find the correct word sense for each ambiguous word. For `vsd` there is one ambiguous verb per sentence, and the labels we manually annotated with "wiktionary senses". For `nsd`, there are multiple ambiguous word per sentence, and the labels are WordNet Princeton Identifiers (hence the WN tag). This dataset was translated in french & automatically aligned.

Imo, for these 2 datasets, each example should be made of:
-  a list of string tokens (the words of the sentence)
- a list of string labels (the word senses or 'O' when the word is not ambiguous.

In fact, for `vsd` it could be even simpler, with a single string label (as there is only one ambiguous verb), + some "idx" feature to indicate the location of the ambiguous verb.

Unfortunately, I cannot update your PR as I'm not a maintainer of the project. Maybe we could work together on a fork ? Here's [mine](https://github.com/TheophileBlard/nlp/commits/flue-benchmark).
fotisj (NONE) commented on 2020-06-09 16:06:47:
On Windows Python supports Unicode almost everywhere, but one of the notable exceptions is open() where it uses the locale encoding schema. So platform independent python scripts would always set the encoding='utf-8' in calls to open explicitly. 
In the meantime: since Python 3.7 Windows users can set the default encoding for everything including open() to Unicode by setting this environment variable: set PYTHONUTF8=1 (details can be found in [PEP 540](https://www.python.org/dev/peps/pep-0540/))

For me this fixed the problem described by the OP.
sarahwie (NONE) commented on 2020-06-09 16:46:39:
Given another source of data loaded in, I want to pre-add it to the dataset so that it aligns with the indices of the arrow dataset prior to performing map.

E.g. 
```
new_info = list of length dataset['train']

dataset['train'] = dataset['train'].map(lambda x: some_function(x, new_info[index of x]))

def some_function(x, new_info_x):
    # adds new_info[index of x] as a field to x
    x['new_info'] = new_info_x
    return x
```
I was thinking to instead create a new field in the arrow dataset so that instance x contains all the necessary information when map function is applied (since I don't have index information to pass to map function).
sarahwie (NONE) commented on 2020-06-09 16:49:13:
This is what I have so far: 

```
import pyarrow as pa
from nlp.arrow_dataset import Dataset

aug_dataset = dataset['train'][:]
aug_dataset['new_info'] = new_info

#reformat as arrow-table
schema = dataset['train'].schema

# this line doesn't work:
schema.append(pa.field('new_info', pa.int32()))

table = pa.Table.from_pydict(
    aug_dataset,
    schema=schema
)
dataset['train'] = Dataset(table) 
```
thomwolf (MEMBER) commented on 2020-06-09 16:51:05:
Maybe you can use `with_indices`?

```python
new_info = list of length dataset['train']

def some_function(indice, x):
    # adds new_info[index of x] as a field to x
    x['new_info'] = new_info_x[indice]
    return x

dataset['train'] = dataset['train'].map(some_function, with_indices=True)
```
lhoestq (MEMBER) commented on 2020-06-10 08:59:44:
Hi ! This is because `.map` added the new column `input_ids` to the dataset, and so all the other columns were kept. Therefore the dataset size increased a lot.
 If you want to only keep the `input_ids` column, you can stash the other ones by specifying `remove_columns=["title", "text"]` in the arguments of `.map`
nateraw (CONTRIBUTOR) commented on 2020-06-10 21:45:32:
If others run into this issue, a quick fix is to use python 3.6 instead of 3.7+. Serialization differences between the 3rd party `dataclasses` package for 3.6 and the built in `dataclasses` in 3.7+ cause the issue.

Probably a dumb fix, but it works for me.
lhoestq (MEMBER) commented on 2020-06-11 08:30:14:
Hello ! Thanks for adding this one :)

This looks great, you just have to do the last steps to make the CI pass.
I can see that two things are missing:
1. the dummy data that is used to test that the script is working as expected
2. the json file with all the infos about the dataset

You can see the steps to help you create the dummy data and generate the dataset_infos.json file right [here](https://github.com/huggingface/nlp/blob/master/CONTRIBUTING.md#how-to-add-a-dataset)
lhoestq (MEMBER) commented on 2020-06-11 09:13:45:
When you install `nlp` for the first time on a Colab runtime, it updates the `pyarrow` library that was already on colab. This update shows this message on colab:
```
WARNING: The following packages were previously imported in this runtime:
  [pyarrow]
You must restart the runtime in order to use newly installed versions.
```
You just have to restart the runtime and it should be fine.
If you don't restart, then it breaks like in your message.
lhoestq (MEMBER) commented on 2020-06-11 09:37:08:
Currently you can indeed split a dataset using `ds_test = nlp.load_dataset('imdb, split='test[:5000]')` (works also with percentages).

However right now we don't have a way to shuffle a dataset but we are thinking about it in the discussion in #166. Feel free to share your thoughts about it.

One trick that you can do until we have a better solution is to shuffle and split the indices of your dataset:
```python
import nlp
from sklearn.model_selection import train_test_split

imdb = nlp.load_dataset('imbd', split='test')
test_indices, val_indices = train_test_split(range(len(imdb)))
```

and then to iterate each split:
```python
for i in test_indices:
    example = imdb[i]
   ...
```
richarddwang (CONTRIBUTOR) commented on 2020-06-11 09:53:19:
In `books.py`
```
URL = "https://storage.googleapis.com/huggingface-nlp/datasets/toronto_books_corpus/bookcorpus.tar.bz2"
```
```
Python 3.7.6 (default, Jan  8 2020, 19:59:22) 
[GCC 7.3.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from nlp import load_dataset
>>> book = load_dataset("nlp/datasets/bookscorpus/books.py", cache_dir='~/tmp')
Downloading and preparing dataset bookscorpus/plain_text (download: 1.10 GiB, generated: 4.52 GiB, total: 5.62 GiB) to /home/yisiang/tmp/bookscorpus/plain_text/1.0.0...
Downloading: 100%|███████████████████████████████████████████████████████████| 1.18G/1.18G [00:39<00:00, 30.0MB/s]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/yisiang/nlp/src/nlp/load.py", line 520, in load_dataset
    save_infos=save_infos,
  File "/home/yisiang/nlp/src/nlp/builder.py", line 420, in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
  File "/home/yisiang/nlp/src/nlp/builder.py", line 460, in _download_and_prepare
    verify_checksums(self.info.download_checksums, dl_manager.get_recorded_sizes_checksums())
  File "/home/yisiang/nlp/src/nlp/utils/info_utils.py", line 31, in verify_checksums
    raise ExpectedMoreDownloadedFiles(str(set(expected_checksums) - set(recorded_checksums)))
nlp.utils.info_utils.ExpectedMoreDownloadedFiles: {'16KCjV9z_FHm8LgZw05RSuk4EsAWPOP_z'}
>>>
```

BTW, I notice the path `huggingface-nlp/datasets/toronto_books_corpus`, does it mean I have to change  folder name "bookscorpus" to "toronto_books_corpus"
lhoestq (MEMBER) commented on 2020-06-11 10:08:27:
> In `books.py`
> 
> ```
> URL = "https://storage.googleapis.com/huggingface-nlp/datasets/toronto_books_corpus/bookcorpus.tar.bz2"
> ```
> 
> ```
> Python 3.7.6 (default, Jan  8 2020, 19:59:22) 
> [GCC 7.3.0] :: Anaconda, Inc. on linux
> Type "help", "copyright", "credits" or "license" for more information.
> >>> from nlp import load_dataset
> >>> book = load_dataset("nlp/datasets/bookscorpus/books.py", cache_dir='~/tmp')
> Downloading and preparing dataset bookscorpus/plain_text (download: 1.10 GiB, generated: 4.52 GiB, total: 5.62 GiB) to /home/yisiang/tmp/bookscorpus/plain_text/1.0.0...
> Downloading: 100%|███████████████████████████████████████████████████████████| 1.18G/1.18G [00:39<00:00, 30.0MB/s]
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
>   File "/home/yisiang/nlp/src/nlp/load.py", line 520, in load_dataset
>     save_infos=save_infos,
>   File "/home/yisiang/nlp/src/nlp/builder.py", line 420, in download_and_prepare
>     dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
>   File "/home/yisiang/nlp/src/nlp/builder.py", line 460, in _download_and_prepare
>     verify_checksums(self.info.download_checksums, dl_manager.get_recorded_sizes_checksums())
>   File "/home/yisiang/nlp/src/nlp/utils/info_utils.py", line 31, in verify_checksums
>     raise ExpectedMoreDownloadedFiles(str(set(expected_checksums) - set(recorded_checksums)))
> nlp.utils.info_utils.ExpectedMoreDownloadedFiles: {'16KCjV9z_FHm8LgZw05RSuk4EsAWPOP_z'}
> >>>
> ```
> 
> BTW, I notice the path `huggingface-nlp/datasets/toronto_books_corpus`, does it mean I have to change folder name "bookscorpus" to "toronto_books_corpus"

Let me change the url to match "bookscorpus", so that you don't have to change anything. Good catch.

About the error you're getting: you just have to remove the `dataset_infos.json` and regenerate it
richarddwang (CONTRIBUTOR) commented on 2020-06-11 11:53:43:
Hi, I found I made a mistake. I found the ELECTRA paper refer it as "BooksCorpus", but actually it is caleld "BookCorpus", according to the original paper. Sorry, I should have checked the original paper .

Can you do me a favor and change the url path to ` https://storage.googleapis.com/huggingface-nlp/datasets/bookcorpus/bookcorpus.tar.bz2` ?
richarddwang (CONTRIBUTOR) commented on 2020-06-11 15:03:06:
Thank you @lhoestq ,
Just to confirm it fits your naming convention
* make the file path `book_corpus/book_corpus.py` ?
* make `class Bookscorpus(nlp.GeneratorBasedBuilder)` -> `BookCorpus`  (which make cache folder name `book_corpus` and user use `load_dataset('book_corpus')`) ?
(Cuz I found "HellaSwag" dataset is named "nlp/datasets/hellaswag" and `class Hellaswag` )
lhoestq (MEMBER) commented on 2020-06-11 15:37:06:
Oh yea you're right about the Hellaswag example. We should keep the "_" symbol to replace spaces. As there are no space in BookCorpus, what we should do here is use:
- class name: 'Bookcorpus'
- script name: `bookcorpus/bookcorpus.py`
- use url https://storage.googleapis.com/huggingface-nlp/datasets/bookcorpus/bookcorpus.tar.bz2
And therefore the dataset name will be `bookcorpus`

Don't forget to regenerate the `dataset_infos.json` and we'll be good :D
yjernite (MEMBER) commented on 2020-06-11 16:29:05:
There we go: added a `test_dummy_data` class variable that is `False` by default for the `BeamBasedBuilder` and `True` for everyone else (except the new `explainlikeimfive` and `wiki_snippets`)

Note that `wiki_snippets` should become obsolete as soon as @lhoestq adds in the `IndexedDataset` class
lhoestq (MEMBER) commented on 2020-06-12 08:42:47:
Nice !

Also it looks like we can have a train_test_split method for free:
```python
train_indices, test_indices = train_test_split(range(len(dataset)))
train = dataset.sort(indices=train_indices)
test = dataset.sort(indices=test_indices)
```

and a shuffling method for free:
```python
shuffled_indices = shuffle(range(len(dataset)))
shuffled_dataset = dataset.sort(indices=shuffled_indices)
```

Maybe we can have a specific API for train_test_split and shuffle. They are two features asked quite often (see #147, #166)
ghomasHudson (CONTRIBUTOR) commented on 2020-06-15 09:10:49:
From thinking upon @thomwolf 's suggestion, I've started experimenting:
```python
class MultitaskDataset(DatasetBuilder):
    def __init__(self, *args, **kwargs):
        super(MultitaskDataset, self).__init__(*args, **kwargs)
        self._datasets = kwargs.get("datasets")

    def _info(self):
        return nlp.DatasetInfo(
            description=_DESCRIPTION,
            features=nlp.Features({
                    "source": nlp.Value("string"),
                    "target": nlp.Sequence(nlp.Value("string"))
                })
        )

    def _get_common_splits(self):
        '''Finds the common splits present in all self._datasets'''
        min_set = None
        for dataset in self._datasets:
            if min_set != None:
                min_set.intersection(set(dataset.keys()))
            else:
                min_set = set(dataset.keys())
        return min_set

....

# Maybe this?:
squad = nlp.load_dataset("squad")
cnn_dm = nlp.load_dataset("cnn_dailymail","3.0.0")
multitask_dataset = nlp.load_dataset(
    'multitask_dataset',
    datasets=[squad,cnn_dailymail], 
    k=1000, 
    t=2.0
)

```

Does anyone know what methods of `MultitaskDataset` I would need to implement? Maybe `as_dataset` and `download_and_prepare`? Most of these should be just calling the methods of the sub-datasets. 

I'm assuming DatasetBuilder is better than the more specific `GeneratorBasedBuilder`, `BeamBasedBuilder`, etc....

One of the other problems is that the dataset size is unknown till you construct it (as you can pick the sub-datasets). Am hoping not to need to make changes to `nlp.load_dataset` just for this class.

I'd appreciate it if anyone more familiar with nlp's internal workings could tell me if I'm on the right track!
lhoestq (MEMBER) commented on 2020-06-15 16:53:58:
Since the latest release 0.2.1 you can use 
```bash
nlp-cli upload_dataset <path/to/dataset>
```
where `<path/to/dataset>` is a path to a folder containing your script (ex: `squad.py`).
This will upload the script under your namespace on our S3.

Optionally the folder can also contain `dataset_infos.json` generated using
```bash
nlp-cli test <path/to/dataset> --all_configs --save_infos
```

Then you should be able to do
```python
nlp.load_dataset("my_namespace/dataset_name")
```
fotisj (NONE) commented on 2020-06-15 17:50:04:
Thanks a lot, the new explanation is very helpful!

About using train_test_split from sklearn: I stumbled across the [same error message as this user ](https://github.com/huggingface/nlp/issues/147 )and thought it can't be used at the moment in this context. Will check it out again.

One of the problems is how to shuffle very large datasets, which don't fit into the memory. Well, one strategy could be shuffling data in sections. But in a case where the data is sorted by the labels you have to swap larger sections first.
lhoestq (MEMBER) commented on 2020-06-16 08:39:03:
For some reason the files are not available for unauthenticated users right now (like the download service of this package). Instead of downloading the right files, it downloads the html of the error.
According to the error it should be back again in 24h.

![image](https://user-images.githubusercontent.com/42851186/84751599-096c6580-afbd-11ea-97f3-ee4aef791711.png)
lhoestq (MEMBER) commented on 2020-06-16 14:22:24:
The configuration menu is fine imo.
It was more about the code snippet presented in the viewer.
For example for Allociné it currently shows this snippet to load the dataset:
```python
!pip install nlp
from nlp import load_dataset
dataset = load_dataset('allocine', 'allocine')
```
However for datasets with one or zero configurations, the second argument in `load_dataset` is optional. For Allociné, that has one configuration, we can expect to show instead:
```python
!pip install nlp
from nlp import load_dataset
dataset = load_dataset('allocine')
```
stefan-it (CONTRIBUTOR) commented on 2020-06-16 15:24:44:
Awesome! This is working now:

```python
import nlp 
seqeval = nlp.load_metric("seqeval") 
y_true = [['O', 'O', 'O', 'B-MISC', 'I-MISC', 'I-MISC', 'O'], ['B-PER', 'I-PER', 'O']] 
y_pred = [['O', 'O', 'B-MISC', 'I-MISC', 'I-MISC', 'I-MISC', 'O'], ['B-PER', 'I-PER', 'O']]                                                                                                                                                                                    

results = seqeval.compute(y_true, y_pred)
```

I heavily need this fix for an upcoming `nlp` integration PR for Transformers (token classification example)  😅
TheophileBlard (CONTRIBUTOR) commented on 2020-06-16 17:07:05:
> Actually when there is only one configuration, then you don't need to specify the configuration in `load_dataset`. You can run:
> 
> ```python
> dataset = load_dataset('allocine')
> ```
> 
> and it works.
> 
> Maybe we should take that into account in the nlp viewer @srush ?

Oh ok, I didn't expect it would work! 

Anyway, I think it's intrinsically better to simply remove the optional parameter. 
The dummy data folder architecture seems also more logical this way.
lhoestq (MEMBER) commented on 2020-06-16 23:15:25:
> @mariamabarham Just wanted to note that default behavior `cos_e = load_dataset("cos_e")` now loads `v1.0`. Not sure if this is intentional (but the flag specification does work as intended).

In the new version of `nlp`, if you try `cos_e = load_dataset("cos_e")` it throws this error:
```
ValueError: Config name is missing.
Please pick one among the available configs: ['v1.0', 'v1.11']
Example of usage:
	`load_dataset('cos_e', 'v1.0')`
```
For datasets with at least two configurations, we now force the user to pick one (no default)
thomwolf (MEMBER) commented on 2020-06-17 09:33:48:
Ok, I think this one is ready to merge.

@patrickvonplaten @jplu @mariamabarham @joeddav @n1t0 @julien-c you may want to give it a look, it adds a bunch of methods to reorder/split/select rows in a dataset:
- `dataset.select(indices)`: Create a new dataset with rows selected following the list/array of indices (which can have a different size than the dataset and contain duplicated indices, the only constrain is that all the integers in the list must be smaller than the dataset size, otherwise we're indexing outside the dataset...)
- `dataset.sort(column_name)`: sort a dataset according to a column (has to be a column with a numpy compatible type)
- `dataset.shuffle(seed)`: shuffle a dataset rows
- `dataset.train_test_split(test_size, train_size)`: Return a dictionary with two random train and test subsets (`train` and `test` ``Dataset`` splits)

All these methods are **not** in-place which means they return new ``Dataset``, which is the default behavior in the library.
lhoestq (MEMBER) commented on 2020-06-17 15:22:02:
Hi !

As you noticed, "big" datasets like Wikipedia require apache beam to be processed.
However users usually don't have an apache beam runtime available (spark, dataflow, etc.) so our goal for this library is to also make available processed versions of these datasets, so that users can just download and use them right away.

This is the case for english and french wikipedia right now: we've processed them ourselves and now they are available from our google storage. However we've not processed the german one (yet).
gregburman (NONE) commented on 2020-06-17 15:33:03:
Hi @lhoestq 

Thank you for your quick reply. I thought this might be the case, that the processing was done for some languages and not for others. Is there any set timeline for when other languages (German, Italian) will be processed?

Given enough memory, is it possible to process the data ourselves by specifying the `beam_runner`?
lhoestq (MEMBER) commented on 2020-06-17 16:31:31:
Adding them is definitely in our short term objectives. I'll be working on this early next week :)

Although if you have an apache beam runtime feel free to specify the beam runner. You can find more info [here](https://github.com/huggingface/nlp/blob/master/docs/beam_dataset.md) on how to make it work on Dataflow but you can adapt it for Spark or any other beam runtime (by changing the `runner`).

However if you don't have a beam runtime and even if you have enough memory, I discourage you to use the `DirectRunner` on the german or italian wikipedia. According to Apache Beam documentation it was made for testing purposes and therefore it is memory-inefficient.
lhoestq (MEMBER) commented on 2020-06-17 17:32:59:
When you're processing a dataset with `.map`, it checks whether it has already done this computation using a hash based on the function and the input (using some fancy serialization with `dill`). If you found that it doesn't work as expected in some cases, let us know !

Given that, you can still force to re-process using `.map(my_func, load_from_cache_file=False)` if you want to.

I am curious about the problem you have with splits. It makes me think about #160 that was an issue of version 0.1.0. What version of `nlp` are you running ? Could you give me more details ?
jplu (CONTRIBUTOR) commented on 2020-06-18 11:17:40:
Hi @MFreidank, it is possible to load a dataset from your local storage, but only CSV/TSV and JSON are supported. To load a dataset in JSON format:

```
nlp.load_dataset(path="json", data_files={nlp.Split.TRAIN: ["path/to/train.json"], nlp.Split.TEST: ["path/to/test.json"]})
```

For CSV/TSV datasets, you have to replace `json` by `csv`.
lhoestq (MEMBER) commented on 2020-06-18 15:32:32:
> Might be a solution to put 0.25 and 0.75 as default values for respectively `test_size` and `train_size`. WDYT?

According to sklearn documentation, it is indeed set to 0.25 and 0.75 if both `test_size` and `train_size` are None.
Let me add it.
lhoestq (MEMBER) commented on 2020-06-18 15:50:13:
This print statement comes from the official implementation of bert_score (see [here](https://github.com/Tiiiger/bert_score/blob/master/bert_score/utils.py#L343)). The warning shows up only if the attention mask outputs no candidate.
Right now we want to only use official code for metrics to have fair evaluations, so I'm not sure we can do anything about it. Maybe you can try to create an issue on their [repo](https://github.com/Tiiiger/bert_score) ?
sshleifer (CONTRIBUTOR) commented on 2020-06-18 17:14:44:
But I get
```python
ConnectionError: Couldn't reach https://s3.amazonaws.com/datasets.huggingface.co/nlp/datasets/./datasets/wmt16/wmt16.py
```
When I try from jupyter on brutasse or my mac. (the jupyter server is run from transformers).


Both machines can run
```bash
aws s3 ls s3://datasets.huggingface.co/nlp/datasets/wmt16/
```
but it seems one must be in the nlp directory to run the command?

(I ran `pip install -e . ` on this branch in both situations.)
lhoestq (MEMBER) commented on 2020-06-18 22:20:24:
We added a way to shuffle datasets (shuffle the indices and then reorder to make a new dataset).
You can do `shuffled_dset = dataset.shuffle(seed=my_seed)`. It shuffles the whole dataset.
There is also `dataset.train_test_split()` which if very handy (with the same signature as sklearn).

Closing this issue as we added the docs for splits and tools to split datasets. Thanks again for your feedback !
lhoestq (MEMBER) commented on 2020-06-19 07:40:05:
> But I get
> 
> ```python
> ConnectionError: Couldn't reach https://s3.amazonaws.com/datasets.huggingface.co/nlp/datasets/./datasets/wmt16/wmt16.py
> ```
> 
> When I try from jupyter on brutasse or my mac. (the jupyter server is run from transformers).
> 
> Both machines can run
> 
> ```shell
> aws s3 ls s3://datasets.huggingface.co/nlp/datasets/wmt16/
> ```
> 
> but it seems one must be in the nlp directory to run the command?
> 
> (I ran `pip install -e . ` on this branch in both situations.)

As soon as it is on master, the dataset script wmt16.py will be synced on S3 and you'll be able to do `load_dataset("wmt16")`
lhoestq (MEMBER) commented on 2020-06-19 09:41:11:
I added the German and the Italian Wikipedia to our google cloud storage:
First update the `nlp` package to 0.3.0:
```bash
pip install nlp --upgrade
```
and then
```python
from nlp import load_dataset
wiki_de = load_dataset("wikipedia", "20200501.de")
wiki_it = load_dataset("wikipedia", "20200501.it")
```
The datasets are downloaded and directly ready to use (no processing).